首届上海大数据流处理Meetup.pdf

System Technologies and Optimization 首届上海大数据流处理 Meetup姜伟华，Intel 2015年8月22日System Technologies and Optimization 主办：Intel金牌赞助伙伴： 太库System Technologies and Optimization 日程安排 12:45 – 13:00 签到  13:00 – 13:15 开场白  13:15 – 14:00 JStorm/Storm在阿里的实践和社区进展  14:00 – 14:30 StreamCQL - 构建在分布式流处理平台 上的查询语言  14:30 – 14:50 茶歇  14:50 – 15:35 Pulsar– Real-time analytics at Scale  15:35 – 16:15 Gearpump – 基于Akka的开源流处理引擎  16:20 – 17:00 继续讨论System Technologies and Optimization 互联的数据中心2020年*将有 500亿设备互联Zettabytes*级 的数据流量*Estimate by Cisco Systems, Inc.*大数据，大机会大数据，大机会System Technologies and Optimization 5大数据实时流处理流处理：数据到达即处理（而不是先存储以待将来 处理）“ It’s about the ability to make better decisions and take meaningful actions at the right time.”– Michael Minelli, Big Data, Big AnalyticsSystem Technologies and Optimization 流处理应用智能城市• 安全 • 效率智能交通• 道路畅通 • 违章处理 • 套牌实时反欺 诈• 电信、金 融实时推荐• 商品、广 告 • 基于位置 的推荐实时监控• 舆情监控 • 运维监控实时统计实时分析实时数据 导入ETL （清洗、 变换）6System Technologies and Optimization 批处理和流处理平台7毫秒秒分钟小时流处理批处理HeronMillwheelSystem Technologies and Optimization 大数据流处理历程和趋势• 无流处理 • 高延迟批处理• 批处理和流处理 相结合 • 结果以批处理为 准lambda架构• 完全的流处理框 架 • 辅以批处理实时数据分析8System Technologies and Optimization 流处理目前的热点多租户，资源隔离•更好的抽象/语言（例如StreamCQL） •易于部署（和YARN的结合） •可用的各种算法模块（比方说机器学习） •可调试性易用性事务长期运行支持9System Technologies and Optimization Intel和大数据10ComputeStorageNetworkAnalyticsSolutions通过示范解决方案展现出大数据的价值通过降低复杂性和成本来驱动大数据分析的 推广推进开源大数据平台提供开放、标准化的硬件和SDI来催化业界 增长Data PlatformSystem Technologies and Optimization No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. Forecasts: Any forecasts of requirements for goods and services are provided for discussion purposes only. Intel will have no liability to make any purchase pursuant to forecasts. Any cost or expense you incur to respond to requests for information or in reliance on any forecast will be at your own risk and expense.Intel DisclaimerSystem Technologies and Optimization Business Forecast: Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward- looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. © 2015 Intel Corporation.Intel Disclaimer。