注册 / 登录

Building Pinterest Real-Time Ads Platform Using Kafka Streams

分会场:  爆款架构/数据平台/工程实践

 

案例来源 :

案例讲师

陈博玚

Pinterest 架构师

Pinterest广告系统架构工程师、架构师,工作中大量使用了 Kafka Streams来组建实时的消费系统,从而实现有效且高效的广告投放,最大程度避免了过度投放对营收带来的影响。曾经在Kafka Summit 2018中分享过实际工作中书写、部署、和优化 Kafka Streams的经验,以期能够给听众更多在实际生产中使用 Kafka Streams需要注意的事项,以及制作广告系统中可能会遇到的困难以及解决思路。

扫描二维码分享案例

 

案例简述

 

广告系统中,有很多需要实时解决的问题。对于Pinterest这样月活2.5亿,图片量100B的平台来说,这些问题是什么?解决起来又有哪些挑战?希望通过本次分享,能够对具有相同情况的朋友起到抛砖引玉之效。

 

案例目标

 

In this talk, we are sharing the experience of building Pinterest’s real-time Ads Platform utilizing Kafka Streams. The real-time budgeting system is the most mission-critical component of the Ads Platform as it controls how each ad is delivered to maximize user, advertiser and Pinterest value. The system needs to handle over 50,000 queries per section (QPS) impressions, requires less than five seconds of end-to-end latency and recovers within five minutes during outages. It also needs to be scalable to handle the fast growth of Pinterest’s ads business.

 

成功(或教训)要点

 

The real-time budgeting system is composed of real-time stream-stream joiner, real-time spend aggregator and a spend predictor. At Pinterest’s scale, we need to overcome quite a few challenges to make each component work. For example, the stream-stream joiner needs to maintain terabyte size state while supporting fast recovery, and the real-time spend aggregator needs to publish to thousands of ads servers while supporting over one million read QPS. We choose Kafka Streams as it provides milliseconds latency guarantee, scalable event-based processing and easy-to-use APIs. In the process of building the system, we performed tons of tuning to RocksDB, Kafka Producer and Consumer, and pushed several open source contributions to Apache Kafka. We are also working on adding a remote checkpoint for Kafka Streams state to reduce the time of code start when adding more machines to the application. We believe that our experience can be beneficial to people who want to build real-time streaming solutions at large scale and deeply understand Kafka Streams.

 

案例ROI分析

 

WIP

 

案例启示

 

1.对新兴流处理技术有一个基本的理解;

2.对广告平台架构的认识得到深入;

3.对如何搭建公司通用数据平台有一些新的思考。

 

领取大会PPT

我要参会

大会全套演讲PPT

立即领取

大会即将开幕,点击抢票!

我要参会