Intro
Eyeview is a video advertising company which specializes in delivering relevant ads to users across their devices in real-time. As for any data driven company, the quality of our data determines the power of our algorithms, and ultimately, the effectiveness of our ads. Part of the challenge we face is maintaining data integrity across many services that use it. In this post, we will focus on one challenge as it relates to audience data – a dataset which is critical for accurate ad targeting.
The Challenge of Many Hops and Data Integrity:
Audience data comes to us from external partners. It travels across five different services before it gets to its most important destination (Aerospike). There, it gets accessed during real-time bidding to match a user with an audience in a matter of milliseconds. The services consuming the data are:
- S3 : distributed file storage provided by AWS
- Spark : our big data lake
- AWS Kinesis : data bus used to persist and share data via streams
- AWS KCL apps : data enrichment and dispatching services on top of Kinesis
- Aerospike : scalable key value store used for real time bidding
Each data hop presents unique challenges in terms of checking integrity and ensuring no data loss occurs. Here, we will describe an issue we found in one of these interactions – namely between Spark and Kinesis and the innovative approach we took to solve the issue.
The Challenge of Dealing with AWS Kinesis:
Sending data to Kinesis is not straightforward, due to the number of limits it imposes on the sender. The limits are expressed in terms of rate of batches sent, rate of bytes sent as well as max entries and bytes per batch. Violating these limits results in rejection of sent data via an exception. Depending on the capacity of the stream, (expressed in units of parallelism called shards) the effective limits can vary significantly. To deal with the issue we developed a component called Kinesis Buffered Client (KBC) which uses buffering and throttling to enforce rate limits. KBC uses a common worker pool pattern based on java’s executor framework (a queue for requests and a pool of thread-based workers taking items off the queue and processing them in the background).
The global data classification market size is expected to reach USD 2.44 billion by 2025, expanding at a CAGR of 25.7% over the forecast period, according to a new report by Grand View Research, Inc. Increasing stringency on regulatory compliance, generation of large data volume, and the introduction of big data and machine learning technology are some of the major growth drivers for the market.
Moreover, the data-driven decision-making for sales, rising product launch, and increasing number of enterprises are some of the factors providing impetus to the growth.Data classification offered as a standalone and integrated solution is changing the way the information is generated, stored, and processed.
The homogeneously tagged information with industry defined classification is vital in generating actionable insights.
Hence, enterprises are inclined to opt for integrated data security platform for data classification as well as protection.
Moreover, repercussion of non-compliance can lead to huge financial and reputational losses.
Hence, the adoption of data classification solution is vital to address the data security concerns of the organizations as it classifies the information based on the nature of content.Data classification solution is anticipated to witness high adoption across major developed countries such as U.S., Canada, Germany, and Japan.
