Developing a data preparation layer
Overview
Industry
Location
Telecom
Poland
Technology Used
Airflow
Ambari
Ambari Metrics
Ansible
Apache Hive
Git
Hadoop
Jenkins
Oracle DB
PySpark
Python
Ranger KMS
Spark
SQL
ZooKeeper
Show more
Project overview
Shortly after deciding to use a cloud-based AI solution, our client faced a challenge of interacting with the AI platform by exporting event data from a Big Data platform and Data Warehouse to the cloud and next, ingesting produced predictions from models back to the Data Lake for further analysis. Our client needed a generic solution to prepare and produce thousands of events daily/weekly/monthly, regardless of transformation logic, and support various techniques of event calculation.
Client background
A major operator and service provider in Poland – part of global international mobile communications group.
Goals
Our client needed a solution that could support thousands of data pipelines and calculate events using a huge volume of data. Security was paramount, so the process of exporting data to the cloud had to include pseudonymization. Additionally, we needed to implement configurable data quality checks as part of the process.
Results
Managing thousands of data pipelines
Developing a secure solution that handles vast amounts of data from diverse sources, while enabling fully automated deployments and scheduling.
Increased automation
Ensuring fully automated deployments and scheduling (based on configuration) with custom monitoring of flows.
Boosted security
Exporting data that does not include personal information and ensuring secure S3 interface interactions.
We'd love to hear from you!
Fill out the form - we'll get back to you as soon as possible
