c2c it jobs
Houston, TX
Long-term Contract
Onsite Work
Job Description
Here is the job description, this is a client facing role so need someone who can talk, interact, and interface with the clients weekly. Located in Houston, TX
Develop, construct, test and maintain data acquisition pipelines for large volumes of structed and unstructured data. This includes batch and real-time processing (in google cloud)
Build large and complex datasets based on business requirements
Construct ‘big data’ pipeline architecture
Identifies opportunities for data acquisition via working with stakeholders and business clients
Leverages a variety of tools such as Python, Data Flow, DataStream, Google Cloud Functions, Spark, Google Cloud Run, SAP SLT, Google Pub Sub etc. to integrate systems and data pipelines
Recommends ways to improve data quality, reliability, and efficiency
Develop JSON messaging structure for integrating with various applications
Drive a data engineering strategy focused on delivering data in near real-time using serverless technologies
Develop data architecture to optimize performance for dashboards and digital applications
Leverage DevOps and CI/CD practices to ensure the reliability and scalability of data pipelines
Experience:
Requires 7+ years of experience in a data engineering role
Strong SQL background working with a variety of data sources
Expert using languages such as Python
Expert leveraging messaging queue technologies such as Kafka and Pub/Sub
Expert transforming streaming data using technologies such as Data Flow, DataStream, Spark streaming
Expert developing pipelines for unstructured data
Expert in native cloud back-end technologies such as Big Query, Data Flow, Pub/Sub, Cloud Functions, Cloud Run
Expertise with cloud-based data platforms (GCP).
Expertise with DevOps and deploying and maintaining CI CD pipelines
To apply for this job email your details to kiran@keybusinessglobal.com