Lead Data Engineer - Remote Job at Lorven technologies, Remote

ZUgvSmVIeFVBZWd5SlhHYVh0YjFic1c2QXc9PQ==
  • Lorven technologies
  • Remote

Job Description

Hi,

Our client is looking Technical Lead/Data Librarian with Python and Pyspark and AWS in Remote below is the detailed requirements.

Job Title : Technical Lead/Data Librarian with Python and Pyspark and AWS

Location : Remote

Duration : Long-Term Contract

The ideal candidate will have a strong background in data engineering and development, with expertise in various data skills and tools. The role involves working on complex data projects, collaborating with cross-functional teams, and ensuring the efficient processing and management of data.

Key Responsibilities:

  • SQL skills to understand the transformation and perform analysis
  • PySpark and MWAA DAGs skills to understand the data pipelines
  • AWS Glue Data Catalog/ LF skills to understand the access control
  • Lakehouse concepts and databases to understand consumption patterns Architectural ownership
  • Cross functional collaboration
  • Strategic vision
  • Develop and maintain data pipelines using Python and Pyspark.
  • Design and implement efficient SQL queries for data extraction and manipulation.
  • Work with Aurora Postgres and Redshift for data storage and management.
  • Utilize Informatica or other ETL tools for data integration and transformation.
  • Implement and manage AWS cloud services, including Athena, S3, ECS/Docker, EMR, EC2, Lambda, CloudWatch, and EventBridge.
  • Use Airflow or other orchestration tools for workflow management.
  • Ensure data quality and integrity through rigorous testing and validation.
  • Collaboxxxvith other teams to understand data requirements and deliver solutions.

Required Skills:

  • Proficiency in Python and Pyspark.
  • Strong SQL skills.
  • Experience with Aurora Postgres and Redshift.
  • Knowledge of Informatica or other ETL tools.
  • Experience with Al.JS cloud services, including Athena, S3, Lambda, CloudWatch, and EventBridge.
  • Ability to work with other cloud technologies (GCP, Azure) and understand AWS touch points.

Preferred Skills:

  • ECS/ Docker (or any docker experience/concepts)
  • EMR, EC2 (or knowledge of any other distributed computing)
  • Familiarity with Airflow or other orchestration tools.

Job Tags

Contract work,

Similar Jobs

Endeavor Agency

Project Manager Job at Endeavor Agency

 ...Our GC client is searching for a Project Manager or Assistant Project Manager to join their growing team. Project Manager The ideal...  ...Responsibilities Manage commercial tenant improvement projects from contract award through project closeout Oversee day-to-day project... 

Gilero, A Sanner Group Company

Senior Systems Engineer - Medical Devices Job at Gilero, A Sanner Group Company

 ...Senior Systems Engineer Gilero is a leading medical device design, development, and manufacturing partner. We specialize in creating innovative solutions that improve patient care and enhance lives. Our growing dynamic team of professionals works across disciplines... 

The Bolton Group

Finance Manager Job at The Bolton Group

 ...The Bolton Group is partnering with a well-established and large homebuilder to look for a Finance Manager for their Edinburg, Texas location. The Finance Manager is responsible for providing financial and analytical support to help drive operational success in sales... 

DPH Delivery

Amazon Package Delivery Driver @$23.25/hour- CDL Not Required Job at DPH Delivery

 ...DPH Delivery is a locally owned and operated Amazon DSP dedicated to providing exceptional...  ...to our fast-paced team of Delivery Drivers! NOW HIRING for Full-Time (Looking for...  ...vans in an organized manner and deliver packages by driving, climbing in and out of the vehicle... 

SAS Retail Services

Traveling Retail Representative Job at SAS Retail Services

 ...experiences that enrich their lives. Joining SAS Retail Services, an Advantage Solutions...  ...this role, you will provide in-store merchandising support to Retailers to meet shoppers'...  ...maximize sales opportunities. This is a traveling position that requires regular travel...