Job Title

Data Engineer

  • Position:
  • Salary: $$50
  • Location:
      Remote
  • Work Eligibility: USC, GC, GC-EAD, TN, H1, H4-EAD, OPT-EAD, CPT
  • Job ID: 07800
Share This Job
Required Skills:

Pandya

239 Active Positions

Job Description

Job Description:

Key Responsibilities:

Designs and automates deployment of our distributed system for ingesting and transforming data from various types of sources (relational, event-based, unstructured).
Designs and implements framework to continuously monitor and troubleshoot data quality and data integrity issues.
Implements data governance processes and methods for managing metadata, access, retention to data for internal and external users.
Designs and provide guidance on building reliable, efficient, scalable and quality data pipelines with monitoring and alert mechanisms that combine a variety of sources using ETL/ELT tools or scripting languages.
Designs and implements physical data models to define the database structure.
Optimizing database performance through efficient indexing and table relationships.
Participates in optimizing, testing, and troubleshooting of data pipelines.
Designs, develops and operates large scale data storage and processing solutions using different distributed and cloud based platforms for storing data (e.g. Data Lakes, Hadoop, Hbase, Cassandra, MongoDB, Accumulo, DynamoDB, others).
Uses innovative and modern tools, techniques and architectures to partially or completely automate the most-common, repeatable and tedious data preparation and integration tasks in order to minimize manual and error-prone processes and improve productivity.
Assists with renovating the data management infrastructure to drive automation in data integration and management.
Ensures the timeliness and success of critical analytics initiatives by using agile development technologies such as DevOps, Scrum, Kanban Coaches and develops less experienced team members
Skillset:

Intermediate experience in a relevant discipline area is required.
Knowledge of the latest technologies and trends in data engineering are highly preferred and includes:
Familiarity analyzing complex business systems, industry requirements, and/or data regulations – Background in processing and managing large data sets
Design and development for a Big Data platform using open source and third-party tools – SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka or equivalent college coursework
SQL query language
Clustered compute cloud-based implementation experience
Experience developing applications requiring large file movement for a Cloud-based environment and other data extraction tools and methods from a variety of sources
Experience in building analytical solutions Intermediate experiences in the following are preferred: Experience with IoT technology and Experience in Agile software development
Skills:

Intermediate experience in a relevant discipline area is required.
Knowledge of the latest technologies and trends in data engineering are highly preferred and includes:
Familiarity analyzing complex business systems, industry requirements, and/or data regulations
Background in processing and managing large data sets
Design and development for a Big Data platform using open source and third-party tools
SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka or equivalent college coursework
SQL query language
Clustered compute cloud-based implementation experience
Experience developing applications requiring large file movement for a Cloud-based environment and other data extraction tools and methods from a variety of sources
Experience in building analytical solutions Intermediate experiences in the following are preferred: Experience with IoT technology and Experience in Agile software development

[jobboard-shortcode-map-2][/jobboard-shortcode-map-2]
Tags: