Job Description
JOB DESCRIPTION (Sep 2021)
————————————————–
Senior Data Engineer / Software Engineer
You would join the team and work on a variety of projects in Oil & Gas but also projects in renewable energy, B2B, and B2C. Plus you may also be able to work on some modernization focused initiatives.
Our client has expanded their scope of “Data Engineers”, so they are not just focused on data migration or building data pipelines. They need a strong Software Engineer / Software Developer background that incorporates the Big Data experience. Since this role is a merger between those two areas, we need Software Engineers who are also Data Engineers. Candidates will need both skillsets to be successful.
Will work with Data Engineers and Software Engineers, Data Scientists, Machine Learning Engineers, Data Managers, and Business Partners. No prior experience in the energy industry is required.
Not Looking for…
· Data Engineers only focused on migration or building data pipelines.
· Data Architects who only design but not development own solutions.
· Machine Learning Engineers who only want to work on ML models.
· Full Stack Java or Python Developers who prefer front end UI dev.
· Software Engineers without SQL queries or database development.
Minimum Experience Required:
· At least 2+ years of RECENT SPARK experience using own code in Scala, Python (PySpark), Java.
· At least 2+ years of senior level data structures and algorithms experience building, using own code.
· At least 2-5 years of relational database experience using own SQL queries and stored procedures.
· At least 3-5 years of database development or ORM object relational mapping using .Net or Java.
· At least 3-5 years of integrating batch and streaming data pipelines with modern data warehouses.
Top Requirements / Essential Skills:
• Degree in Computer Science or Computer Information Systems will be preferred – at least a BS Degree.
• Software Development experience in one or more object-oriented programming languages (e.g. Python, Go, Java, Scala, C#, C++) Experience with algorithms, data structures, and building systems architecture.
• Ideally has a mix of multiple coding languages – they want someone who is open to different languages and somewhat flexible with the target for writing programming code or rewriting it in another language.
• Advanced SQL coding skills: writing complex queries and stored procedures for database development.
• Experience with multiple relational databases, plus NoSQL, and modern data warehouses on the cloud.
• Uses best practices and standards for data engineering, security, data privacy, reliability, & scalability.
• Recent deep hands-on experience (typically 5+ years) with developing reliable and scalable Big Data infrastructure and data products in large-scale distributed systems for complex enterprise environments.
Key Technologies / Tools:
· Spark applications in Databricks, AWS EMR, Azure HDInsight, Cloudera/Hortonworks, or GCP Dataproc.
· Data integration to/from Data lakes like AWS S3, Azure Blob, Databricks Delta Lake, or Google Storage.
· Building custom data pipelines using Airflow (Python/PySpark), or Kafka (using Scala, Python, or Java).
· Cloud data orchestration tools: Azure Data Factory (ADF), AWS Glue, GCP DataFlow, or Apache Beam.
· Modern data warehouses: Snowflake Cloud DW, AWS Redshift, Azure Synapse/DW, or GCP Big Query.
· Deploying machine learning models into production using Azure ML, SageMaker, TensorFlow, or Keras.
Soft Skills / What Gets the Win:
• Thirst for continuous learning and making technological improvements across all data lifecycle stages.
• Great communication skills with stakeholders and ability to lead large organizations through influence.
• Desire to grow with the team; senior level roles will do some architecture and design their own systems.
• Prefer candidates from IT companies or ones using the latest innovations and cutting edge technology.