Job Description
Job Description:
Required:
Strong verbal and written communication skills to effectively articulate messages to internal and external teams.
Hands on experience with Azure data platform stack (Azure Databricks, Azure Data Factory, Azure Data Lake storage, Azure DevOps, Azure Functions)
Experience with dimensional modeling and data warehousing concepts
Experience with designing, building, optimizing, troubleshooting end-to-end big data pipelines using structured (relational and files) and semi-structured data
Experience with building metadata driven data processing frameworks
Strong experience in Spark, SQL, python, Pyspark, shell scripting.
Ability to work in an agile team.
Ability to take ownership of a request from initial requirements, design, development and production deployment
Nice to have:
PowerShell, Scala, java
Apache Airflow
Azure EventHub’s
Apace Kafka
Streaming data
Cosmos DB/NoSQL Database