Data Engineer 4
Join our dynamic team at our energy partner as a Data Engineer 4, where you will play a pivotal role in harnessing the power of data to optimize operations, enhance decision-making processes, and drive innovation in the rapidly evolving energy sector. As a seasoned professional, you will leverage your expertise to design, implement, and maintain robust data pipelines, ensuring the seamless flow of information critical to powering the future of sustainable energy solutions
· Provides technical direction, guides the team on key technical aspects and responsible for product tech delivery
· Lead the Design, Build, Test and Deployment of components
· Where applicable in collaboration with Lead Developers (Data Engineer, Software Engineer, Data Scientist, Technical Test Lead)
· Understand requirements / use case to outline technical scope and lead delivery of technical solution
· Confirm required developers and skillsets specific to product
· Provides leadership, direction, peer review and accountability to developers on the product (key responsibility)
· Works closely with the Product Owner to align on delivery goals and timing
· Assists Product Owner with prioritizing and managing team backlog
· Collaborates with Data and Solution architects on key technical decisions
· The architecture and design to deliver the requirements and functionality
Core Experience and Abilities
· Power Systems engineering experience
· DER Dispatch experience
· Automated fault analysis systems experience
· Ability to perform hands on development and peer review for certain components / tech stack on the product
· Standing up of development instances and migration path (with required security, access/roles)
· Develop components and related processes (e.g. data pipelines and associated ETL processes, workflows)
· Lead implementation of integrated data quality framework
· Ensures optimal framework design and load testing scope to optimize performance (specifically for Big Data)
· Supports data scientist with test and validation of models
· Performs impact analysis and identifies risk to design changes
· Ability to build new data pipelines, identify existing data gaps and provide automated solutions to deliver analytical capabilities and enriched data to applications
· Ensures Test Driven development
· Experience leading teams to deliver complex products
· Strong technical skills and communication skills
· Strong skills with business stakeholder interactions
· Strong solutioning and architecture skills
· Experience building real time data ingestion streams (event driven)
· Ensure data security and permissions solutions, including data encryption, user access controls and logging
Core Technical Skills
· Experience with native AWS technologies for data and analytics such as Athena, S3, Lambda, Glue, EMR, Kinesis, SNS, CloudWatch, etc.
· Tools and Languages such as Django, Python, Java, Scala, Pandas
· Infrastructure as Code technology such as Terraform
· AWS services such as S3, EMR, Glue, Lambda, Athena, Kinesis, EC2, SNS, SQS, Cloudwatch
· Experience with databases such as Redshift, Document DB, DynamoDB and Mongo DB
· Experience transitioning on premise big data platforms into cloud based platforms such as AWS
Additional Technical Skills (nice to have, but not required for the role)
· Hadoop platform (Hive; HBase; Druid)
· Spark
· PySpark
· SQL
· Workflow Automation
· DevOps pipeline (CI/CD); Bitbucket; Concourse
· API frameworks
