Senior Big Data Engineer
The Business Intelligence Department is looking to add a Senior Big Data Engineer to their current team. The position is responsible for the design, build and implementation of cloud-based analytics platform which includes an MPP Enterprise Data Warehouse and other Big Data technologies.
Essential Duties and Responsibilities:
- Designs and develops high performance distributed data warehouse, distributed analytic systems and cloud architecture.
- Develops, launches and maintains efficient and fault tolerant, batch and streaming, data pipelines (ETL/ELT) to populate databases and object stores from multiple disparate data sources.
- Performs complex data calculations through data integration tools and scripting languages.
- Designs and implements data quality metrics, standards, guidelines; automates data quality checks/routines as part of data processing frameworks; validates flow of information.
- Determines Data Warehousing and Big Data infrastructure needs, including but not limited to, automation of system builds, security requirements, performance requirements and logging/monitoring in collaboration with DevOps engineers.
- Troubleshoots complex data and performance related issues; implements adjustments, documents root cause and corrective measure; transfers knowledge to operations support team.
- Documents technical specifications and participates with peers in design and code review sessions.
- Develops complex cross application architectures in collaboration with cross functional teams.
- Stays current on the latest industry technologies, trends and strategies.
- Assists employees, vendors and other customers by answering questions related to Data Warehousing and Big Data processes, procedures and services.
- Completes work in a timely and accurate manner while providing exceptional customer service.
- Other duties as assigned.
- This position requires a minimum of eight years of progressive database development and integration experience.
- Proven understanding of logical and physical data modeling is imperative.
- Ability to translate a logical data model into a relational or non-relational solution is necessary.
- Understanding of multiple relational (RDMS) and non-relational (NoSQL) data platforms is needed.
- Expert level SQL experience is required.
- Scripting knowledge with SQL, Python, Java or R is necessary.
- Proven experience in SQL tuning, indexing, partitioning, data access patterns and scaling strategies is needed.
- Proven experience with data integrations and data processing for business intelligence and analytics workloads is required.
- Experience with AWS S3 or other distributed object stores, AWS Redshift, Elastic MapReduce a plus.
- Hands-on experience in database development using views, SQL scripts and transformations is needed.
- Proficiency with Microsoft Office, including skills with Word and Excel is necessary.
- Experience working with large complex data sets is required.
- Understanding of Software Development Life Cycle (SDLC) methodologies such as Agile and Waterfall is needed.
- Proven analytical problem solving and decision making skills are critical.
- Demonstrated ability to communicate across all levels of the organization is necessary; must be able to clearly articulate technical ideas to a non-technical audience both verbally and in writing.
- Ability to work independently and in a team is vital.
- Customer service skills including the ability to manage and respond to different customer situations while maintaining a positive and friendly attitude are essential.
- The ability to multi-task and manage multiple projects to meet various deadlines simultaneously is required.
- The ability to work efficiently and accurately under pressure, meet deadlines and present a professional demeanor is essential.
- In addition, troubleshooting and organizational skills with a can-do attitude and the ability to adjust to changing requirements are essential.
- This position requires a Bachelor’s degree in Computer Science, Computer Information Systems or related or equivalent experience.