This role will be working on a staffing project for Rackspace's client
The SRE will be responsible for designing, implementing, and optimizing the infra-structure, tools and environment for Finance Data Repository (FDR). The position will involve working closely with Architects and Engineers on a day-to-day basis.
JOB DESCRIPTION
Role and Responsibilities
· Configure, and maintain Databricks clusters and workspaces
· Implement and manage access controls and security policies to protect data
· Provide technical support to Databricks engineers & other users, including troubleshooting and issue resolution
· Collaborate with data engineers to integrate Databricks with other data sources, distribution points
· Implement and enforce security policies, access controls, and encryption mechanisms
· Optimize cluster configurations for performance and cost-effectiveness
· Monitor and manage auto-scaling to ensure resources are allocated efficiently
· Monitor cluster performance, resource utilization, and troubleshoot issues to ensure optimal performance
· Perform configuration management, operations using Terraform (Infra As Code)
· Define DevSecOps strategy for Platform and application
· Evaluate the deployment CICD pipelines and apply techniques to eliminate failed attempts and troubleshoot problems
· Set up a highly automated, easy-to-use DevSecOps pipeline for data workload to meet the necessary enterprise and industry standards
Requirements
· Strong background in Databricks implementation, with experience architecting complex data systems in large-scale environments
· Experience in successful delivery of complex data related projects end to end.
· Experience in Agile and DevSecOps delivery, quality practices, techniques and tools required data platforms
· Experience with cloud-based technologies
· Experience with JIRA, Confluence, Github, Bitbucket and other similar tools