Responsibilities

• Design, develop, and maintain scalable data pipelines using Spark, Hive, and Airflow
• Develop and deploy data processing workflows on the Databricks platform
• Develop API services to facilitate data access and integration
• Create interactive data visualizations and reports using AWS QuickSight
• Builds required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies
• Monitor and optimize the performance of data infrastructure and processes
• Develop data quality and validation jobs
• Assembles large, complex sets of data that meet non-functional and functional business requirements
• Write unit and integration tests for all data processing code
• Work with DevOps engineers on CI, CD, and IaC

Requirements

• Bachelor's degree
• 7+ years of hands-on software development experience
• 4+ years of experience building data pipelines using Python, Java, and cloud technologies, with hands-on experience leveraging Spark and Hive for large-scale data processing.
• Candidate must be able to obtain and maintain a Public Trust clearance
• Candidate must reside in the US, be authorized to work in the US, and work must be performed in the US
• Must have lived in the US 3 full years out of the last 5 years

Preferred

• Experience building job workflows with the Databricks platform
• Strong understanding of AWS products including S3, Redshift, RDS, EMR, AWS Glue, AWS Glue DataBrew, Jupyter Notebooks, Athena, QuickSight, EMR, and Amazon SNS
• Familiar with work to build processes that support data transformation, workload management, data structures, dependency and metadata
• Experienced in data governance process to ingest (batch, stream), curate, and share data with upstream and downstream data users.
• Experienced in data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up.
• Demonstrated understanding using software and tools including relational NoSQL and SQL databases including Cassandra and Postgres; workflow management and pipeline tools such as Airflow, Luigi and Azkaban; stream-processing systems like Spark-Streaming and Storm; and object function/object-oriented scripting languages including Scala, C++, Java and Python.
• Familiar with DevOps methodologies, including CI/CD pipelines (Github Actions) and IaC (Terraform)
• Ability to obtain and maintain a Public Trust; residing in the United States
• Experience with Agile methodology, using test-driven development.