DataOps Engineer
ADVIT Software Solutions · Hyderabad, India
Owned and optimized GitLab CI/CD pipelines for PySpark ETL jobs on AWS, ensuring reliable automated deployments of pipelines handling sensitive financial datasets for fraud scoring and customer risk profiling. Built and maintained batch and streaming data pipelines with integrated quality checks and alerting, enhancing data reliability for high-stakes financial reporting and transaction reconciliation. Automated operational tasks using Python and Shell scripts, reducing manual intervention and improving pipeline robustness during volatile market data surges. Designed and optimized data transformation workflows using DBT and Snowflake, improving query performance and cost efficiency for financial analytics teams analyzing portfolio performance. Collaborated cross-functionally to integrate infrastructure-as-code and monitoring solutions, strengthening pipeline observability and operational excellence in finance data processing. Acted as a key point of contact, providing stakeholder management across development, operations, and business teams, ensuring clear communication, managing requirements, and coordinating priorities for fintech compliance projects. Modeled and defined optimized data warehouses in Snowflake, enhancing query performance and reducing storage costs for historical financial records used in audits. Optimized Snowflake usage and performance, resulting in significant cost savings and faster data retrieval for analytics teams tracking financial metrics. Developed and maintained infrastructure-as-code templates using Terraform to provision and manage cloud resources for data pipelines, improving deployment automation and consistency in financial environments. Implemented data ingestion, transformation, and analytics workflows on Google Cloud Platform BigQuery and Cloud Storage, improving data availability and supporting scalable financial analytics.