Available for opportunities

Hi, I'm Qadir

Senior Data & Backend Engineer at JPMorgan Chase. Crafting high-performance distributed systems and real-time microservices, data pipelines that process petabytes of data daily.

View My Work Download Resume

01

About Me

Building the future of finance

Contribution Activity

Last 16 weeks

With 8+ years of experience in software engineering, I specialize in designing and building distributed data systems and high-throughput backend services at enterprise scale.

At JPMorgan Chase, I lead the architecture and development of real-time data pipelines processing billions of financial events daily, powering critical risk analytics, trading systems, and regulatory reporting infrastructure.

I'm passionate about system design, data engineering, and building resilient architectures. When I'm not coding, you'll find me writing technical blogs, contributing to open-source, mentoring engineers, or diving deep into distributed systems research.

0

Years Exp

0

Projects

0

Team Size

0

B+ Events/Day

Location

New York, NY

Education

MS CS, Columbia University

Languages

English, Hindi, Tamil

Publications

5 papers, 1 patent

02

Experience

2021 — Present

Senior Data & Backend Engineer

JPMorgan Chase

Lead architect for real-time data pipelines processing 3B+ events/day. Designed event-driven microservices for risk analytics. Reduced data latency from hours to seconds. Manage 15 engineers.

KafkaFlinkJavaK8sDatabricks

2018 — 2021

Backend Engineer

Amazon Web Services

Built core services for DynamoDB handling millions of req/sec. Implemented auto-scaling reducing costs by 40%.

GoPythonDynamoDBLambda

2016 — 2018

Software Engineer

Google

Data Infrastructure team. Built internal ETL frameworks. Optimized MapReduce jobs achieving 60% improvement.

JavaBigQueryDataflowC++

03

Skills & Expertise

Languages

Python95%

Java92%

Go85%

SQL / Scala90%

Data Engineering

Apache Kafka96%

Spark / Databricks92%

Flink / Snowflake85%

Airflow / dbt88%

Backend & Cloud

Spring Boot / gRPC94%

Kubernetes / Docker95%

AWS / GCP / Azure93%

Terraform / Pulumi88%

Databases

PostgreSQLMongoDBRedisCassandraSnowflakeElasticsearchClickHouseNeo4jDynamoDBOracle

Practices

System DesignCI/CDTDDEvent SourcingCQRSSREGraphQLOpenTelemetry

Tools & Observability

GrafanaPrometheusDatadogJaegerArgoCDGitHub ActionsJenkinsGit

04

Featured Projects

A selection of impactful projects I've designed, built, and shipped at scale.

ProductionLead

Real-Time Risk Analytics Pipeline

Event-driven architecture processing 3B+ financial events/day with sub-second P99 latency and exactly-once semantics.

Kafka·Flink·Java·Redis·K8s

< 200ms P9999.99% uptime

Production

Distributed Microservices Platform

Service mesh managing 200+ microservices with canary deployments and distributed tracing across multi-region.

Go·Istio·gRPC·AWS·Terraform

200+ servicesMulti-region

ProductionAward

Enterprise Data Lakehouse

Unified lakehouse on Databricks + Snowflake serving 500+ analysts. ACID transactions on petabyte-scale. JPM Tech Innovation Award.

Spark·Iceberg·Databricks·Snowflake·dbt

2.5 PB500+ users

Open Source

Observability & Alerting Framework

ML-powered anomaly detection. 10K+ GitHub stars. Reduced false alarms by 85% across 300+ services.

Python·OpenTelemetry·Grafana·ML·Go

10K+ stars2.1K forks

05

Blog & Publications

Thoughts on distributed systems, data engineering, and building at scale.

Distributed SystemsJan 15, 2025

Processing 3B Events/Day: Lessons from Financial Services

A deep dive into how we architected Kafka + Flink pipelines at JPMorgan to handle extreme throughput with exactly-once guarantees.

12 min readRead →

Data EngineeringDec 3, 2024

Why We Chose Iceberg Over Delta for Our Petabyte-Scale Lakehouse

An honest comparison of Apache Iceberg vs Delta Lake based on our real-world migration of 2.5PB of financial data.

9 min readRead →

Distributed SystemsOct 20, 2024

Exactly-Once Semantics in Kafka: The Complete Practical Guide

Everything I learned about implementing exactly-once processing in production, including the pitfalls that aren't in the docs.

15 min readRead →

CareerSep 8, 2024

From IC to Staff: What Nobody Tells You About the Transition

My journey from Senior to Staff Engineer at JPMorgan — the skills gaps, the mindset shift, and what I'd do differently.

8 min readRead →

Data EngineeringAug 1, 2024

ClickHouse vs Elasticsearch for Real-Time Analytics: A Benchmark

We benchmarked both on 100M rows of financial trade data. The results surprised everyone on the team.

11 min readRead →

CareerJun 15, 2024

The Hidden Cost of Maintaining an Open Source Project with 10K Stars

What they don't tell you about managing a popular OSS project: issues, PRs, burnout, and how I learned to set boundaries.

7 min readRead →

06

Case Studies

JPMorgan ChaseRisk AnalyticsReal-Time

Reducing Risk Calculation Latency from 4 Hours to Under 200ms

The risk analytics team was running batch jobs every 4 hours to calculate VaR and stress tests across the firm's trading portfolio. This created unacceptable gaps in risk visibility. We redesigned the entire pipeline as a real-time stream processing system.

99.99%

Latency reduction

3B+

Events processed daily

$0

Data loss incidents

40%

Infra cost savings

KafkaFlinkJavaRedisKubernetesGrafana

AWSDynamoDBAuto-Scaling

Optimizing DynamoDB Costs by 40% Without Sacrificing Performance

DynamoDB costs were growing unsustainably with traffic. Through intelligent partitioning, adaptive capacity mode, and read-replica optimization, we cut costs dramatically while maintaining sub-10ms P99 reads.

$2.1M

Annual savings

8ms

P99 read latency

99.999%

Availability

3x

Throughput increase

GoDynamoDBCloudWatchTerraformLambda

07

Achievements

JPM Tech Innovation Award

Enterprise data lakehouse that transformed analytics across the firm.

2023

AWS Solutions Architect Pro

Professional certification in distributed systems on AWS.

2022

Kafka Summit Speaker

"Processing 3B Events/Day: Lessons from Financial Services."

2023

US Patent Holder

"Real-Time Financial Risk Calculation Using Stream Processing."

2023

Open Source Maintainer

Contributor to Kafka, Spark, Flink. Maintainer of 3 observability tools.

Ongoing

5 Published Papers

VLDB, SIGMOD, and ICDE publications on stream processing and data systems.

2019–2024

08

Get In Touch

Prefer email, WhatsApp, or a quick form? I respond within 24 hours.

Email

Qadir.dev@email.com

+1 (234) 567-890

linkedin.com/in/Qadir

GitHub

github.com/Qadir-dev