4) Technical and professional questions (the real filter)
Technical rounds for a Data Engineer in the US are rarely “trivia night.” They’re closer to: can you design a pipeline that won’t embarrass the company, and can you debug it when it inevitably misbehaves?
You’ll see SQL, orchestration, distributed processing, modeling, and cloud. If the role leans toward ETL Developer or Data Pipeline Engineer work, expect deeper questions on incremental loads, CDC, and scheduling. If it’s closer to a Data Platform Engineer or Data Infrastructure Engineer seat, expect more on reliability, IAM, and cost controls.
Q: Walk me through how you’d design an incremental load for a large fact table.
Why they ask it: They want to see if you understand watermarks, idempotency, and late data.
Answer framework: WILD: Watermark, Idempotency, Late arrivals, Downstream impact.
Example answer: “I start by choosing a reliable watermark—event time if it’s trustworthy, otherwise ingestion time with a lag window. I make the load idempotent by writing to a staging area and merging/upserting based on a natural key plus version or updated_at. For late arrivals, I reprocess a sliding window (say 3–7 days) and track corrections. Then I communicate downstream expectations: which partitions can change and how consumers should handle restatements.”
Common mistake: Saying “just do upserts” without addressing late-arriving data and restatements.
Q: Here’s a SQL problem: deduplicate events and keep the latest per user_id, then compute daily active users. How would you write it and make it fast?
Why they ask it: They’re testing window functions, partitioning logic, and performance instincts.
Answer framework: Correctness first, then performance: window function + filter, then indexing/partition pruning.
Example answer: “I’d use a CTE with row_number() over (partition by user_id, event_date order by event_ts desc) to keep the latest event per user per day, then aggregate count(distinct user_id) by date. For speed, I’d ensure the table is partitioned by date and clustered/sorted by user_id or event_ts depending on the warehouse. I’d also avoid count(distinct) on raw events if we can pre-deduplicate into a daily user table.”
Common mistake: Writing a query that works on 1M rows but explodes cost/time on 1B rows.
US interviews often include Spark because it’s a common denominator for Big Data Engineer roles.
Q: In Spark, what causes shuffles, and how do you reduce them?
Why they ask it: They want to know if you can control distributed cost, not just call APIs.
Answer framework: Identify–Mitigate–Validate: name shuffle triggers, mitigation tactics, and how you confirm improvements.
Example answer: “Shuffles happen on wide transformations like joins, groupBy, distinct, and repartition. I reduce them by filtering early, selecting only needed columns, using broadcast joins when one side is small, and choosing partition keys that match downstream operations. I validate with the Spark UI: stage time, shuffle read/write, and skew indicators. If skew is the issue, I’ll use salting or skew hints depending on the Spark version.”
Common mistake: Saying “cache it” as a universal fix.
Orchestration is another favorite. Airflow shows up constantly in US job posts on LinkedIn Jobs and Indeed.
Q: How do you design Airflow DAGs for reliability and maintainability?
Why they ask it: They’re testing whether your orchestration scales beyond one-off DAGs.
Answer framework: DAG as product: interfaces, retries, observability, and ownership.
Example answer: “I keep tasks small and idempotent, push heavy logic into versioned code, and use clear SLAs and retries with backoff. I standardize sensors and external dependencies so we don’t create deadlocks, and I add data-quality checks as first-class tasks. For maintainability, I use consistent naming, shared operators/hooks, and I document runbooks for common failures. The goal is that on-call can fix issues without reading my mind.”
Common mistake: Building monolithic DAGs with hidden side effects and no runbook.
Cloud specifics matter in the US because many teams are all-in on AWS, GCP, or Azure.
Q: If you’re on AWS, how would you secure S3 data used by analytics while keeping it usable?
Why they ask it: They want practical security: least privilege, encryption, and auditability.
Answer framework: CIA + Audit: confidentiality, integrity, availability, plus logging.
Example answer: “I’d enforce encryption at rest with SSE-KMS and restrict access via IAM roles, not shared keys. I’d use bucket policies to block public access and require TLS, and I’d separate raw/curated zones with different permissions. For auditability, I’d enable CloudTrail data events for S3 and log access patterns. If we have PII, I’d add column-level controls in the warehouse and consider tokenization before data lands in broadly accessible layers.”
Common mistake: Treating security as ‘the security team’s problem’ instead of designing it into the pipeline.
Data modeling is where many candidates get exposed. US teams often want you to support analytics quickly, which means you need a point of view.
Q: How do you choose between a star schema, a wide table, and a Data Vault approach?
Why they ask it: They’re testing whether you can match modeling style to usage and change rate.
Answer framework: Consumers–Change–Cost: who queries, how often definitions change, and what it costs to maintain.
Example answer: “If the primary consumers are BI tools and analysts, a star schema usually wins for clarity and performance. If the use case is a single product surface with stable definitions, a wide table can be pragmatic—if you control governance and avoid metric drift. If the sources are messy and changing, and we need auditability and historization, Data Vault can help, but it’s heavier and needs strong conventions. I pick the simplest model that still survives change.”
Common mistake: Declaring one modeling style as universally ‘best.’
Here’s an insider question that shows up when teams have been burned by “works on my machine” pipelines.
Q: What data quality checks do you implement, and where do they live (pipeline vs. warehouse vs. BI)?
Why they ask it: They want to see if you understand layered quality and ownership boundaries.
Answer framework: Layered defenses: ingestion checks, transformation tests, semantic checks.
Example answer: “At ingestion I validate schema, null rates for key fields, and freshness. During transformation I add unit-like tests for joins and uniqueness, plus reconciliation checks against source counts when possible. At the semantic layer I validate business rules—like ‘paid orders must have a payment timestamp’—and I publish those checks as visible monitors. I prefer checks close to where the data is produced, but I’ll also add consumer-facing alerts so issues are caught fast.”
Common mistake: Only checking for nulls and calling it ‘data quality.’
US companies also care about compliance, especially around personal data. Even if you’re not in healthcare, you’ll get asked about privacy basics.
Q: How do you handle PII in pipelines, and what US regulations or standards do you consider?
Why they ask it: They’re testing whether you’ll accidentally create a compliance incident.
Answer framework: Identify–Minimize–Control–Prove: classify PII, reduce exposure, enforce controls, keep evidence.
Example answer: “First I classify fields and tag datasets so we know what’s sensitive. I minimize exposure by not copying raw PII into broad analytics layers unless there’s a clear need, and I use masking or tokenization where possible. Access is role-based with least privilege, and I log access for audits. Depending on the business, I’m mindful of frameworks like SOC 2 expectations and privacy laws like CCPA/CPRA in California; the practical outcome is the same: tight access, clear retention, and traceable lineage.”
Common mistake: Saying “we just put it in a private bucket” without access controls, retention, or audit trails.
Now the “tool fails” question—because it will.
Q: A critical pipeline fails during month-end close and dashboards are wrong. What do you do in the first 60 minutes?
Why they ask it: They want incident leadership: triage, communication, and containment.
Answer framework: Triage–Contain–Communicate–Recover (TCCR).
Example answer: “First I confirm impact: which tables and dashboards are affected and whether we have partial loads. I pause downstream jobs to prevent bad data from spreading and I roll back or mark the affected partitions as invalid. In parallel I post an incident update with ETA ranges, not guesses, and I pull logs to find the failure point—credentials, schema drift, upstream delay, or compute exhaustion. Once recovered, I backfill with validation checks and write a short postmortem with one or two concrete prevention actions.”
Common mistake: Going silent while debugging, letting stakeholders discover the issue themselves.
Finally, expect at least one question about cost. US teams feel cloud bills immediately.
Q: How have you reduced data platform cost without hurting reliability?
Why they ask it: They want proof you can operate responsibly at scale.
Answer framework: Measure–Target–Change–Verify: baseline, pick biggest drivers, implement, confirm.
Example answer: “I started by attributing cost to workloads—who runs what, how often, and how expensive. The biggest wins were reducing unnecessary full refreshes, tightening partition filters, and right-sizing Spark clusters with autoscaling. We also introduced lifecycle policies for raw data and moved some infrequent queries to cheaper storage/compute patterns. We verified savings by tracking cost per pipeline run and cost per query, not just the monthly bill.”
Common mistake: Cutting cost by turning off monitoring or reducing retries—saving dollars while increasing incidents.