What Data Engineering Actually Is

Why This Matters

Data engineering is the discipline of designing, building, and maintaining the systems that collect, store, transform, and deliver data. Without data engineers, data scientists have no clean data, analysts have no dashboards, and executives have no reports.

The Data Value Chain

Raw Sources -> Ingestion (Fivetran, Airbyte, Kafka) -> Storage (S3, Delta Lake, Snowflake) -> Transformation (dbt, Spark) -> Orchestration (Airflow) -> Consumers (BI, ML, APIs)

What Data Engineers Actually Do

Build/maintain ETL pipelines (35%), Debug data quality issues (20%), Optimize query performance (15%), Infrastructure management (15%), Code reviews and documentation (10%), Meetings and planning (5%).

DE vs Software Engineering

Backend engineers optimize for request latency; data engineers optimize for data throughput. Backend failures cause visible errors (500s); data failures cause silent wrong data in dashboards.

Code Example

A data engineer's typical morning

import subprocess

Check if overnight ETL completed

result = subprocess.run(
["airflow", "tasks", "list", "daily_etl", "--tree"],
capture_output=True, text=True
)
print(result.stdout)

Key metrics to verify:

- rows_extracted = 1.2M

- rows_transformed = 1.18M

- rows_loaded = 1.18M

- Data quality check pass rate = 99.7%

Best Practices

• Understand the distinction between DE and adjacent roles
• Know the components of a modern data stack
• Be able to trace data from source to consumption
• Explain why data quality matters more than data quantity

Interview Tips

• Draw the data flow architecture
• Discuss trade-offs between tools
• Mention monitoring and failure handling

The Modern Data Stack

Layer Architecture

Sources (MySQL, APIs, logs) -> Ingestion (Fivetran, Airbyte) -> Storage (Snowflake, S3+Delta) -> Transformation (dbt, Spark) -> Orchestration (Airflow, Dagster) -> Consumption (Looker, Tableau, Jupyter)

Tool Selection

Managed vs Self-Hosted: Fivetran vs Airbyte (budget vs control), Snowflake vs S3+Delta (team skills), dbt Cloud vs dbt Core (scale needs), Astronomer vs Airflow (compliance)

Decision Framework

Evaluate: Team Skills, Data Volume, Latency Requirements, Budget, Vendor Lock-in. Start with managed services; move to self-hosted only when necessary.

Code Example

Modern data stack example

stack = {
"ingestion": "Airbyte", # Open-source, self-hosted
"storage": "Snowflake", # Managed, scales easily
"transformation": "dbt", # SQL-based, version controlled
"orchestration": "Airflow", # Workflow scheduling
"visualization": "Looker" # BI dashboards
}

Best Practices

• Choose tools based on team skills and budget
• Prefer managed services for faster time-to-value
• Plan for vendor lock-in risks
• Document tool selection decisions

Interview Tips

• Explain why you'd choose one tool over another
• Discuss migration strategies between tools
• Know the trade-offs of managed vs self-hosted