Python for Data Engineers — What You Actually Need

The Data Engineer's Toolkit

Data engineers use Python for: file I/O (csv, json, parquet), database queries (sqlalchemy), API calls (requests), data processing (pandas, polars), and pipeline automation.

File Operations

Use pathlib for path manipulation, csv.DictReader for CSV files, json.loads/dumps for JSON. Always handle encoding and large files with streaming.

Control Flow Patterns

Process files with error handling, conditional transformations, and retry logic. Use try/except around all I/O operations.

Code Example

from pathlib import Path
import csv, json

Read CSV efficiently

with open("sales.csv") as f:
reader = csv.DictReader(f)
for row in reader:
process(row["amount"]) # Note: amount is a string!

Write JSON output

data = {"rows_processed": 1500, "errors": 3}
Path("output/summary.json").write_text(json.dumps(data, indent=2))

Path manipulation (never hardcode paths)

base = Path("/data/warehouse")
input_file = base / "raw" / "2024" / "01" / "sales.parquet"
print(input_file.exists()) # True/False

Best Practices

• Always use try/except around file I/O
• Use pathlib instead of string concatenation
• Log every step for debugging
• Validate data types after reading
• Handle None/missing values explicitly

Interview Tips

• Be ready to write Python on a whiteboard
• Know list comprehensions and generators
• Explain error handling patterns

Functions You'll Actually Write

Pure Transformation Functions

Write functions with clear inputs and outputs, no side effects. Easy to test, compose, and debug. Add type hints and docstrings.

Data Validation Functions

Return (is_valid, error_message) tuples. Check required fields, data types, value ranges, and format constraints.

Code Example

Pure function — easy to test

def normalize_email(email: str) -> str:
"""Lowercase and strip whitespace from email."""
if not email:
return None
return email.strip().lower()

Validation function

def validate_order(order: dict) -> tuple[bool, str | None]:
required = ["order_id", "customer_id", "amount", "timestamp"]
missing = [f for f in required if f not in order]
if missing:
return False, f"Missing fields: {missing}"
try:
amount = float(order["amount"])
except (ValueError, TypeError):
return False, f"Invalid amount: {order['amount']}"
if amount < 0:
return False, f"Negative amount: {amount}"
return True, None

Best Practices

• Keep functions under 30 lines
• Use type hints for clarity
• Write docstrings for all public functions
• Test edge cases (None, empty, negative)

Interview Tips

• When asked to write a function, handle edge cases first
• Mention how you'd test it
• Discuss error handling strategy