🎯 Goal
Customize dbt‘s logging so that:
- Logs go to the console as usual.
- You capture warnings and errors in a file.
- You can later plug in external destinations (like Slack, Datadog, etc.).
🧱 Step-by-step Implementation
✅ Step 1: Create a custom log_utils.py in your dbt project
You can place this file in a folder like utils/log_utils.py.
# utils/log_utils.py
import logging
import os
def setup_dbt_custom_logging():
logger = logging.getLogger("dbt")
logger.setLevel(logging.DEBUG) # dbt uses DEBUG internally
# Remove default handlers to prevent duplicate logs
if logger.hasHandlers():
logger.handlers.clear()
# === Console Handler (default) ===
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(logging.Formatter(
fmt="%(asctime)s - %(levelname)s - %(message)s",
datefmt="%H:%M:%S"
))
logger.addHandler(console_handler)
# === File Handler for WARN+ ===
warning_handler = logging.FileHandler("logs/dbt_warnings.log")
warning_handler.setLevel(logging.WARNING)
warning_handler.setFormatter(logging.Formatter(
fmt="%(asctime)s - %(levelname)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S"
))
logger.addHandler(warning_handler)
# === Optional: File handler for DEBUG ===
debug_handler = logging.FileHandler("logs/dbt_debug.log")
debug_handler.setLevel(logging.DEBUG)
debug_handler.setFormatter(logging.Formatter(
fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S"
))
logger.addHandler(debug_handler)
✅ Step 2: Call setup_dbt_custom_logging() in a dbt macro
Unfortunately, you can’t call raw Python inside Jinja/macros, so the best place to integrate this is:
- In a custom materialization
- Or in a dbt Python model (if using dbt with Python support, e.g. dbt-snowflake, dbt-databricks)
Example in a Python model (models/logging_example.py):
import pandas as pd
from utils.log_utils import setup_dbt_custom_logging
import logging
def model(dbt, session):
setup_dbt_custom_logging()
logger = logging.getLogger("dbt")
logger.info("Starting model execution")
logger.warning("This is a warning example")
logger.error("This is an error example")
df = pd.DataFrame({"x": [1, 2, 3]})
return df
You’ll now have 3 destinations:
- Console (INFO+)
logs/dbt_warnings.log(WARNING+)logs/dbt_debug.log(all levels)
📌 Key Concepts
| Concept | Purpose |
|---|---|
| Single Logger | Ensures central management of logs (logging.getLogger("dbt")) |
| Multiple Handlers | Send logs to different destinations with filters |
| Formatter | Customize output format per handler |
| Thread-safe | Python logging is safe for concurrent dbt runs |
| No duplicate logs | Handlers cleared before attaching custom ones |
✅ Benefits
- Keeps dbt’s native logging intact.
- Adds fine-grained control over log destinations.
- Avoids polluting macros or model code with log logic.
- Easily extendable to:
- Send alerts via HTTP/Slack.
- Push logs to tools like ELK/Datadog.
- Rotate logs or timestamp filenames.