Configuring Format Layouts with Timestamps and Fields

🏷️ Python Scripting Best Practices / Structured Production Logging

📚 All-in-One📖 Theory Only💻 Code Only

🧭 Context Introduction

When running Python scripts in production environments, the default print statements quickly become unmanageable. Without structured formatting, logs lack critical context like timestamps, severity levels, and module names. Configuring format layouts allows you to control exactly how each log entry appears, making it easier to search, filter, and debug issues across distributed systems.

⚙️ Why Format Layouts Matter

Timestamps provide chronological context for tracing events and identifying when issues occurred.
Field labels (like log level, module name, and line number) help engineers quickly filter relevant entries.
Consistent structure enables log aggregation tools (like Elasticsearch or Splunk) to parse entries automatically.
Readability improves when every log line follows the same predictable pattern.

🛠️ Core Components of a Format Layout

A typical format layout combines several fields into a single string template:

Timestamp — Usually in ISO 8601 format for machine readability (e.g., 2025-03-15 14:30:22,456)
Log Level — Severity indicator: DEBUG, INFO, WARNING, ERROR, CRITICAL
Logger Name — Identifies which module or component generated the log
Message — The actual log content you want to record
Line Number — Helps pinpoint exact source location during debugging

📊 Common Format Layout Examples

Layout Style	Example Output	Best For
Basic Timestamp	2025-03-15 14:30:22 - INFO - Connection established	Simple scripts
Detailed Fields	2025-03-15 14:30:22,456	my_module:42
Compact Format	14:30:22	WARN
JSON Structured	{"time": "2025-03-15T14:30:22", "level": "ERROR", "module": "db_conn", "msg": "Connection failed"}	Log aggregation systems

🕵️ Building a Format Layout String

The format layout is defined using a template string with placeholder variables:

%(asctime)s — Inserts the timestamp when the log record was created
%(levelname)s — Displays the severity level in uppercase
%(name)s — Shows the logger name (often the module path)
%(message)s — Contains the actual log message you pass
%(lineno)d — Inserts the line number where the log call was made
%(funcName)s — Shows the function name where the log originated

A common production layout string looks like: %(asctime)s | %(levelname)-8s | %(name)s:%(lineno)d | %(message)s

This produces output such as: 2025-03-15 14:30:22,456 | INFO | app.main:128 | Server started successfully

⏰ Configuring Timestamp Formats

Timestamps can be customized to match your team's standards:

Default format — 2025-03-15 14:30:22,456 (ISO 8601 with milliseconds)
Date only — 2025-03-15
Time only — 14:30:22
Custom format — 15/Mar/2025:14:30:22 +0000 (common in web server logs)

Common timestamp format codes include: - %Y — Four-digit year - %m — Two-digit month - %d — Two-digit day - %H — Hour in 24-hour format - %M — Minute - %S — Second - %f — Microsecond (truncated to milliseconds in practice)

📋 Applying Format Layouts in Practice

When configuring a logger, you apply the format layout through a formatter object:

Create a formatter with your desired layout string and timestamp format
Attach the formatter to a handler (console, file, or network)
Add the handler to your logger instance

The formatter controls the final appearance of every log entry passing through that handler. Different handlers can use different formats — for example, a console handler might use a compact format while a file handler uses a detailed JSON layout.

🧪 Testing Your Format Layout

Before deploying to production, verify your layout produces readable output:

Check that timestamps display in the expected timezone (UTC is recommended for production)
Confirm that long messages wrap or truncate appropriately
Verify that special characters in messages (like quotes or newlines) are escaped correctly
Test with all log levels to ensure alignment and spacing remain consistent

🚀 Best Practices for Production Layouts

Always include timestamps in UTC to avoid timezone confusion across distributed systems
Use consistent field separators like pipes (|) or tabs for easy parsing
Keep the layout compact for high-volume logging to reduce storage costs
Include module and line numbers for faster debugging without needing source maps
Avoid sensitive data in format layouts — never include passwords, tokens, or PII in field names or messages

🔁 Summary

Configuring format layouts with timestamps and fields transforms raw log output into structured, actionable data. By carefully designing your layout string, you ensure every log entry carries the context needed for monitoring, debugging, and auditing. A well-structured format is the foundation of any reliable production logging system.

This guide shows how to control the appearance of log messages by configuring format layouts with timestamps and custom fields.

📘 Example 1: Basic timestamp with default fields

Shows how to add a timestamp to every log message using the simplest format string.

import logging

logging.basicConfig(
    format='%(asctime)s - %(message)s',
    level=logging.INFO
)

logging.info('Pipeline started')
logging.warning('Disk space low')

📤 Output: 2025-04-09 14:30:15,123 - Pipeline started
2025-04-09 14:30:15,124 - Disk space low

📘 Example 2: Custom timestamp format without milliseconds

Shows how to control the timestamp style using the datefmt parameter.

import logging

logging.basicConfig(
    format='%(asctime)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    level=logging.INFO
)

logging.info('Data extraction complete')

📤 Output: 2025-04-09 14:30:15 - Data extraction complete

📘 Example 3: Adding log level and module name fields

Shows how to include the severity level and the name of the Python file generating the log.

import logging

logging.basicConfig(
    format='%(asctime)s | %(levelname)-8s | %(name)s | %(message)s',
    datefmt='%H:%M:%S',
    level=logging.DEBUG
)

logger = logging.getLogger('etl_pipeline')
logger.debug('Connecting to database')
logger.info('Query executed successfully')

📘 Example 4: Including line number and function name for debugging

Shows how to add the exact line number and function name where the log was called, useful for troubleshooting.

import logging

logging.basicConfig(
    format='%(asctime)s | %(levelname)s | %(funcName)s:%(lineno)d | %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    level=logging.WARNING
)

def validate_data():
    logging.warning('Missing values in column "age"')

def load_results():
    logging.error('Connection timeout after 30 seconds')

validate_data()
load_results()

📘 Example 5: Custom field with process ID and thread name

Shows how to add system-level fields like process ID and thread name for multi-threaded or multi-process engineers.

import logging
import threading

logging.basicConfig(
    format='%(asctime)s | PID:%(process)d | TID:%(threadName)s | %(levelname)s | %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    level=logging.INFO
)

def worker():
    logging.info('Worker thread started processing')

thread = threading.Thread(target=worker, name='DataLoader')
thread.start()
thread.join()

logging.info('Main process finished')

📊 Format Field Reference Table

Field Code	Description	Example Output
`%(asctime)s`	Timestamp from `datefmt`	2025-04-09 14:30:15
`%(levelname)s`	Log severity level	INFO, WARNING, ERROR
`%(name)s`	Logger name (often module name)	etl_pipeline
`%(funcName)s`	Function where log was called	validate_data
`%(lineno)d`	Line number in source file	12
`%(process)d`	Operating system process ID	12345
`%(threadName)s`	Thread name	DataLoader, MainThread
`%(message)s`	The actual log message text	Pipeline started