Regression Metrics
Computes regression performance metrics including WAPE and Forecast Accuracy.
Regression Metrics
Processing
This brick evaluates the performance of a predictive model by comparing its results against the actual (real) data. It calculates a single statistical score to determine accuracy or error rates.
It works by taking two lists of numbers—the "Truth" (what actually happened) and the "Prediction" (what the model guessed)—and applying a mathematical formula to quantify the difference. This is essential for validating forecasting models, sales predictions, or any regression analysis.
Inputs
- y true
- The actual, observed values (often called the "Ground Truth"). If you provide a table or dataframe here, you must specify which column to read in the Target Column (y_true) setting.
- y pred
- The predicted values generated by your model. If you provide a table or dataframe here, you must specify which column to read in the Prediction Column (y_pred) setting. Note: This list must have the exact same number of items as
y true.
Inputs Types
| Input | Types |
|---|---|
y true |
DataFrame, ArrowTable, DataSeries, NDArray, List |
y pred |
DataFrame, ArrowTable, DataSeries, NDArray, List |
You can check the list of supported types here: Available Type Hints.
Outputs
- score
- The calculated result of the selected metric. A single floating-point number representing the error or accuracy score.
Outputs Types
| Output | Types |
|---|---|
score |
Float |
You can check the list of supported types here: Available Type Hints.
Options
The Regression Metrics brick contains some changeable options:
- Metric
- The specific mathematical formula used to calculate the score. Different metrics tell you different things about your data quality.
- Forecast Accuracy: A percentage score (0 to 1) indicating how close the predictions were to the truth. Calculated as
1 - WAPE. - Weighted Absolute Percentage Error (WAPE): The sum of absolute errors divided by the sum of actual values. Useful for data with varying scales (e.g., sales volume).
- Mean Absolute Error (MAE): The average size of the errors. It treats all errors equally.
- Mean Squared Error (MSE): The average of the squared errors. This penalizes large errors more heavily than small ones.
- Root Mean Squared Error (RMSE): The square root of MSE. It represents the standard deviation of the residuals and is in the same unit as the original data.
- R2 Score: (Coefficient of Determination) Indicates how well the predictions explain the variation in the real data. A score of 1.0 is perfect; 0.0 is equivalent to just guessing the average; negative scores indicate very poor predictions.
- Mean Absolute Percentage Error (MAPE): The average percentage difference between predicted and actual values.
- Target Column (y_true)
- If the
y trueinput is a DataFrame or Table, type the name of the specific column containing the actual values here. * Default: "y_true" - Prediction Column (y_pred)
- If the
y predinput is a DataFrame or Table, type the name of the specific column containing the predicted values here. * Default: "y_pred" - Verbose
- If enabled, detailed logs about the computation steps (including potential errors regarding column names) will be printed to the system logs.
import logging
import numpy as np
import pandas as pd
import polars as pl
import pyarrow as pa
from sklearn.metrics import (
mean_absolute_error,
mean_squared_error,
r2_score,
mean_absolute_percentage_error,
root_mean_squared_error,
)
from coded_flows.types import (
Union,
DataFrame,
ArrowTable,
DataSeries,
List,
NDArray,
Float,
Any,
)
from coded_flows.utils import CodedFlowsLogger
logger = CodedFlowsLogger(name="Regression Metrics", level=logging.INFO)
def _extract_vector(
data: Any, column_name: str, verbose: bool = False, label: str = "Input"
) -> Union[List, np.ndarray]:
"""
Extracts a 1D vector/list from various data structures (DF, Series, Arrow, List, Numpy).
"""
result = None
if isinstance(data, pd.DataFrame):
if column_name not in data.columns:
verbose and logger.error(
f"Column '{column_name}' not found in Pandas DataFrame for {label}."
)
raise ValueError(
f"Column '{column_name}' not found in Pandas DataFrame for {label}."
)
result = data[column_name].tolist()
elif isinstance(data, pl.DataFrame):
if column_name not in data.columns:
verbose and logger.error(
f"Column '{column_name}' not found in Polars DataFrame for {label}."
)
raise ValueError(
f"Column '{column_name}' not found in Polars DataFrame for {label}."
)
result = data[column_name].to_list()
elif isinstance(data, (pa.Table, pa.lib.Table)):
if column_name not in data.column_names:
verbose and logger.error(
f"Column '{column_name}' not found in Arrow Table for {label}."
)
raise ValueError(
f"Column '{column_name}' not found in Arrow Table for {label}."
)
result = data[column_name].to_pylist()
elif isinstance(data, pd.Series):
result = data.tolist()
elif isinstance(data, pl.Series):
result = data.to_list()
elif isinstance(data, np.ndarray):
result = data.flatten().tolist()
elif isinstance(data, list):
result = data
if result is None:
verbose and logger.error(
f"Unsupported data type for {label}: {type(data)}. Expected DataFrame, Series, Array, or List."
)
raise ValueError(
f"Unsupported data type for {label}: {type(data)}. Expected DataFrame, Series, Array, or List."
)
return result
def regression_metrics(
y_true: Union[DataFrame, ArrowTable, DataSeries, NDArray, List],
y_pred: Union[DataFrame, ArrowTable, DataSeries, NDArray, List],
options: dict = None,
) -> Float:
options = options or {}
verbose = options.get("verbose", True)
metric = options.get("metric", "Forecast Accuracy")
y_true_col = options.get("y_true_column", "y_true")
y_pred_col = options.get("y_pred_column", "y_pred")
score = 0.0
try:
verbose and logger.info(f"Starting regression metric computation: {metric}")
y_true_vec = _extract_vector(
y_true,
column_name=y_true_col,
verbose=verbose,
label="Real Target (y_true)",
)
y_pred_vec = _extract_vector(
y_pred, column_name=y_pred_col, verbose=verbose, label="Prediction (y_pred)"
)
y_true_np = np.array(y_true_vec, dtype=float)
y_pred_np = np.array(y_pred_vec, dtype=float)
if len(y_true_np) != len(y_pred_np):
raise ValueError(
f"Length mismatch: y_true has {len(y_true_np)} rows, y_pred has {len(y_pred_np)} rows."
)
if metric == "Mean Absolute Error (MAE)":
score = mean_absolute_error(y_true_np, y_pred_np)
elif metric == "Mean Squared Error (MSE)":
score = mean_squared_error(y_true_np, y_pred_np)
elif metric == "Root Mean Squared Error (RMSE)":
score = root_mean_squared_error(y_true_np, y_pred_np)
elif metric == "R2 Score":
score = r2_score(y_true_np, y_pred_np)
elif metric == "Mean Absolute Percentage Error (MAPE)":
score = mean_absolute_percentage_error(y_true_np, y_pred_np)
elif metric in [
"Forecast Accuracy",
"Weighted Absolute Percentage Error (WAPE)",
]:
eps = np.finfo(np.float64).eps
sum_abs_error = np.sum(np.abs(y_true_np - y_pred_np))
sum_abs_truth = np.maximum(np.sum(np.abs(y_true_np)), eps)
if metric == "Forecast Accuracy":
score = 1 - sum_abs_error / sum_abs_truth
else:
score = sum_abs_error / sum_abs_truth
verbose and logger.info(f"Computation successful. {metric}: {score}")
except Exception as e:
verbose and logger.error(f"Error computing regression metrics: {str(e)}")
raise e
score = float(score)
return score
Brick Info
- shap>=0.47.0
- scikit-learn
- numpy
- pandas
- pyarrow
- polars[pyarrow]
- numba>=0.56.0