📖 Predictive Sourcing Analytics Playbook

Step 2: Data Engineering & Feature Pipeline

A forecasting model is only as good as its underlying data. Sourcing daily data across global exchanges requires a robust data engineering pipeline to handle asynchronous trading calendars, reporting lags, and feature scaling.

1. Sourcing and verifying primary data

Commodity pricing data must be gathered from official exchanges. For base metals like Copper, Aluminum, and Lead, the London Metal Exchange (LME) is the primary global benchmark. Since trading calendars differ between exchanges (e.g., LME in London, SHFE in Shanghai, COMEX in New York), cross-verification with premium institutional market databases is necessary to ensure data sanctity.

2. Handling missing values and data gaps

Financial time series contain natural gaps (weekends and national holidays). Simply dropping these rows disrupts sequential time series models like ARIMA.

• Linear Interpolation: Calculates intermediate values based on the adjacent active trading days, preserving continuous sequential paths.
Y_t = Y_{t-1} + \left( \frac{Y_{t+k} - Y_{t-1}}{k+1} \right)
• Forward Fill: Carries forward the last known price. Useful for exchange inventory levels (stocks) since stock levels remain unchanged during holidays.

3. Variable selection and multicollinearity

To explain metal price volatility, we collect macroeconomic indicators (USD Index, DXY), energy costs (Brent Crude), exchange rates (USD/INR), and regional exchange prices (SHFE, COMEX).

Multicollinearity (high correlation between independent variables) can inflate coefficients and destabilize regression models. We drop redundant features (e.g., keeping only one benchmark treasury yield instead of several closely correlated tenors).

4. Scaling and Normalization

Independent variables exist on vastly different scales (e.g., stock levels in millions of tons, crude oil in dollars, interest rates in percentages). Distance-based and gradient-descent algorithms (like SVM and MLP Neural Networks) will prioritize larger-scale features if not scaled. We apply Standardization:

This centers features around a mean of 0 with a standard deviation of 1, preventing numerical instability during training.

Step 3: Exploratory Data Analysis (EDA)

EDA identifies statistical characteristics, variance, and seasonal trends to guide model selection and highlight key market relationships.

1. Descriptive Statistics & Volatility Signals

Analyzing the range, mean, and standard deviation reveals the historical volatility of each target commodity.

2. Distribution Analysis: Skewness & Kurtosis

Normal distributions have a kurtosis of 3. Commodities like Copper (4.11) and Aluminum (4.98) display high kurtosis (leptokurtic). This indicates heavy-tailed distributions with a high frequency of extreme outliers (sudden price spikes). Model validation must prepare for these extreme shifts.

3. Correlation Heatmaps & Cross-Market Indicators

Heatmaps reveal how commodities move in relation to exogenous variables:

• Energy correlation: Brent Crude Oil correlates strongly with Copper and Lead (0.68), reflecting the impact of logistics and refining energy costs.
• Exchange rate correlation: The USD Index (DXY) correlates negatively with global commodities, as a stronger dollar makes dollar-denominated metals more expensive in other currencies.
• Arbitrage correlation: LME Aluminum has a 0.89 correlation with Shanghai Futures Exchange (SHFE) Copper, identifying a strong cross-commodity, cross-exchange market link.

4. Seasonality and Day-of-Week Evaluation

Analyzing returns by day of the week helps identify recurring short-term trends:

• Monday: Historically has the highest price drops, but exhibits severe volatility (high risk/reward).
• Thursday: Offers balanced, moderate savings with lower volatility risk.
• Wednesday: Displays the lowest price movements (safest day to buy, but with minimal potential savings).

Step 4: Time Series & ARIMA/SARIMA Modeling

Classical time series models rely on historical lags and residuals. To model these patterns, we must understand stationarity and lag structures.

1. Stationarity Testing: Augmented Dickey-Fuller (ADF) Test

A stationary series has a constant mean, variance, and autocorrelation over time. If a series is non-stationary, models like ARIMA will yield unreliable, spurious regressions. The ADF test evaluates:

If the test statistic is more negative than the critical value (p-value < 0.05), we reject the null hypothesis and confirm stationarity.

Differencing to Achieve Stationarity

Daily commodity prices are almost always non-stationary due to trend structures (Copper p-value: 0.695, Aluminum p-value: 0.434). We apply Differencing (d=1):

Subtracting the previous value removes trends, stabilizing the mean.

2. Reading ACF and PACF Plots

After differencing, Autocorrelation (ACF) and Partial Autocorrelation (PACF) plots help identify the lag structures for the model:

• Autocorrelation Function (ACF): Measures correlation between $Y_t$ and lag $Y_{t-k}$, including indirect effects. Cutoffs in ACF help identify the Moving Average (MA) order q.
• Partial Autocorrelation Function (PACF): Measures correlation between $Y_t$ and lag $Y_{t-k}$, controlling for intermediate lags. Cutoffs in PACF identify the Autoregressive (AR) order p.

3. Inside the SARIMA Model

SARIMA combines lag parameters and seasonal adjustments:

• p (AR order): Number of historical price lags used to predict the current price.
• d (Differencing order): Number of times differencing is applied to achieve stationarity.
• q (MA order): Number of historical residual forecast error lags utilized.
• P, D, Q: Seasonal AR, seasonal differencing, and seasonal MA orders.
• s: Seasonal period length (e.g., 12 for monthly cycles).

4. Model Selection Criteria: AIC & BIC

To select the best model orders, we balance model fit against complexity (preventing overfitting) using AIC (Akaike Information Criterion):

where $k$ is the number of parameters and $\hat{L}$ is the maximum likelihood. The model order with the lowest AIC is selected (e.g., Copper's optimal model: `SARIMA(2,0,4)(0,0,2)[12]`, AIC: 1775.78).

Step 5: Machine Learning & Ensemble Regressors

While classical time series models focus on linear historical lag structures, machine learning regressors are designed to capture complex, non-linear relationships across multiple market factors.

3. Advanced Non-Linear Models: SVM & MLP Neural Networks

• Support Vector Regression (SVR): Maps input data into higher-dimensional spaces using kernel functions (like RBF) to find a hyperplane that fits data within a tolerance boundary $\epsilon$. SVR is effective for high-dimensional data, but selecting optimal parameters (C, $\gamma$) is difficult.
• Multi-Layer Perceptron (MLP): A feedforward neural network that learns complex non-linear mappings through hidden layers and backpropagation. MLP is highly powerful, but prone to overfitting on noisy financial data if not constrained by early stopping.

Step 6: Training, Testing & Validation Methodology

To guarantee that a forecasting model performs well in production, validation must be structured to prevent data leakage and measure generalization.

3. Production Simulation: Out-of-Sample Rolling Validation

In production, models generate forecasts over a forward horizon (e.g. 30 days) using rolling averages of input features.

We validate this by comparing the model's 30-day predicted averages against actual monthly prices. The table below illustrates this comparison using actual sample data:

Step 7: Evaluation Metrics & Selection Criteria

To select the final production model, performance must be measured across multiple metrics. Different metrics capture different types of prediction errors.

1. Metric Definitions & Sourcing Intuition

• R-Squared (R² - Coefficient of Determination): Measures the proportion of variance in the target variable explained by the model:
  R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}
  *Sourcing Intuition:* Indicates general model fit. However, a high $R^2$ can still mask large absolute errors in volatile periods.
• Root Mean Squared Error (RMSE): Measures the standard deviation of residuals:
  \text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}
  *Sourcing Intuition:* RMSE squares the errors before averaging, penalizing larger deviations more heavily. Useful when large pricing errors carry severe financial risks.
• Mean Absolute Error (MAE): Measures the average magnitude of absolute errors:
  \text{MAE} = \frac{1}{n} \sum_{i=1}^n |y_i - \hat{y}_i|
  *Sourcing Intuition:* MAE treats all errors linearly. It represents the typical dollar-value deviation of predicted prices, making it highly intuitive for procurement managers.

2. Model Selection Benchmarks (Sample Results)

Step 8: Operational Sourcing Decision Framework

The final stage of the analytics lifecycle is deployment, converting predicted values into actionable procurement decisions.