ARIMA and advanced forecasting models

Substituting recursively the expression for the next step predictions:

This is a multiple regression with past noise as predictors.

where the lag operator is defined as $B(y_t) = y_{t-1}$. If the roots of the $\Phi^{-1}(B)$ polynomial are all outside the unit circle in the complex plane, we can write the ARMA process as per the last equation in the third column. This makes clear that an ARMA model has an equivalent AR representation.

There are several ways to transform ARMA models into a state-space form. One easy way is to define the $x_t$ and $u_t$ vectors as:

Since $y_t$ is the first element of $x_t$ at the next step, one can write an update equation for $x$

→ ARMA process can be seen as structured SS models

As we have seen in the State Space models lesson, the stability of a SS system can be linked to eigenvalues of the matrix $\tilde{A} = A-KC$​

Appropriate for non-stationary signals

The autoarima procedure implemented in major forecasting libraries takes care of the steps 3-5:

ARIMA models produce forecasts by recursively using the produced forecast as covariate. We look at one example for an ARIMA(3,1,1) model:

$\left(1-\hat{\phi}_1 B-\hat{\phi}_2 B^2-\hat{\phi}_3 B^3\right)(1-B) y_t=\left(1+\hat{\theta}_1 B\right) \varepsilon_t$​

Then we expand the left hand side to obtain:

$\left[1-\left(1+\hat{\phi}_1\right) B+\left(\hat{\phi}_1-\hat{\phi}_2\right) B^2+\left(\hat{\phi}_2-\hat{\phi}_3\right) B^3+\hat{\phi}_3 B^4\right] y_t=\left(1+\hat{\theta}_1 B\right) \varepsilon_t$​

and applying the backshift operator gives

$y_t-\left(1+\hat{\phi}_1\right) y_{t-1}+\left(\hat{\phi}_1-\hat{\phi}_2\right) y_{t-2}+\left(\hat{\phi}_2-\hat{\phi}_3\right) y_{t-3}+\hat{\phi}_3 y_{t-4}=\varepsilon_t+\hat{\theta}_1 \varepsilon_{t-1}$​

Finally, we move all terms other than y to the right hand side:

$y_t=\left(1+\hat{\phi}_1\right) y_{t-1}-\left(\hat{\phi}_1-\hat{\phi}_2\right) y_{t-2}-\left(\hat{\phi}_2-\hat{\phi}_3\right) y_{t-3}-\hat{\phi}_3 y_{t-4}+\varepsilon_t+\hat{\theta}_1 \varepsilon_{t-1}$​

The expression at time T can be written as:

$y_{T+1}=\left(1+\hat{\phi}_1\right) y_T-\left(\hat{\phi}_1-\hat{\phi}_2\right) y_{T-1}-\left(\hat{\phi}_2-\hat{\phi}_3\right) y_{T-2}-\hat{\phi}_3 y_{T-3}+\varepsilon_{T+1}+\hat{\theta}_1 \varepsilon_T$​

since we cannot observe the error at the next time, we set it to zero and replace $\varepsilon_T$ with the last observer error

$\hat{y}_{T+1 \mid T}=\left(1+\hat{\phi}_1\right) y_T-\left(\hat{\phi}_1-\hat{\phi}_2\right) y_{T-1}-\left(\hat{\phi}_2-\hat{\phi}_3\right) y_{T-2}-\hat{\phi}_3 y_{T-3}+\hat{\theta}_1 e_T$​

The successive forecast is obtained as

$\hat{y}_{T+2 \mid T}=\left(1+\hat{\phi}_1\right) \hat{y}_{T+1 \mid T}-\left(\hat{\phi}_1-\hat{\phi}_2\right) y_T-\left(\hat{\phi}_2-\hat{\phi}_3\right) y_{T-1}-\hat{\phi}_3 y_{T-2}$​

Why forecasting is important at Amazon:

General purpose regressors, like kNN, xgboost, QRFs, can be used as a input-output map.

The predictions are obtained by transforming the hidden states into contexts $\mathbf{c}[t+1: t+H]$ that are decoded and adapted into $\hat{\mathbf{y}}_{[t+1: t+H],[q]}$​

Where $h_t, y_t, \mathbf{x}^{(s)}, \mathbf{x}_t^{(h)}, \mathbf{x}_{[: t+H]}^{(f)}$ are the hidden state, the input, the static, past and future exogenous signals available at prediction time.

It’s basically a naive approach coupled with a model for the likelihood of the observations

Training (left): At each time step t, the inputs to the network are the covariates $x_{i, t}$, the target value at the previous time step $z_{i, t-1}$, as well as the previous network output $\mathbf{h}_{i, t-1}$. The network output $\mathbf{h}_{i, t}=h\left(\mathbf{h}_{i, t-1}, z_{i, t-1}, \mathbf{x}_{i, t}, \Theta\right)$ is then used to compute the parameters $\theta_{i, t}=\theta\left(\mathbf{h}_{i, t}, \Theta\right)$ of the likelihood $\ell(z \mid \theta)$, which is used for training the model parameters. Prediction (right): the history of the time series $z_{i, t}$ is fed in for $t<t_0$, then in the prediction range (right) for $t \geq t_0$ a sample $\hat{z}_{i, t} \sim \ell\left(\cdot \mid \theta_{i, t}\right)$ is drawn and fed back for the next point until the end of the prediction range $t=t_0+T$ generating one sample trace. Repeating this prediction process yields many traces representing the joint predicted distribution.

The model’s parameters are fitted maximizing the log-likelihood. The authors proposed two forms for $\ell$, for continuous and counting variables

The model is usually augmented with hand-crafted features, such as the hour of the day and past lags of the target