What can be forecasted

A time series is a collection of values of a variable $x$ over time:

where $\mathcal{T}$ can be regularly or irregularly spaced. Most of the literature assumes discrete regularly sampled time series.

The single most important conditions in forecasting a time series:

In many cases, this condition holds. If it doesn’t, we can try to force it:

We have some tools that help understand whether a given time series is forecastable. These tools are based on a stochastic process representation of time series.

The joint distribution function of $\left(x_{t_1}(\omega), x_{t_2}(\omega),\dots, x_{t_s}(\omega)\right)$ can be defined as:

The above expression is known also as a copula, that is, a cumulative density function over a multivariate space.

If $F$ was known for a particular stochastic process $x_t(\omega)$, we could answer questions such as:

It turns out that this ability is pretty much all we need to know about the stochastic process.

In fact, once $F$ is known, we can generate an infinite set of realizations from the stochastic process $x_t(\omega_i) \quad i=[1, 2,...\infty]$, also called scenarios

If the future is not similar to the past, we cannot estimate $F_{t_1, t_2, \cdots, t_s}$ for arbitrary values of s. Even for a univariate stochastic process $\left\{x_t(\omega) ; t \in \mathbb{Z}\right\}, \text { where } x_t \sim N\left(\mu_t, \sigma_t^2\right)$, this requires to estimate

To reduce the number of parameters to be learned (in the example above $\{(\mu_t,\sigma_t)_{t=1}^{t_s}\}$), some regularity conditions must hold.

Let’s do a step back and try to ask a somehow simpler question:

We define the mean and (auto)covariance of a stochastic process as:

$\mu_x(t)=\mathbb{E}\left(x_t\right)$

$\gamma_x(t_1, t_2)=\gamma_x(t, t-k)=\gamma_x(k)=\mathbb{E}\left[\left(x_{t}-\mu\right)\left(x_{t -k}-\mu\right)\right]$​

Since we can only observe one realization of the stochastic process in time, estimates for mean and variance can only be done by time averaging (as opposed to sample path average).

A process is said to be stationary (in a given window) if sliding along the time axis the joint CDF doesn’t change

Additionally, a process is said to be ergodic w.r.t. one of its moment if we can estimate them by observing a sufficiently long realization of the process and if estimations converge quadratically to the true values.

In other words, imagine to observe a set of sample paths $x_j = \{x_{j,t_1},x_{j,t_2},x_{j,t_3}\dots x_T\}$. The process is ergodic if averaging over different $x_j$ gets the same results of averaging over time.

Which one of the two examples of stochastic processes above is ergodic?

Intuitively, this result tell us that if any two random variables positioned far apart in the sequence are almost uncorrelated, then some new information can be continually added so that the sample mean will approach the population mean.

By the law of large numbers, good estimates of mean and covariance for ergodic processes are:

$\hat{\mu}_T(t)=\frac{1}{T}\sum_{t}^{t+T}x_t$

$\hat{\gamma}_x(k)=\frac{1}{T}\sum_{t}^{t+T}\left(x_t-\hat{\mu}_T\right)\left(x_{t-k}-\hat{\mu}_T\right)$​

Beware! In some cases these estimations do not converge quickly

The autocorrelation function is defined as:

If we get only T timesteps to evaluate $\rho_k$, we must be careful to keep $T-k$ large enough for the estimation to be meaningful

Which is which?

The ACF function can be used to construct the following hypothesis test:

$H_0:x_t \sim i.i.d.(0, \sigma^2)$​

if a process is IID, we have that $\rho_k = 0$.

A decision rule for the hypothesis test could be:

that is, for a given lag $k$ we require the magnitude of the correlation to be less than a constant value.

How do we choose c? We can request a 95% confidence level:

Instead of testing a given lag $k$, we can test for a set of lags. For an i.i.d. process, the following random variables are approximately chi-square with K degrees of freedom.

Entropy can be used as a measure of complexity and forecastability. Entropy is a measure of information contained in a random event. Since the information contained in two independent events $x \perp y$ will sum, while their joint probability is $p(x, y) = p(x)p(y)$, the only possible form of the information is:

Entropy gives an idea of complexity of the distribution of the signal. However, entropy for these two signals will be the same:

A better measure of forecastability is approximate entropy, which computes the entropy of maximum discrepancy of segments of length m of a time series.

Some forecasting methods are extremely simple and yet surprisingly effective. They are stateless methods

This method performs well for $h=1$, or at scale which are smaller than the characteristic time scale of the signal

$\hat{y}_{t+h \mid t}=y_t$​

We just take the mean value of the time series observed so far

$\hat{y}_{t+h \mid t}=\bar{y}=\left(y_1+\cdots+y_t\right) / t$​

We take the mean of the observations that were registered in similar conditions

$\hat{y}_{t+h \mid t}=\frac{1}{T}\sum_{t\in \mathcal{T_x}}y_t \quad \mathcal{T_x}=\{t: dist(x_t, x_{t+h})<\epsilon \}$​

Example: I’m predicting sales for a given product on an online platform. The platform applies time-varying discounts based on different market conditions. We can forecast the product taking the mean sales in the same day of the week when the same discount was applied.

Seasonal Naïve method

$\hat{y}_{t+h \mid t}=y_{t+h-m(k+1)}$​

where $m$ is the period of the seasonality

Assuming residuals are normal, uncorrelated:

Note that when h=1 and T is large, these all give the same approximate forecast variance

Standard ML metrics are:

For time series, it is usually better to normalize these scores.

Example: mean absolute scaled error $MASE: mean(\vert e_{T+h}\vert / Q)$. Here $Q$ is a normalizing factor.

The mean model is

Where $\epsilon_t \sim \mathcal{N}(0, \sigma)$. For this method, we need to estimate $c$, which is done via empirical mean:

Since the model treats the predicted variable as constant, the prediction error is independent of the step ahead $h$. The predictive distribution is Gaussian, with variance:

The first quantity is the variance associated to the estimation of the parameter $c$, and stems directly from the central limit theorem: the uncertainty in the estimation of the distribution mean through empirical mean is normally distributed with variance ~ (1/T).

The naive method assumes that the true data generation process has the following form:

We can obtain the formula for the $h_{th}$ step ahead prediction as:

Since the errors are considered to be zero mean, the predictive variable and its variance are:

Where the last expression is due to the fact that a summation of Gaussian random variables is still a Gaussian with variance equal to the sum of their variances.

The seasonal naive method assumes the following datageneration process $\hat{y}_{t+h \mid t}=y_{t+h-m(k+1)}$​

This can be rewritten as:

Thus: