rcognita.controllers.CtrlOptPred

class rcognita.controllers.CtrlOptPred(dim_input, dim_output, mode='MPC', ctrl_bnds=[], action_init=[], t0=0, sampling_time=0.1, Nactor=1, pred_step_size=0.1, sys_rhs=[], sys_out=[], state_sys=[], prob_noise_pow=1, is_est_model=0, model_est_stage=1, model_est_period=0.1, buffer_size=20, model_order=3, model_est_checks=0, gamma=1, Ncritic=4, critic_period=0.1, critic_struct='quad-nomix', stage_obj_struct='quadratic', stage_obj_pars=[], observation_target=[])

Class of predictive optimal controllers, primarily model-predictive control and predictive reinforcement learning, that optimize a finite-horizon cost.

Currently, the actor model is trivial: an action is generated directly without additional policy parameters.

dim_input, dim_output

Dimension of input and output which should comply with the system-to-be-controlled.

Type

: integer

mode

Controller mode. Currently available (\(\rho\) is the stage objective, \(\gamma\) is the discounting factor):

Controller modes

Mode	Cost function
‘MPC’ - Model-predictive control (MPC)	\(J_a \left( y_1, \{action\}_1^{N_a} \right)= \sum_{k=1}^{N_a} \gamma^{k-1} \rho(y_k, u_k)\)
‘RQL’ - RL/ADP via \(N_a-1\) roll-outs of \(\rho\)	\(J_a \left( y_1, \{action\}_{1}^{N_a}\right) = \sum_{k=1}^{N_a-1} \gamma^{k-1} \rho(y_k, u_k) + \hat Q^{\theta}(y_{N_a}, u_{N_a})\)
‘SQL’ - RL/ADP via stacked Q-learning	\(J_a \left( y_1, \{action\}_1^{N_a} \right) = \sum_{k=1}^{N_a-1} \hat \gamma^{k-1} Q^{\theta}(y_{N_a}, u_{N_a})\)

Here, \(\theta\) are the critic parameters (neural network weights, say) and \(y_1\) is the current observation.

Add your specification into the table when customizing the agent.

Type

: string

ctrl_bnds

Box control constraints. First element in each row is the lower bound, the second - the upper bound. If empty, control is unconstrained (default).

Type

: array of shape [dim_input, 2]

action_init

Initial action to initialize optimizers.

Type

: array of shape [dim_input, ]

t0

Initial value of the controller’s internal clock.

Type

: number

sampling_time

Controller’s sampling time (in seconds).

Type

: number

Nactor

Size of prediction horizon \(N_a\).

Type

: natural number

pred_step_size

Prediction step size in \(J_a\) as defined above (in seconds). Should be a multiple of sampling_time. Commonly, equals it, but here left adjustable for convenience. Larger prediction step size leads to longer factual horizon.

Type

: number

sys_rhs, sys_out

Functions that represent the right-hand side, resp., the output of the exogenously passed model. The latter could be, for instance, the true model of the system. In turn, state_sys represents the (true) current state of the system and should be updated accordingly. Parameters sys_rhs, sys_out, state_sys are used in those controller modes which rely on them.

Type

: functions

prob_noise_pow

Power of probing noise during an initial phase to fill the estimator’s buffer before applying optimal control.

Type

: number

is_est_model

Flag whether to estimate a system model. See _estimate_model().

Type

: number

model_est_stage

Initial time segment to fill the estimator’s buffer before applying optimal control (in seconds).

Type

: number

model_est_period

Time between model estimate updates (in seconds).

Type

: number

buffer_size

Size of the buffer to store data.

Type

: natural number

model_order

Order of the state-space estimation model

\[\begin{array}{ll} \hat x^+ & = A \hat x + B action, \newline observation^+ & = C \hat x + D action. \end{array}\]

See _estimate_model(). This is just a particular model estimator. When customizing, _estimate_model() may be changed and in turn the parameter model_order also. For instance, you might want to use an artifial neural net and specify its layers and numbers of neurons, in which case model_order could be substituted for, say, Nlayers, Nneurons.

Type

: natural number

model_est_checks

Estimated model parameters can be stored in stacks and the best among the model_est_checks last ones is picked. May improve the prediction quality somewhat.

Type

: natural number

gamma

Discounting factor. Characterizes fading of stage objectives along horizon.

Type

: number in (0, 1]

Ncritic

Critic stack size \(N_c\). The critic optimizes the temporal error which is a measure of critic’s ability to capture the optimal infinite-horizon cost (a.k.a. the value function). The temporal errors are stacked up using the said buffer.

Type

: natural number

critic_period

The same meaning as model_est_period.

Type

: number

critic_struct

Choice of the structure of the critic’s features.

Currently available:

Critic structures

Mode	Structure
‘quad-lin’	Quadratic-linear
‘quadratic’	Quadratic
‘quad-nomix’	Quadratic, no mixed terms
‘quad-mix’	Quadratic, no mixed terms in input and output, i.e., \(w_1 y_1^2 + \dots w_p y_p^2 + w_{p+1} y_1 u_1 + \dots w_{\bullet} u_1^2 + \dots\), where \(w\) is the critic’s weight vector

Add your specification into the table when customizing the critic.

Type

: natural number

stage_obj_struct

Choice of the stage objective structure.

Currently available:

Critic structures

Mode	Structure
‘quadratic’	Quadratic \(\chi^\top R_1 \chi\), where \(\chi = [observation, action]\), stage_obj_pars should be [R1]
‘biquadratic’	4th order \(\left( \chi^\top \right)^2 R_2 \left( \chi \right)^2 + \chi^\top R_1 \chi\), where \(\chi = [observation, action]\), stage_obj_pars should be [R1, R2]

Pass correct stage objective parameters in stage_obj_pars (as a list)

When customizing the stage objective, add your specification into the table above

Type

: string

References

1

Osinenko, Pavel, et al. “Stacked adaptive dynamic programming with unknown system model.” IFAC-PapersOnLine 50.1 (2017): 4150-4155

__init__(dim_input, dim_output, mode='MPC', ctrl_bnds=[], action_init=[], t0=0, sampling_time=0.1, Nactor=1, pred_step_size=0.1, sys_rhs=[], sys_out=[], state_sys=[], prob_noise_pow=1, is_est_model=0, model_est_stage=1, model_est_period=0.1, buffer_size=20, model_order=3, model_est_checks=0, gamma=1, Ncritic=4, critic_period=0.1, critic_struct='quad-nomix', stage_obj_struct='quadratic', stage_obj_pars=[], observation_target=[])

Parameters

• dim_input (: integer) – Dimension of input and output which should comply with the system-to-be-controlled.

• dim_output (: integer) – Dimension of input and output which should comply with the system-to-be-controlled.

• mode (: string) –

  Controller mode. Currently available (\(\rho\) is the stage objective, \(\gamma\) is the discounting factor):

  Controller modes

  Mode	Cost function
  ’MPC’ - Model-predictive control (MPC)	\(J_a \left( y_1, \{action\}_1^{N_a} \right)= \sum_{k=1}^{N_a} \gamma^{k-1} \rho(y_k, u_k)\)
  ’RQL’ - RL/ADP via \(N_a-1\) roll-outs of \(\rho\)	\(J_a \left( y_1, \{action\}_{1}^{N_a}\right) = \sum_{k=1}^{N_a-1} \gamma^{k-1} \rho(y_k, u_k) + \hat Q^{\theta}(y_{N_a}, u_{N_a})\)
  ’SQL’ - RL/ADP via stacked Q-learning	\(J_a \left( y_1, \{action\}_1^{N_a} \right) = \sum_{k=1}^{N_a-1} \gamma^{k-1} \hat Q^{\theta}(y_{N_a}, u_{N_a})\)

  Here, \(\theta\) are the critic parameters (neural network weights, say) and \(y_1\) is the current observation.

  Add your specification into the table when customizing the agent .

• ctrl_bnds (: array of shape [dim_input, 2]) – Box control constraints. First element in each row is the lower bound, the second - the upper bound. If empty, control is unconstrained (default).

• action_init (: array of shape [dim_input, ]) – Initial action to initialize optimizers.

• t0 (: number) – Initial value of the controller’s internal clock

• sampling_time (: number) – Controller’s sampling time (in seconds)

• Nactor (: natural number) – Size of prediction horizon \(N_a\)

• pred_step_size (: number) – Prediction step size in \(J\) as defined above (in seconds). Should be a multiple of sampling_time. Commonly, equals it, but here left adjustable for convenience. Larger prediction step size leads to longer factual horizon.

• sys_rhs (: functions) – Functions that represent the right-hand side, resp., the output of the exogenously passed model. The latter could be, for instance, the true model of the system. In turn, state_sys represents the (true) current state of the system and should be updated accordingly. Parameters sys_rhs, sys_out, state_sys are used in those controller modes which rely on them.

• sys_out (: functions) – Functions that represent the right-hand side, resp., the output of the exogenously passed model. The latter could be, for instance, the true model of the system. In turn, state_sys represents the (true) current state of the system and should be updated accordingly. Parameters sys_rhs, sys_out, state_sys are used in those controller modes which rely on them.

• prob_noise_pow (: number) – Power of probing noise during an initial phase to fill the estimator’s buffer before applying optimal control.

• is_est_model (: number) – Flag whether to estimate a system model. See _estimate_model().

• model_est_stage (: number) – Initial time segment to fill the estimator’s buffer before applying optimal control (in seconds).

• model_est_period (: number) – Time between model estimate updates (in seconds).

• buffer_size (: natural number) – Size of the buffer to store data.

• model_order (: natural number) –

  Order of the state-space estimation model

  \[\begin{array}{ll} \hat x^+ & = A \hat x + B action, \newline observation^+ & = C \hat x + D action. \end{array}\]

  See _estimate_model(). This is just a particular model estimator. When customizing, _estimate_model() may be changed and in turn the parameter model_order also. For instance, you might want to use an artifial neural net and specify its layers and numbers of neurons, in which case model_order could be substituted for, say, Nlayers, Nneurons

• model_est_checks (: natural number) – Estimated model parameters can be stored in stacks and the best among the model_est_checks last ones is picked. May improve the prediction quality somewhat.

• gamma (: number in (0, 1]) – Discounting factor. Characterizes fading of stage objectives along horizon.

• Ncritic (: natural number) – Critic stack size \(N_c\). The critic optimizes the temporal error which is a measure of critic’s ability to capture the optimal infinite-horizon cost (a.k.a. the value function). The temporal errors are stacked up using the said buffer.

• critic_period (: number) – The same meaning as model_est_period.

• critic_struct (: natural number) –

  Choice of the structure of the critic’s features.

  Currently available:

  Critic feature structures

  Mode	Structure
  ’quad-lin’	Quadratic-linear
  ’quadratic’	Quadratic
  ’quad-nomix’	Quadratic, no mixed terms
  ’quad-mix’	Quadratic, no mixed terms in input and output, i.e., \(w_1 y_1^2 + \dots w_p y_p^2 + w_{p+1} y_1 u_1 + \dots w_{\bullet} u_1^2 + \dots\), where \(w\) is the critic’s weights

  Add your specification into the table when customizing the critic.

• stage_obj_struct (: string) –

  Choice of the stage objective structure.

  Currently available:

  Running objective structures

  Mode	Structure
  ’quadratic’	Quadratic \(\chi^\top R_1 \chi\), where \(\chi = [observation, action]\), stage_obj_pars should be [R1]
  ’biquadratic’	4th order \(\left( \chi^\top \right)^2 R_2 \left( \chi \right)^2 + \chi^\top R_1 \chi\), where \(\chi = [observation, action]\), stage_obj_pars should be [R1, R2]

Methods

__init__(dim_input, dim_output[, mode, …])	param dim_input Dimension of input and output which should comply with the system-to-be-controlled.
compute_action(t, observation)	Main method.
receive_sys_state(state)	Fetch exogenous model state.
reset(t0)	Resets agent for use in multi-episode simulation.
stage_obj(observation, action)	Stage (equivalently, instantaneous or running) objective.
upd_accum_obj(observation, action)	Sample-to-sample accumulated (summed up or integrated) stage objective.