Backtesting guidelines for US, UK and Swiss Regulators
Summary
Banks have the obligation to deliver back-testing results of Counterparty Credit Risk (CCR) exposure models to the regulators on a quartely basis. Back Testing Platforms have been designed to meet Basel III requirements and Capital Requirements Directive 4 CRD4 compliance.
Back testing is the process of challenging models by confronting its results with what has actually happened in the past. The models challenged here are the exposure models that predict the future Mark-to-Market (MtM) distribution from which we derive the banks’ CCR measures. Banks will have to design and implement automated strategic solutions allowing monthly updates and reporting across all Product types covered by the VaR models.
Backtesting Policy
- What is backtesting?
- Regulatory notification requirements
- Capital impact
- New Backtesting Exceptions Categories and Exceptions classification & remedial action
- Actual and Predicted losses
What is backtesting?
- FED: “Backtesting is one form of outcomes analysis that involves the comparison of actual outcomes with model forecasts during a sample time period not used in model development at a frequency that matches the model’s forecast horizon or performance window.”
- In simple terms, we have two forms of backtesting:
- Hypothetical P&L: takes into account the P&L generated by the same portfolio in existence at the previous COB – i.e. any intra-day activity is excluded.
- Actual P&L: takes into account the overall P&L generated during the day, including any intra-day activity.
Both measures are calculated on Trading Book only positions, to match the Backtesting VaR (which is a 1-day version of the Regulatory VaR*).
Regulatory notification requirements
| Regulator | P&L Event | Timelines |
| PRA | Hypothetical and Actual P&L | t+2: Initial notification (including Var, P&L and main business driving the exception)
t+3 (i.e. 3 business days after the month in which the exception occurred): Detailed notification to the regulator, outlining the key drivers behind the P&L loss. |
| FINMA | Hypothetical | t+3 (i.e. 3 business days after the month in which the exception occurred): Detailed notification to the regulator, outlining the key drivers behind the P&L loss. |
| SFC | Hypothetical and Actual P&L* | t+3 (i.e. 3 business days after the month in which the exception occurred): Detailed notification to the regulator, outlining the key drivers behind the P&L loss. |
| SEC | Hypothetical P&L | t+3 (i.e. 3 business days after the month in which the exception occurred): Detailed notification to the regulator, outlining the key drivers behind the P&L loss. |
Capital impact
| Number of exceptions | Plus Factor | |
| 0 | 0.00 | |
| 1 | 0.00 | |
| Green Zone | 2 | 0.00 |
| 3 | 0.00 | |
| 4 | 0.00 | |
| 5 | 0.40 | |
| 6 | 0.50 | |
| Amber Zone | 7 | 0.65 |
| 8 | 0.75 | |
| 9 | 0.85 | |
| Red Zone | 10 | 1.00 |
- The increased capital multiplier has impact on:
- VaR
- SVaR
- VaR-equivalent RNIVs/SRNIVs
which means the 5th exception at entity level is going to increase by ~10% capital/RWAs from the above.
New Backtesting Exceptions Categories
Summary
- PRA requested a more structured approach to handling backtesting exceptions.
- New more granular approach:
- allows more accurate identification of the root cause; and
- facilitates the definition of any remedial action with clearly defined time-frames and actors.
- Currently 20 categories split in 4 key groups:
1.Extreme Move: exceptions which are truly beyond the VaR confidence level;
2.VaR Accuracy: exceptions which are driven by issues with risks the VaR model should have captured but did not;
3.VaR Completeness: exceptions which are driven by issues with risks not included in the perimeter of the VaR model (but which are for example included in RNIVs); and
4.P&L: exceptions driven by issues on the P&L side.
- Categories are dynamic and will evolve based on feedback from the backtesting process itself.
Backtesting Exceptions Categories: Extreme Move
| Category | Remedial Action | Remedial Actors |
| Extreme Market Move: Tail Event (a.k.a. ‘1% event’) | This classification does not indicate any issue with the VaR model. | N/A
|
| Extreme Market Move: VaR Dataset (a.k.a. ‘recent volatility increase’) | This classification does not indicate any issue with the VaR model, however raises the need to have a dataset update in order to avoid similar future exceptions. | Market risk mngm |
| Extreme Market Move: Correlation Break (a.k.a. ‘structural break’) | This classification does not indicate any issue with the VaR model, however could be mitigated by a dataset update or potential other adjustments (i.e. scaling, time-weighting, etc). | Market risk mngm Models & Risk Methodologies |
Extreme Market Move: Tail Event
| Category | Remedial Action | Remedial Actors |
| Extreme Market Move: Tail Event (a.k.a. ‘1% event’) | This classification does not indicate any issue with the VaR model. | N/A
|
Example
An exception where the key losses are driven by EUR/USD FX rate falling by 4% when the 99% move implied by the VaR model is 2% would be classified in this category.
| Category | Remedial Action | Remedial Actors |
| Extreme Market Move: Tail Event (a.k.a. ‘1% event’) | This classification does not indicate any issue with the VaR model. | N/A
|
| Extreme Market Move: VaR Dataset (a.k.a. ‘recent volatility increase’) | This classification does not indicate any issue with the VaR model, however raises the need to have a dataset update in order to avoid similar future exceptions. | Market Risk Mngmt
|
| Extreme Market Move: Correlation Break (a.k.a. ‘structural break’) | This classification does not indicate any issue with the VaR model, however could be mitigated by a dataset update or potential other adjustments (i.e. scaling, time-weighting, etc). | Market Risk Mngmt Models & Risk Methodologies |
Extreme Market Move: Correlation Break
| Category | Remedial Action | Remedial Actors |
| Extreme Market Move: Correlation Break (a.k.a. ‘structural break’) | This classification does not indicate any issue with the VaR model, however could be mitigated by a dataset update or potential other adjustments (i.e. scaling, time-weighting, etc). | Market Risk Mngmt
Models & Risk Methodologies |
Example
An exception where the key losses are driven by a market rally on short delta occurring together with additional losses from a rapid increase in implied volatilities on a short vega position would fall under this category.
Backtesting Exceptions Categories: Model Accuracy
| Category | Remedial Action | Remedial Actors |
| VaR Model Accuracy: VaR Dataset (a.k.a. ‘time-series issue’) | In this case the incorrect time-series should be fixed and any related time-series potentially affected should be reviewed. | Market Risk Mngmt
|
| VaR Model Accuracy: Model Specification (a.k.a. ‘model specification issue’) | In this case the VaR model should be reviewed and steps taken to improve the model specification, and any other VaR model with similar characteristics should be reviewed to ensure is not affected by a similar issue. | Market Risk Mngmt
Models & Risk Methodologies |
| VaR Model Accuracy: Model Implementation (a.k.a. ‘model implementation issue’) | In this case the VaR model implementation should be corrected and any other VaR model with similar characteristics should be reviewed to ensure is not affected by a similar issue. | Models & Risk Methodologies |
| VaR Model Accuracy: Regression | This classification does not indicate any issue with the VaR model, however the reason for the regression should be identified and steps taken to avoid it occurring again in the future. | Market Risk Mngmt
|
| VaR Model Accuracy: VaR/P&L Mismatch | Market Risk Units should work with PC to identify the reason for the mismatch, resolve the issue and prevent (or at least mitigate) from happening in the future. | Market Risk Mngmt
PC |
| VaR Model Accuracy: Other | The remedial action/actors will be defined on a case by case basis. | The remedial action/actors will be defined on a case by case basis. |
VaR Model Accuracy: VaR Dataset
| Category | Remedial Action | Remedial Actors |
| VaR Model Accuracy: VaR Dataset (a.k.a. ‘time-series issue’) | In this case the incorrect time-series should be fixed and any related time-series potentially affected should be reviewed. | Market Risk Mngmt |
Example
Under this category would be an exception where: the key losses are driven by the 3m yield falling by 4%, and where the 99% move implied by the VaR model should have been 5%, but where the 99% move implied by the VaR model was actually only 3% due to having used an incorrect time-series (i.e. wrong currency, tenor, onshore/offshore, etc).
VaR Model Accuracy: Model Specification
| Category | Remedial Action | Remedial Actors |
| VaR Model Accuracy: Model Specification (a.k.a. ‘model specification issue’) | In this case the VaR model should be reviewed and steps taken to improve the model specification, and any other VaR model with similar characteristics should be reviewed to ensure is not affected by a similar issue. | Market Risk Mngmt
Models & Risk Methodologies |
Example
Under this category would be an exception where the model specification for a certain risk factor misses a key dynamic of the risk factor (i.e. a model unable to deal with negative interest rates).
VaR Model Accuracy: Model Implementation
| Category | Remedial Action | Remedial Actors |
| VaR Model Accuracy: Model Implementation (a.k.a. ‘model implementation issue’) | In this case the VaR model implementation should be corrected and any other VaR model with similar characteristics should be reviewed to ensure is not affected by a similar issue. | Models & Risk Methodologies |
Example
Under this category would be an exception where the model specification for a certain risk factor is correct, but has been incorrectly implemented (i.e. formula error).
VaR Model Accuracy: Regression
| Category | Remedial Action | Remedial Actors |
| VaR Model Accuracy: Regression | This classification does not indicate any issue with the VaR model, however the reason for the regression should be identified and steps taken to avoid it occurring again in the future. | Market Risk Mngmt
|
Example
Under this category would be an exception where on day X a desk increased EUR FX delta by +$100m to +$200m, but subsequently regressed the risk to day X-1 to +$100m. An exception triggered on day X+1 due to P&L on EUR being -$4m due to a -2% move, but where the 99% VaR move implied by the VaR model is -3%, would be classified under this category (as with the correct sensitivity VaR would have been $6m rather than $3m).
VaR Model Accuracy: VaR/P&L Mismatch
| Category | Remedial Action | Remedial Actors |
| VaR Model Accuracy: VaR/P&L Mismatch | Market Risk Management should work with PC to identify the reason for the mismatch, resolve the issue and prevent (or at least mitigate) from happening in the future. | Market Risk Mngmt
PC |
Example
An exception driven predominantly by a market move which appears less extreme than the one which should have been predicted by the VaR model at the given confidence level, but where the sensitivity/exposure feeding the system is not aligned with the one driving the P&L.
Backtesting Exceptions Categories: Model Completeness
| Category | Remedial Action | Remedial Actors |
| VaR Model Completeness: RNiV – Adequate Coverage | This classification does not indicate any issue with the VaR model. The exact details of the RNIV(s) covering the losses should be provided, including an estimate of the 1-day VaR equivalent amount (with underlying assumptions). | Market Risk Mngmt
Models & Risk Methodologies |
| VaR Model Completeness: RNiV – Lack of Coverage | This classification does not indicate any issue with the VaR model, but with the RNIV model. The exact details of the RNIV(s) covering the losses should be provided, including an estimate of the 1-day VaR equivalent amount (with underlying assumptions). An investigation on why the RNIV was insufficient to cover for the losses should take place, potentially triggering a review of the RNIV model. | Market Risk Mngmt
Models & Risk Methodologies |
| VaR Model Completeness: Incomplete Risk Capture | In this instance the identification of an appropriate way to capitalise the driver of the losses, for example through a new VaR model or a new RNIV, should occur. | Market Risk Mngmt
Models & Risk Methodologies |
| VaR Model Completeness: Net Interest Income (a.k.a. ‘carry’) | This classification does not indicate any issue with the VaR model. | N/A |
VaR Model Completeness: RNiV – Adequate Coverage
| Category | Remedial Action | Remedial Actors |
| VaR Model Completeness: RNiV – Adequate Coverage | This classification does not indicate any issue with the VaR model. The exact details of the RNIV(s) covering the losses should be provided, including an estimate of the 1-day VaR equivalent amount (with underlying assumptions). | Market Risk Mngmt
Models & Risk Methodologies |
Example
An exception where:
1) the majority of the losses are driven by ETF vs NAV basis, and
2) the losses from this risk are less severe than the 1-day VaR equivalent for this RNIV
would fall under this category.
VaR Model Completeness: RNiV – Lack of Coverage
| Category | Remedial Action | Remedial Actors |
| VaR Model Completeness: RNiV – Lack of Coverage | This classification does not indicate any issue with the VaR model, but with the RNIV model. The exact details of the RNIV(s) covering the losses should be provided, including an estimate of the 1-day VaR equivalent amount (with underlying assumptions). An investigation on why the RNIV was insufficient to cover for the losses should take place, potentially triggering a review of the RNIV model. | Market Risk Mngmt
Models & Risk Methodologies |
Example
An exception where:
1) the majority of the losses are driven by ETF vs NAV basis, and
2) the losses from this risk are more severe than the 1-day VaR equivalent for this RNIV
would fall under this category.
VaR Model Completeness: Incomplete Risk Capture
| Category | Remedial Action | Remedial Actors |
| VaR Model Completeness: Incomplete Risk Capture | In this instance the identification of an appropriate way to capitalise the driver of the losses, for example through a new VaR model or a new RNIV, should occur. | Market Risk Mngmt
Models & Risk Methodologies |
Example
An exception driven predominantly by a gap in risk capture in the official model where that gap is not adequately capitalised by an appropriate Risk Not In VaR model.
Backtesting Exceptions Categories: P&L
| Category | Remedial Action | Remedial Actors |
| P&L: New Trade | This classification does not indicate any issue with the VaR model. | N/A |
| P&L: Inception Profit | This classification does not indicate any issue with the VaR model, however an investigation should take place into the underlying trades and steps taken to mitigate it where possible in the future. | Market Risk Mngmt
PC |
| P&L: Multiple Days Market Move | The reason for the multiple days market move should be investigated and steps taken to prevent or mitigate the issue in the future. | Market Risk Mngmt
PC |
| P&L Adjustment: Extraordinary | The reason for the extraordinary P&L adjustment should be investigated, for example whether it might be more suitable to be included in a P&L category outside backtesting, and steps taken to prevent or mitigate the issue in the future. | Market Risk Mngmt
PC |
| P&L Adjustment: Month-end Price-Testing Adjustment | The reason for the P&L adjustment should be investigated and steps taken to prevent or at least mitigate the issue in the future. | Market Risk Mngmt
PC |
| P&L Adjustment: Other Month-end Adjustment | The reason for the P&L adjustment should be investigated and steps taken to prevent or at least mitigate the issue in the future. | Market Risk Mngmt
PC |
| P&L: Other | The remedial action/actors will be defined on a case by case basis. | Market Risk Mngmt
PC |
Actual and Predicted Loss
Once the portfolios are defined and mapped to Backtesting Data, the system calculates the predicted and actual loss figures each day for the horizon and confidence level as defined for the portfolio and stored in the system. A 3-day VaR means VaR figure calculated for a 3 business day horizon. Thus, the actual MTM P&L calculations for Backtesting are done for a horizon of 3 business days. The business days are calculated based on the Entity calendar. The system identifies the deals or holdings in that portfolio and stores the risk events for each of the constituents of the portfolio as defined for Backtesting on a daily basis to identify the value of the portfolio on the day of acquisition. Also, since one deal or holding can belong to multiple portfolios, you have to ensure that the system maintains this data so that there is no duplicate or redundant information.
Once the acquisition cost and the portfolio are defined and captured in terms of risk events, then from the next day onwards, the system starts computing the MTM value for each of the risk events. Using the MTM value and the original risk events, you can calculate the MTM P&L between the acquisition cost and the MTM price. The sum of MTM P&L for each of the constituents of the portfolio represents the actual loss incurred on the defined portfolio. This MTM P&L for the day is stored in the system for future reference. The system also stores the portfolio MTM P&L for each day of every portfolio defined for Backtesting.
The MTM P&L of the portfolio on each calculation day in the horizon is then compared with the Horizon Level P&L, which represents the maximum MTM P&L loss of the portfolio for the horizon. Thus, if MTM P&L calculated on a day is less than the Horizon Level P&L, then it is updated with the MTM P&L calculated on that day.
The Horizon MTM P&L is in the VaR currency. For comparison and updation, the Portfolio MTM P&L figure converted to the VaR currency is used. The conversion rate to be used for this conversion can be either the conversion rate used for VaR calculation or the existing spot rate depending upon the value of the parameter Backtesting. If the value of this parameter is validated, then the conversion rate used for VaR calculation is used for MTM calculation.
Matured Deal Behaviour
In case any of the deal is maturing during the horizon period, it is ignored from the actual Portfolio MTM P&L calculations from its maturity date or end date. The end date of the particular deal is considered for the VaR calculations.
Matured FX deals and positions are also ignored for Backtesting purposes. The VaR calculations in the system also do not take into accounting matured FX positions.
Deleted Deal Behaviour
In case any of the deal is deleted during the horizon period, it is considered for the actual Portfolio MTM P&L calculations for the entire horizon period. Deletion is a user action and is not incorporated during the VaR calculations.
If a given trade is deleted after maturity, it is not considered for subsequent P&L calculations.
riskderivatives 