Consolidate each workshop result instead of pooling ratings

An assumed network of organizations has its crisis preparedness assessed in four workshops, each with a different group and context. For the final presentation, all individual ratings are transferred into one joint analysis. The hope is that the larger sample makes the result more reliable. The opposite happens.

Our published working paper “Bewertung von organisationaler Krisen-Resilienz anhand des Resilienzrisikoreifegrades” (Rico Kerstan and André Röhl, NBS Northern Business School, No. 1/2025; the title means assessing organizational crisis resilience using the resilience risk maturity level) explains why: the workshop ratings were collected with the same method, but context and group composition differed. Pooling all ratings into a larger sample increases variance and blurs prioritization and differentiation. Comparability within and between organizations arises from consolidated individual workshop results, not from the totality of all individual ratings.

My recommendation for program governance is therefore that each workshop remains its own unit of analysis. Consolidation means bringing a workshop's individual ratings and the reasons given together into one agreed result; the plausibility check tests that result against the scenario descriptions and the recorded reasons. Each individual report records the composition of the group and the context considered; only on that basis is a later comparison sound. The cross-cutting review compares consolidated findings, not raw scale values – for example, which scenarios show a high resilience risk in several workshops, that is, an unfavorable ratio of likelihood and impact to resistance and coping capability, or how risks are distributed between scenarios inside the organization and scenarios in its environment. The reasoning recorded in the individual reports is what carries that comparison.

Refraining from a pooled overall analysis is not a loss of precision. The method is qualitative, subjective and consistent only within itself; it claims no objective measurement. Forming a single indicator across all workshops suggests statistical reliability that the data cannot support.

For an assessment program, the rules should therefore be fixed in advance: how each workshop is consolidated, how individual results are compared across workshops – and that no overall sample is formed.