Kappa is a way to measure agreements or reliability and to correct the frequency with which ratings might consent to chance. Cohens Kappa,[5] who works for two councillors, and Fleiss` Kappa,[6] an adaptation that works for any fixed number of councillors, improve the common likelihood that they would take into account the amount of agreement that could be expected by chance. The original versions suffered from the same problem as the probability of joints, as they treat the data as nominal and assume that the evaluations have no natural nature; if the data does have a rank (ordinal measurement value), this information is not fully taken into account in the measurements. The common probability of an agreement is the simplest and least robust measure. It is estimated as a percentage of the time advisors agree in a nominal or categorical evaluation system. It ignores the fact that an agreement can only be made on the basis of chance. The question arises as to whether a random agreement should be “corrected” or not; Some suggest that such an adaptation is in any case based on an explicit model of the impact of chance and error on business decisions. [3] There are a number of statistics that can be used to determine the reliability of inter-advisors. Different statistics are adapted to different types of measurement. Some options are the common probability of an agreement, Cohens Kappa, Scott`s pi and the Fleiss`Kappa associated with it, inter-rate correlation, correlation coefficient, intra-class correlation and Krippendorff alpha. Haber, Barnhart and his colleagues [16, 17, 18] introduced the individual agreement coefficients (ICAs), which are graduated from an acceptable disagreement, in order to establish the interchangeability of observers.

An “acceptable disagreement” requires that the differences between the measures taken by different observers be similar to the differences between the same observer`s countered measures. The notion of individual concordance derives from the idea of individual bioequivalence in bioequivalence studies [17, 19, 20, 21]. Similar agreements were proposed by Haber et al. [22] and Shao and Zhong [23]. The CIA compares the differences between the measurements taken by different observers and the differences between the same observer`s countered measures. Therefore, they require replications that allow us to assess variability within the observer. The number of replications may vary depending on the topics and observers. In medical and related sciences, many statistical approaches have been proposed to assess the consistency between observers or measurement methods.

In a recent review paper, Barnhart et al. [1] categorize existing methods for the evaluation agreement as follows: (1) descriptive tools such as descriptive statistics and diagrams, (2) uncased agreement indices, such as the average square spread (MSD), coverage probability (CP) and global deviance index (TDI) [2, 3, 4] and (3) indexes on a scale 2. Kappa is similar to a correlation coefficient, as it can`t exceed 1.0 or -1.0.