ReMIA score

The ReMIA score is an attack-based metric used to quantify the level of privacy protection provided by a generated dataset. Unlike the Privacy score, which measures how statistically close generated records are to the real ones, the ReMIA score measures whether an attacker could determine that a specific real record was used to train the generator. This class of attack is known as a membership inference attack.

Unlike the Privacy and Similarity scores, the ReMIA score is not computed automatically with every generation. You start each calculation yourself from the generated dataset view page; see View details and evaluate generated dataset.

Calculation methodology

The score computation involves the following steps:

First, a target set of records is selected at random from the real data. The generator is then retrained twice, each run producing its own generated dataset, in such a way that each target record contributes to only one of the two. Every target record is therefore a member of exactly one of the two generations, and a non-member of the other.

Next, a discriminator model is trained to recover that membership: given a target record, it predicts which of the two generations the record contributed to. A control set is tracked alongside the target set throughout training, and provides the reference for how the discriminator behaves when there is no membership signal to find.

The score is finally given by the area under the ROC curve (AUC-ROC) reached by the discriminator, reported together with its standard deviation.

Interpretation

The ReMIA score ranges from 0 to 1 and, unlike the other scores on the platform, its ideal value is not the maximum but the middle of the range. A score of 0.5 means the discriminator does no better than random guessing: membership cannot be recovered, so the generated data reveals nothing about which real records were used to train the generator. The further the score rises above 0.5, the more reliably membership can be inferred, and the higher the privacy risk.

The platform groups the score into four ratings:

Score	Rating	Meaning
Below 0.45	Anomalous	Worse than random guessing by more than measurement noise; do not trust it
0.45 to below 0.60	Excellent	Membership is effectively unrecoverable
0.60 to below 0.70	Good	A weak membership signal is present
0.70 and above	Bad	Membership can be inferred reliably, indicating a concrete privacy risk

A score meaningfully below 0.5 is not better than 0.5. It means the discriminator did worse than chance, which points to a problem with the calculation rather than to a stronger privacy guarantee. The platform rates these results as Anomalous, marks them with a warning icon, and the calculation should be repeated. The band starts at 0.45 rather than 0.5 to leave room for ordinary measurement noise around the baseline.

Keep in mind that the score computation has some intrinsic variance, so the obtained value can vary somewhat between different runs. This is particularly true for small target sets. The standard deviation reported alongside the score indicates how much confidence to place in a single value.