When you label/decision a user, a few things happen:
- We immediately update your fraud prediction model with the new information. When you label/decision a user Bad, your model immediately becomes better able to find users that exhibit similar fraudulent behaviors and traits. After you send a label/decision, all users we receive events for will be scored using these new insights.
- We immediately re-score the users sharing the same device to help you catch fraudulent accounts as quickly as possible.
- We immediately re-score the user the label/decision was sent for. This re-score is based on your fraud prediction model and not on the label/decision itself. In other words, a label/decision does not add points to a user's score by itself. When the user is re-scored, their score can go up, down, or stay the same. How the score changes depends on the events and labels/decisions we've received for all of your other users in the meantime (in other words, all the learning that's taken place since the last time this user was scored).
As a result of the re-score we do, sometimes the score change will not be intuitive - you'll label/decision a user as Bad and their score will go down, or label/decision a user a Not Bad and their score will go up. Your predictions improve with each new event and label/decision you send, and over time the score threshold for taking actions like "prevent/block" "review/hold/add verification" and "allow/accept" can shift.
As an example, if you label/decision a user with a Score of 95 as Bad their score might change to 93. There are two important things to note here: first, when you automate with our APIs you can use a Bad label/decision when making decisions, and in those cases the score wouldn't be a factor. Second, this change can be an indication that your average score has shifted and your accuracy has improved. If you're currently blocking at scores of 94 and above, this is a good time to take a look at users/orders/posts with scores of 92 and 93. It's quite possible that Sift Science is now doing a better job of separating good and fraudulent users so blocking at scores or 92 or 93 is ideal.
Same with Not Bad: if you label/decision a user as Not Bad and their score goes up this isn't cause for concern. We look at thousands of fraud signals, and all users have some signals that push their score up and some that pull their score down. The higher the score for this user, the more signals they have pushing their score up. Giving the Not Bad label/decision teaches Sift that, for your users, those signals don't always mean a user is fraudulent. However, the user still has those signals so we wouldn't expect their score to take a huge jump down. Often, though, when users you've labeled/decisioned as Not Bad have new activity after the label/decision their score does goes down. If it continues to go up, it may be time to re-evaluate the user as new things we've learned are making them seem suspicious.
If you think a user continues to be way too high or too low, a useful question to ask is: is there anything I know about this user that tells me they're good or bad that isn't in the Sift console? If so, this might be a very useful new event or field to send to improve your integration and help Sift properly score the user.
Finally, it's best to put minor score shifts in perspective. If a user with a score of 50 goes to 46 or 54, they still likely fall in same range as they did before (for example the "allow" range or the "review" range).
When reviewing users in the Sift Science Console, you can filter lists by label/decision. This way, you can choose to look at -- or not look at -- users with a Bad or Not Bad label/decisions when reviewing.