Smart negative sampling strategies

[KulikovNikita]

Two negative sampling strategies were introduced:

1. FrequencyNegativeSamplingTransform - based on frequency of already made choices of negative indices

• Computes frequencies of already made choices (number of times each item was already sampled)
• Adjusts probabilities according to this frequencies like: softmax(1.0 / (1.0 + frequency))

2. ThresholdNegativeSamplingTransform - works as latter but ignores maxed-out items completely

Metrics on MovieLens (as in example):
0) UniformNegativeSamplingTransform(num_negative_samples = 128) - baseline

Validation (81-st epoch):

      "k              1        10        20         5\n",
      "map     0.019705  0.060155  0.067919  0.048319\n",
      "ndcg    0.019705  0.091545  0.120256  0.062412\n",
      "recall  0.019705  **0.196721**  0.311144  0.105647\n",

Link to proof.

Test (best epoch):

       "k                 1        10        20         5\n",
       "MAP        0.016065  0.054039  0.061749  0.043969\n",
       "Precision  0.016065  0.017655  0.014458  0.020073\n",
       "Recall     0.016065  **0.176549**  0.289169  0.100364\n",

Link to proof.

2. FrequencyNegativeSamplingTransform(num_negative_samples = 128)

Validation (96-th epoch):

      "k             1        10        20         5\n",
      "map     0.01954  0.060201  0.068178  0.048722\n",
      "ndcg    0.01954  0.091547  0.121043  0.063432\n",
      "recall  0.01954  **0.196225**  0.313794  0.108627\n",

Link to proof.

Test (best checkpoint):

       "k                 1        10        20         5\n",
       "MAP        0.020371  0.061189  0.068575  0.051101\n",
       "Precision  0.020371  0.018731  0.014790  0.022093\n",
       "Recall     0.020371  **0.187314**  0.295793  0.110467\n",

Link to proof.

Notice: I suspect that better results on the test fold are due to the model seeing more diverse embedding set.

2. ThresholdNegativeSamplingTransform(num_negative_samples = 128)

Validation (99-th epoch):

      "k             1        10        20         5\n",
      "map     0.01391  0.056347  0.064440  0.044130\n",
      "ndcg    0.01391  0.089694  0.119503  0.059647\n",
      "recall  0.01391  **0.201358**  0.319921  0.107468\n",

Link to proof.

Test (best checkpoint):

       "k                 1        10        20         5\n",
       "MAP        0.016065  0.056637  0.064067  0.045948\n",
       "Precision  0.016065  0.018665  0.014798  0.021133\n",
       "Recall     0.016065  **0.186651**  0.295959  0.105664\n",

Link to proof.