mnoukhov/pythia410m-dpo2-tldr

README.md

@@ -16,13 +16,13 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [mnoukhov/pythia410m-sft-tldr](https://huggingface.co/mnoukhov/pythia410m-sft-tldr) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.7753
-- Rewards/chosen: -6.3555
-- Rewards/rejected: -6.7803
-- Rewards/accuracies: 0.5989
-- Rewards/margins: 0.4248
-- Logps/rejected: -192.7698
-- Logps/chosen: -192.7698
+- Loss: 0.6553
+- Rewards/chosen: -0.0115
+- Rewards/rejected: -0.0984
+- Rewards/accuracies: 0.6726
+- Rewards/margins: 0.0869
+- Logps/rejected: -65.8898
+- Logps/chosen: -65.8898
 - Logps/ref Rejected: -59.5615
 - Logps/ref Chosen: -65.6594
 
@@ -61,11 +61,10 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logps/ref Rejected | Logps/ref Chosen |
 |:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:------------------:|:----------------:|
-| 0.412         | 0.1999 | 335  | 0.6786          | -4.5428        | -4.9111          | 0.6222             | 0.3683          | -156.5151      | -156.5151    | -59.5615           | -65.6594         |
-| 0.3588        | 0.3999 | 670  | 0.7264          | -5.6339        | -6.0502          | 0.6107             | 0.4163          | -178.3372      | -178.3372    | -59.5615           | -65.6594         |
-| 0.345         | 0.5998 | 1005 | 0.7470          | -6.0062        | -6.4506          | 0.6086             | 0.4444          | -185.7831      | -185.7831    | -59.5615           | -65.6594         |
-| 0.3419        | 0.7998 | 1340 | 0.7738          | -6.3469        | -6.7796          | 0.6012             | 0.4327          | -192.5978      | -192.5978    | -59.5615           | -65.6594         |
-| 0.3384        | 0.9997 | 1675 | 0.7753          | -6.3555        | -6.7803          | 0.5989             | 0.4248          | -192.7698      | -192.7698    | -59.5615           | -65.6594         |
+| No log        | 0.2016 | 63   | 0.6719          | 0.0165         | -0.0292          | 0.6739             | 0.0457          | -65.3302       | -65.3302     | -59.5615           | -65.6594         |
+| 0.6865        | 0.4032 | 126  | 0.6614          | 0.0030         | -0.0680          | 0.6752             | 0.0710          | -65.5994       | -65.5994     | -59.5615           | -65.6594         |
+| 0.6865        | 0.6048 | 189  | 0.6571          | -0.0046        | -0.0866          | 0.6744             | 0.0820          | -65.7515       | -65.7515     | -59.5615           | -65.6594         |
+| 0.6771        | 0.8064 | 252  | 0.6553          | -0.0115        | -0.0984          | 0.6726             | 0.0869          | -65.8898       | -65.8898     | -59.5615           | -65.6594         |
 
 
 ### Framework versions

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9e6a3dd0e6839f08f09045a2e1f4df4711246f26a8c860ab223837487af08f98
+oid sha256:4a15b3c5bf62d730bed42d234c59b823c7a0ae8c104bafedadef138040b5c64f
 size 25192592