jukofyork

/

creative-writing-control-vectors-v3.0

like 13

For all the control vectors I have uploaded here I already exclude the last layer, but the c4ai-command-r-08-2024model looks a bit odd:

Loading pre/post prompt stems from 'data/prompt_stems.json'... Done (50 + 50 loaded).
Loading prompt continuations from 'data/writing_style_continuations/language.json'... Done (3 classes; each with 10 continuations loaded).
Loading writing prompts from 'data/writing_prompts.txt'... Done (11835 loaded).
Generating dataset samples... Done ([3 classes x 2730 prompts] 8190 generated).
Loading '/mnt/data/c4ai-command-r-08-2024' model and tokenizer...
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:18<00:00,  1.31s/it]
Tokenizing prompts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8190/8190 [00:03<00:00, 2067.67it/s]
Sampling hidden states: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8190/8190 [43:52<00:00,  3.11it/s]
Saving to 'command-r-08-2024:32b-language__hidden_state_samples.pt'... Done.
Testing Eigenvector Directions for layers 1 to 39:
- Layer 1: [1/8192 filtered] [1/8192 selected] Δ = 484%, Δσ² = 82.9%, σ= (0.040, 0.039), μ = (-0.081, 0.093 [53.5%]) -->  μ' = (0.006, -0.087, 0.087)
- Layer 2: [1/8192 filtered] [1/8192 selected] Δ = 328%, Δσ² = 76.6%, σ= (0.071, 0.062), μ = (-0.118, 0.124 [51.4%]) -->  μ' = (0.003, -0.121, 0.121)
- Layer 3: [1/8192 filtered] [1/8192 selected] Δ = 318%, Δσ² = 76.1%, σ= (0.053, 0.058), μ = (-0.095, 0.104 [52.1%]) -->  μ' = (0.004, -0.100, 0.100)
- Layer 4: [1/8192 filtered] [1/8192 selected] Δ = 326%, Δσ² = 76.5%, σ= (0.063, 0.059), μ = (-0.110, 0.110 [50.1%]) -->  μ' = (0.000, -0.110, 0.110)
- Layer 5: [1/8192 filtered] [1/8192 selected] Δ = 311%, Δσ² = 75.7%, σ= (0.104, 0.092), μ = (-0.172, 0.173 [50.2%]) -->  μ' = (0.001, -0.173, 0.173)
- Layer 6: [1/8192 filtered] [1/8192 selected] Δ = 376%, Δσ² = 79.0%, σ= (0.103, 0.093), μ = (-0.190, 0.191 [50.2%]) -->  μ' = (0.001, -0.190, 0.190)
- Layer 7: [1/8192 filtered] [1/8192 selected] Δ = 533%, Δσ² = 84.2%, σ= (0.194, 0.189), μ = (-0.438, 0.445 [50.4%]) -->  μ' = (0.004, -0.442, 0.442)
- Layer 8: [1/8192 filtered] [1/8192 selected] Δ = 177%, Δσ² = 63.9%, σ= (0.355, 0.327), μ = (-0.451, 0.457 [50.4%]) -->  μ' = (0.003, -0.454, 0.454)
- Layer 9: [1/8192 filtered] [1/8192 selected] Δ = 322%, Δσ² = 76.3%, σ= (0.304, 0.256), μ = (-0.513, 0.495 [49.1%]) -->  μ' = (-0.009, -0.504, 0.504)
- Layer 10: [1/8192 filtered] [1/8192 selected] Δ = 359%, Δσ² = 78.2%, σ= (0.361, 0.303), μ = (-0.640, 0.620 [49.2%]) -->  μ' = (-0.010, -0.630, 0.630)
- Layer 11: [1/8192 filtered] [1/8192 selected] Δ = 411%, Δσ² = 80.4%, σ= (0.362, 0.304), μ = (-0.693, 0.663 [48.9%]) -->  μ' = (-0.015, -0.678, 0.678)
- Layer 12: [1/8192 filtered] [1/8192 selected] Δ = 362%, Δσ² = 78.4%, σ= (0.490, 0.347), μ = (-0.819, 0.797 [49.3%]) -->  μ' = (-0.011, -0.808, 0.808)
- Layer 13: [1/8192 filtered] [1/8192 selected] Δ = 402%, Δσ² = 80.1%, σ= (0.440, 0.336), μ = (-0.794, 0.777 [49.5%]) -->  μ' = (-0.008, -0.786, 0.786)
- Layer 14: [1/8192 filtered] [1/8192 selected] Δ = 406%, Δσ² = 80.2%, σ= (0.432, 0.375), μ = (-0.826, 0.805 [49.4%]) -->  μ' = (-0.010, -0.815, 0.815)
- Layer 15: [1/8192 filtered] [1/8192 selected] Δ = 347%, Δσ² = 77.6%, σ= (0.869, 0.837), μ = (-1.601, 1.577 [49.6%]) -->  μ' = (-0.012, -1.589, 1.589)
- Layer 16: [1/8192 filtered] [1/8192 selected] Δ = 348%, Δσ² = 77.7%, σ= (0.758, 0.810), μ = (-1.478, 1.451 [49.5%]) -->  μ' = (-0.014, -1.465, 1.465)
- Layer 17: [1/8192 filtered] [1/8192 selected] Δ = 376%, Δσ² = 79.0%, σ= (0.774, 0.767), μ = (-1.548, 1.440 [48.2%]) -->  μ' = (-0.054, -1.494, 1.494)
- Layer 18: [1/8192 filtered] [1/8192 selected] Δ = 295%, Δσ² = 74.7%, σ= (1.847, 2.050), μ = (-3.320, 3.379 [50.4%]) -->  μ' = (0.029, -3.350, 3.350)
- Layer 19: [1/8192 filtered] [1/8192 selected] Δ = 269%, Δσ² = 72.9%, σ= (1.918, 2.467), μ = (-3.522, 3.725 [51.4%]) -->  μ' = (0.102, -3.623, 3.623)
- Layer 20: [1/8192 filtered] [1/8192 selected] Δ = 264%, Δσ² = 72.5%, σ= (1.966, 2.447), μ = (-3.555, 3.655 [50.7%]) -->  μ' = (0.050, -3.605, 3.605)
- Layer 21: [1/8192 filtered] [1/8192 selected] Δ = 416%, Δσ² = 80.6%, σ= (2.123, 2.250), μ = (-4.450, 4.469 [50.1%]) -->  μ' = (0.010, -4.460, 4.460)
- Layer 22: [1/8192 filtered] [1/8192 selected] Δ = 425%, Δσ² = 80.9%, σ= (2.139, 2.362), μ = (-4.411, 4.873 [52.5%]) -->  μ' = (0.231, -4.642, 4.642)
- Layer 23: [1/8192 filtered] [1/8192 selected] Δ = 414%, Δσ² = 80.5%, σ= (2.409, 2.619), μ = (-4.793, 5.444 [53.2%]) -->  μ' = (0.326, -5.118, 5.118)
- Layer 24: [1/8192 filtered] [1/8192 selected] Δ = 413%, Δσ² = 80.5%, σ= (2.571, 2.794), μ = (-4.981, 5.932 [54.4%]) -->  μ' = (0.475, -5.456, 5.456)
- Layer 25: [1/8192 filtered] [1/8192 selected] Δ = 588%, Δσ² = 85.5%, σ= (3.041, 3.471), μ = (-7.186, 8.640 [54.6%]) -->  μ' = (0.727, -7.913, 7.913)
- Layer 26: [1/8192 filtered] [1/8192 selected] Δ = 425%, Δσ² = 81.0%, σ= (3.390, 4.202), μ = (-6.925, 8.812 [56.0%]) -->  μ' = (0.943, -7.869, 7.869)
- Layer 27: [1/8192 filtered] [1/8192 selected] Δ = 367%, Δσ² = 78.6%, σ= (4.587, 6.006), μ = (-8.345, 12.118 [59.2%]) -->  μ' = (1.887, -10.231, 10.231)
- Layer 28: [1/8192 filtered] [1/8192 selected] Δ = 381%, Δσ² = 79.2%, σ= (4.967, 6.463), μ = (-9.155, 13.327 [59.3%]) -->  μ' = (2.086, -11.241, 11.241)
- Layer 29: [1/8192 filtered] [1/8192 selected] Δ = 337%, Δσ² = 77.1%, σ= (5.460, 7.438), μ = (-9.122, 14.810 [61.9%]) -->  μ' = (2.844, -11.966, 11.966)
- Layer 30: [1/8192 filtered] [1/8192 selected] Δ = 333%, Δσ² = 76.9%, σ= (6.468, 7.391), μ = (-9.324, 16.030 [63.2%]) -->  μ' = (3.353, -12.677, 12.677)
- Layer 31: [1/8192 filtered] [1/8192 selected] Δ = 309%, Δσ² = 75.6%, σ= (7.178, 9.654), μ = (-10.768, 19.141 [64.0%]) -->  μ' = (4.186, -14.955, 14.955)
- Layer 32: [1/8192 filtered] [1/8192 selected] Δ = 288%, Δσ² = 74.2%, σ= (6.315, 9.500), μ = (-9.564, 17.803 [65.1%]) -->  μ' = (4.120, -13.683, 13.683)
- Layer 33: [2/8192 filtered] [1/8192 selected] Δ = 260%, Δσ² = 72.2%, σ= (10.247, 12.558), μ = (-12.651, 24.317 [65.8%]) -->  μ' = (5.833, -18.484, 18.484)
- Layer 34: [1/8192 filtered] [1/8192 selected] Δ = 277%, Δσ² = 73.4%, σ= (9.566, 12.614), μ = (-12.380, 24.846 [66.7%]) -->  μ' = (6.233, -18.613, 18.613)
- Layer 35: [1/8192 filtered] [1/8192 selected] Δ = 292%, Δσ² = 74.5%, σ= (11.524, 13.668), μ = (-15.386, 27.774 [64.4%]) -->  μ' = (6.194, -21.580, 21.580)
- Layer 36: [1/8192 filtered] [1/8192 selected] Δ = 277%, Δσ² = 73.5%, σ= (14.924, 16.169), μ = (-18.578, 33.208 [64.1%]) -->  μ' = (7.315, -25.893, 25.893)
- Layer 37: [1/8192 filtered] [1/8192 selected] Δ = 277%, Δσ² = 73.5%, σ= (17.302, 23.267), μ = (-23.946, 44.281 [64.9%]) -->  μ' = (10.167, -34.113, 34.113)
- Layer 38: [1/8192 filtered] [1/8192 selected] Δ = 265%, Δσ² = 72.6%, σ= (21.397, 29.677), μ = (-29.617, 54.661 [64.9%]) -->  μ' = (12.522, -42.139, 42.139)
- Layer 39: [1/8192 filtered] [1/8192 selected] Δ = 285%, Δσ² = 74.0%, σ= (355.582, 395.628), μ = (-453.536, 815.162 [64.3%]) -->  μ' = (180.813, -634.349, 634.349)

The numbers seem to grow massively for the second to last layer too:

- Layer 38: [1/8192 filtered] [1/8192 selected] Δ = 265%, Δσ² = 72.6%, σ= (21.397, 29.677), μ = (-29.617, 54.661 [64.9%]) -->  μ' = (12.522, -42.139, 42.139)
- Layer 39: [1/8192 filtered] [1/8192 selected] Δ = 285%, Δσ² = 74.0%, σ= (355.582, 395.628), μ = (-453.536, 815.162 [64.3%]) -->  μ' = (180.813, -634.349, 634.349)

To exclude this second to last layer you'll need to use the --control-vector-layer-range llama.cpp command:

         --control-vector-layer-range START END
                                  layer range to apply the control vector(s) to, start and end inclusive

So in this case you would add the command '--control-vector-layer-range 1 38' to your other command line arguments.

NOTE: It may actually turn out to be fine, as even though all models have huge numbers in their last layer like this; including/excluding them didn't make a noticeable difference either way... I just can't test it as both my GPUs are busy creating more control vectors and I didn't notice this until just now.