<jupyter_start><jupyter_text>Laplacian Score-regularized Concrete Autoencoders Demo Let import some tools<jupyter_code>from pathlib import Path
from torch.utils import data
from scipy.stats import uniform
from sklearn.datasets import make_moons
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
from omegaconf import OmegaConf
import numpy as np
import torch
from torch.utils import data<jupyter_output><empty_output><jupyter_text>Do not forget to import lscae<jupyter_code>import lscae<jupyter_output><empty_output><jupyter_text>The default config could be found in src/config.yaml, but you can also pass these arguments as here:<jupyter_code>cfg = OmegaConf.create({
    "input_dim": None,          # Dimension of input dataset (total #features)
    "k_selected": 2,            # Number of selected features
    "decoder_lr": 1e-3,         # Decoder learning rate
    "selector_lr": 1e-1,        # Concrete layer learning rate
    "min_lr": 1e-5,             # Minimal layer learning rate
    "weight_decay": 0,          # l2 weight penalty
    "batch_size": 64,           # Minibatch size
    "hidden_dim": 128,          # Hidden layers size
    "model": 'lscae',           # lscae | cae | ls
    "scale_k": 2,               # Number of neighbors for computation of local scales for the kernel
    "laplacian_k": 50,          # Number of neighbors of each pooint, used for computation of the Laplacian
    "start_temp": 10,           # Initial temperature
    "min_temp": 1e-2,           # Final temperature
    "rec_lambda": .5,           # Balance between reconstruction and LS terms
    "num_epochs": 300,          # Number of training epochs
    "verbose": True             # Whether to print to console during training
})<jupyter_output><empty_output><jupyter_text>Read a dataset / build a demo two-moons dataset<jupyter_code>from scipy.stats import uniform
from sklearn.datasets import make_moons
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import numpy as np

def create_twomoon_dataset(n=1200, d=10, noise=0.1):
    """
    Creates two moon clusters in 2D, adding p nuisance features and d noisy copies of one of the original features
    n: size of data (int)
    d: number of nuisance dimensions (int), and number of redundant copies
    noise: noise level (double)
    """
    relevant, y = make_moons(n_samples=n, shuffle=True, noise=noise, random_state=None)
    nuisance = uniform.rvs(size=[n, d])
    data = np.concatenate([relevant, nuisance], axis=1)
    scaler = StandardScaler()
    data = scaler.fit_transform(data)
    plt.scatter(data[:, 0], data[:, 1])
    plt.show()
    return data
X = create_twomoon_dataset()
# You can load your own dataset as below in numpy format
# path = Path(args.data_dir, args.filename)
# X = np.load(path)
# print('Data shape: ', X.shape)

dataset = data.TensorDataset(torch.Tensor(X))
loader = torch.utils.data.DataLoader(dataset, batch_size=cfg.batch_size, shuffle=True, drop_last=True)
cfg.input_dim = X.shape[1]
lscae_model = lscae.Lscae(kwargs=cfg)
selected_features = lscae_model.select_features(loader)<jupyter_output>Epoch 1\300, loss: -0.000, ls loss: -0.00069, recon loss: 0.961
Selection probs: 
  [0.08895258 0.09500632 0.08862478 0.08452889 0.09643977 0.08996537
 0.09198611 0.08099049 0.09697609 0.09512109 0.09342799 0.08812779] 

LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
Epoch 6\300, loss: 0.000, ls loss: -0.00085, recon loss: 0.834
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
Epoch 11\300, loss: -0.000, ls loss: -0.00107, recon loss: 0.829
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
Epoch 16\300, loss: -0.000, ls loss: -0.00141, recon loss: 0.830
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.0010000
LS-CAE learning rate = 0.[...]
{'fraction_non_alphanumeric': 0.0968208092485549, 'fraction_numerical': 0.08598265895953758, 'mean_word_length': 3.6610549943883277, 'pattern_counts': {'":': 16, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '317594', 'license': 'MIT', 'n_tokens_mistral': 1640, 'n_tokens_neox': 1371, 'n_words': 431, 'path': 'example.ipynb', 'repo_name': 'jsvir/lscae', 'size': '4152'}