<jupyter_start><jupyter_text>Case study: Data normalization for unknown confounding factors! Outline1. Description of common use cases.2. Creation of two corrupted datasets (**A** and **B**).3. Application of matrix recovery.    - Dataset **A** with entry sensing.        - Dataset **B** with blind normalization.4. Discussion of results and advantages. 1. Use casesYou have to work with two datasets that are corrupted by an unknown number of confounding factors. Both datasets consist of a matrix of samples and features. For example, customers and products with a rating of satisfaction, or location and time of temperature measurements across the globe, or the change in value of stocks at closing time at the financial markets. Thus, values in the matrix are continuous and can range from negative to positive.Luckily, for dataset **A** you were able to determine the true values for a small subset of matrix entries, for example through quantitative measurement standards or time intensive in-depth analysis. Thus, the recovery of confounding factors is similar to a matrix recovery problem solvable through entry sensing, as the observed values subtracted by the true values give the necessary entries for the bias matrix of confounding factors to be recovered. For dataset **B** it is more challenging, as you were not able to determine any true values for its entries. However, instead you know with certainty that several of the samples and several of the features are strongly correlated and that you are likely to be able to identify those, as the corruption through the confounding factors is not stronger than the underlying signal. Thus the problem can be approached by blind normalization.In order to remove the unknown confounding factors several assumptions have to be satisfied for dataset **A** and **B**. First of all, the to be recovered bias matrix must lie on a sufficiently low dimensional manifold, such as one modelled by a low rank matrix. Secondly, the dataset must satisfy certain incoherence requirements. If both assumptions are satisfied, the otherwise NP-HARD recovery problem can be solved efficiently in the framework of compressed sensing.   2. Creation of dataset **A** and **B**<jupyter_code># Setup of general parameters for the recovery experiment.
n_restarts = 10
rank = 6
n_measurements = 2800
shape = (50, 70) # samples, features
missing_fraction = 0.1
noise_amplitude = 2.0
m_blocks_size = 5 # size of each block
correlation_threshold = 0.75
correlation_strength = 1.0
bias_model = 'image'
# Creation of the true signal for both datasets.
truth = DataSimulated(shape, rank, bias_model=bias_model, correlation_threshold=correlation_threshold, m_blocks_size=m_blocks_size, noise_amplitude=noise_amplitude, correlation_strength=correlation_strength, missing_fraction=missing_fraction)
true_bias = truth.d['sample']['true_bias']
true_bias_unshuffled = truth.d['sample']['true_bias_unshuffled']
true_signal = truth.d['sample']['signal']
true_signal_unshuffled = truth.d['sample']['signal_unshuffled']
true_correlations = {'sample': truth.d['sample']['true_correlations'], 'feature': truth.d['feature']['true_correlations']}
true_correlations_unshuffled = {'sample': truth.d['sample']['true_correlations_unshuffled'], 'feature': truth.d['feature']['true_correlations_unshuffled']}
true_pairs = {'sample': truth.d['sample']['true_pairs'], 'feature': truth.d['feature']['true_pairs']}
true_pairs_unshuffled = {'sample': truth.d['sample']['true_pairs_unshuffled'], 'feature': truth.d['feature']['true_pairs_unshuffled']}
true_directions = {'sample': truth.d['sample']['true_directions'], 'feature': truth.d['feature']['true_directions']}
true_stds = {'sample': truth.d['sample']['true_stds'], 'feature': truth.d['feature']['true_stds']}
# Creation of the corrupted signal for both datasets.
mixed = truth.d['sample']['mixed']
show_absolute(true_signal_unshuffled, kind='Signal', unshuffled=True, map_backward=truth.map_backward)
show_absolute(true_signal, kind='Signal')
show_dependence_structure(true_correlations, 'sample')
show_dependence_structure(true_correlations_unshuffled, 'sample', unshuffled=True, map_backward=truth.map_backward)
show_dependence_structure(true_correlations, 'feature')
show_dependence_structure(true_correlations_unshuffled, 'feature', unshuffled=True, map_backward=truth.map_backward)
show_dependences(true_signal, true_pairs, 'sample')
show_dependences(true_signal, true_pairs, 'feature')
show_independences(true_signal, true_pairs, 'sample')
show_independences(true_signal, true_pairs, 'feature')
show_absolute(true_bias_unshuffled, unshuffled=True, map_backward=truth.map_backward_bias, kind='Bias', vmin=-1.5, vmax=1.5)
show_absolute(true_bias, kind='Bias', vmin=-1.5, vmax=1.5)
# Here the white dots are missing values as common in real data.
show_absolute(mixed, kind='Mixed')
show_dependences(mixed, true_pairs, 'sample')
show_dependences(mixed, true_pairs, 'feature')<jupyter_output><empty_output><jupyter_text>3. Normalization of dataset **A** with entry sensing<jupyter_code># Construct measurements from known entries.
operator = LinearOperatorEntry(n_measurements)
measurements = operator.generate(true_bias)
# Construct cost function.
cost = Cost(measurements['A'], measurements['y'], sparsity=1)
# Recover the bias.
solver = ConjugateGradientSolver(mixed, cost.cost_func, guess_func, rank, guess_noise_amplitude=noise_amplitude, verbosity=0)
results = solver.recover()
# Recovery performance statistics.
recovery_performance(mixed, cost.cost_func, truth.d['sample']['true_bias'], results['estimated_signal'], truth.d['sample']['signal'], results['estimated_bias'])
show_absolute(results['estimated_bias'], kind='Bias', unshuffle=True, map_backward=truth.map_backward_bias, vmin=-1.5, vmax=1.5)
show_absolute(results['guess_X'], kind='Bias', unshuffle=True, map_backward=truth.map_backward_bias, vmin=-0.1, vmax=0.1)<jupyter_output><empty_output><jupyter_text>3. Normalization of dataset **B** with blind normalization<jupyter_code>possible_measurements(shape, missing_fraction, m_blocks_size=m_blocks_size)
# Prior information estimated from the corrputed signal and used for blind recovery.
signal_characterists = estimate_partial_signal_characterists(mixed, correlation_threshold, true_pairs=true_pairs, true_directions=true_directions, true_stds=true_stds)
estimated_correlations = {'sample': signal_characterists['sample']['estimated_correlations'], 'feature': signal_characterists['feature']['estimated_correlations']}
show_threshold(estimated_correlations, correlation_threshold, 'sample')
show_threshold(estimated_correlations, correlation_threshold, 'feature')
# Construct measurements from corrupted signal and its estimated partial characteristics.
operator = LinearOperatorCustom(n_measurements)
measurements = operator.generate(signal_characterists)
# Construct cost function.
cost = Cost(measurements['A'], measurements['y'], sparsity=2)
# Recover the bias.
solver = ConjugateGradientSolver(mixed, cost.cost_func, guess_func, rank, guess_noise_amplitude=noise_amplitude, verbosity=0)
results = solver.recover()
recovery_performance(mixed, cost.cost_func, truth.d['sample']['true_bias'], results['estimated_signal'], truth.d['sample']['signal'], results['estimated_bias'])
show_absolute(results['estimated_bias'], kind='Bias', unshuffle=True, map_backward=truth.map_backward_bias, vmin=-1.5, vmax=1.5)
show_absolute(results['guess_X'], kind='Bias', unshuffle=True, map_backward=truth.map_backward_bias, vmin=-0.1, vmax=0.1)
# Here red dots are the corrupted signal, green the clean signal, and blue the recovered signal.
# Note: missing blue dots indicate missing values.
show_recovery(mixed, results['guess_X'], true_signal, results['estimated_signal'], true_pairs['sample'], signal_characterists['sample']['estimated_pairs'], true_stds['sample'], signal_characterists['sample']['estimated_stds'], true_directions['sample'], signal_characterists['sample']['estimated_directions'], n_pairs=5, n_points=50)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10850878285785474, 'fraction_numerical': 0.007474772642332129, 'mean_word_length': 8.50059171597633, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '146967', 'license': 'MIT', 'n_tokens_mistral': 2374, 'n_tokens_neox': 2231, 'n_words': 679, 'path': 'bcn/examples/Example.ipynb', 'repo_name': 'a378ec99/bcn', 'size': '8027'}