Iterate excel files and output in one folder in Python

Question: I have a folder and subfolders structure as follows:
D:/src
├─ xyz.xlsx
├─ dist
│  ├─ xyz.xlsx
│  ├─ xxx.zip
│  └─ xxy.xlsx
├─ lib
│  ├─ xy.rar
│  └─ xyx.xlsx
├─ test
│  ├─ xyy.xlsx
│  ├─ x.xls
│  └─ xyz.xlsx

I want to extract all excel files (xls or xlsx) from source directory and subdirectories, drop duplicates based on excel file names and put all the unique files in D:/dst directory. How can I the following result in Python? Thanks.
Expected result:
D:/dst
├─ xyz.xlsx
├─ xxy.xlsx
├─ xyx.xlsx
├─ xyy.xlsx
├─ x.xls

Here is what I have tried:
import os

for root, dirs, files in os.walk(src, topdown=False):
    for file in files:
        if file.endswith('.xlsx') or file.endswith('.xls'):
            #print(os.path.join(root, file))
            try:
                df0 = pd.read_excel(os.path.join(root, file))
                #print(df0)
            except:
                continue
            df1 = pd.DataFrame(columns = [columns_selected])
            df1 = df1.append(df0, ignore_index = True)
            print(df1)
            df1.to_excel('test.xlsx', index = False)

Comment: While all that pandas dataframe stuff you added is kind of distracting and not exactly relevant to your question, it has however exposed—I think—what is called an [XY Problem](http://xyproblem.info/) because it reveals _why_ you want to do this file copying. If I am understand things correctly, then there really is no need to copy all the files to a single folder first—just use the process that finds them to copy to instead drive the concatenation of them you want to do. This will greatly reduce the I/O that needs to be done.
Comment: I think you could do all that via `shutil.copytree()`. See question [Copying specific files to a new folder, while maintaining the original subdirectory tree](https://stackoverflow.com/questions/35155382/copying-specific-files-to-a-new-folder-while-maintaining-the-original-subdirect).
Comment: @ahbon, Any unluck with solving this one yet?
Comment: Thank you for asking. I'll try tomorrow and if I have problem I'll let you know.
Comment: @martineau The solution you've mentioned copy all xlsx and xls files to a new folder D:/dst, but it maintain the original subdirectories. What if I just want put them in a folder and ignore its original subdirectories?
Comment: ahbon: The linked question might not be such a good fit afterall. Your question starts out talking about a "folder and subfolders structure" but you don't show any subfolders (or duplicate excel filenames) in the example source. Also, as far as removing duplicates goes, is the criteria strictly the filename or does file timestamp matter? Please clarify these things.
Comment: Sorry. I updated my question. I have three subfolders dist, lib and test in D:/src, and the filename with extention are files.
Comment: There still are no subfolders (or duplicate excel files) shown under `D:/src`.
Comment: Yes, dist, lib and test are subfolders under D:/src. and xyz.xlsx under D:/src and D:/src/dist are duplicates by name. BTW, I update what I have tried code. Please check., thanks.
Comment: Is my update to your question correct? Also, if there are duplicate excel file names, which one do you want?
Comment: Yes. Much better, thanks for your correction.
Comment: My filenames are quite long and specific, so if the files share same name, keep any one would be OK. But if you need some rules to drop duplicates, let's say keep the one from lowest level and drop the one from upper level.
Comment: The excel files are like the examples in this post, I want to concatenate all excel files into one, please check. https://stackoverflow.com/questions/54347081/locate-dataframe-and-concatenate-based-on-specific-headers-in-python
Comment: You're right. Thanks for your help and remarks. Anyway, I think it will be helpful to manually deal with those excel files in only one folder rather than multiple subfolders. BTW, I'm new in Python, would you please take a look post below if it's possible to concatenate all excel files in one based on specific header columns and ignore the rest?  https://stackoverflow.com/questions/54347081/locate-dataframe-and-concatenate-based-on-specific-headers-in-python
Answer: I think this will do what you want:
import os
import shutil

src = os.path.abspath(r'.\_src')
dst = os.path.abspath(r'.\_dst')
wanted = {'.xls', '.xlsx'}

copied = set()

for root, dirs, filenames in os.walk(src, topdown=False):
    for filename in filenames:
        ext = os.path.splitext(filename)[1]
        if ext in wanted and filename not in copied:
            src_filepath = os.path.join(root, filename)
            shutil.copy(src_filepath, dst)
            copied.add(filename)

Comment: @aneroid: I, too, originally thought that answer would be a good approach, but when I was finally—after much discussion—able to understand what the goal was, decided it wouldn't. Although technically it would be possible by "abusing" the `ignore` option to point where it ignored everything is a little too much in my opinion. All that's need for this question is the ability to walk the file folder system hierarchy, so [`os.walk()`](https://docs.python.org/3/library/os.html#os.walk) would be the more logical (and better) choice.
Comment: Your solution perfectly solved my copying excel files to one folder issues. Thanks a lot.
Comment: ahbon: That's good to hear and you're welcome. Sorry about any distraction my `shutil.copytree()` suggestion may have caused. While it _could_ be done with it, in this situation it would not have been a very good approach.
Comment: Was about to share [this answer you provided](https://stackoverflow.com/a/35161407/1431750) for another question.
Answer: Since you've already got glob.glob, you don't need to also do os.walk, and vice-versa. But since glob only matches one pattern at a time and no way to denote an optional extra 'x' in the extension, you'll either need the glob loop twice - once for each extension; or use glob.glob( 'D:\\src\\*.xls*') which could match '*.xlsm', etc.
For each file matched, use shutil.move:
for file in glob.glob('D:\\src\\*.xls*'):
    shutil.move(file, 'D:\\dst\\' + os.path.basename(file))

With os.walk, you can do each extension check with fnmatch.fnmatch in the same loop:
for root, dirs, files in os.walk('D:\\src'):
    for file in files:
        if fnmatch.fnmatch(file, '*.xls') or fnmatch.fnmatch(file, '*.xlsx'):
            shutil.move(f'{root}\\{file}', f'D:\\dst\\{file}')
            # shutil.move(root + '\\' + file, 'D:\\dst\\' + file)

Comment: Why bother calling `os.walk()` only to then quit after first iteration? I think the question is unclear and have asked the OP for clarification.
Comment: Thanks @aneroid. I think shutil.copy would be better for without changing original data, what do you think?
Comment: Thanks for your help. Your solution only copy D:/src - xyz.xlsx to D:/dst, not excel files from its subfolders.
Comment: @martineau I gathered from the OP's code that he/she wanted to stop after the first folder/iteration. Even with glob, we'd need two loops for each extension, and without `recursive=True`. So figured `os.walk()` could simplify that. Anyway, from OP's post, they did want to do subdirs.
Comment: @ahbon Yes, you can use `copy` instead of `move` if you want the original data to remain in `'D:\src'`. I've removed the `break` so that loop doesn't stop after the first iteration. From your code, it looked like you only want to use the top-level. Also, if you are recursively copying, then `f'D:\\dst\\{file}'` will need to be modified to have the parent folders of each file. Otherwise it will copy flat, without the subdirs. Look at [`shutil.copytree()`](https://docs.python.org/3/library/shutil.html#shutil.copytree) for that.
Comment: aneroid: I guess we disagree in our understanding of what the OP wants. It sounds to me like every subdirectory is going to need to be visited in order to find all the files that will need to be copied.
Comment: martineau: Indeed, we'd need to recurse [sic]. But I got the impression the OP had a flat folder structure at 'src'. That was incorrect since it's now clear that the OP had & wanted subdirs too.


LMM model with nested design and random slopes: how to fit the autocorrelation term?

Question: I am currently running a linear mixed model with a nested design and random slopes.
For example, let's imagine some monthly captures of wild rabbits in kilograms in 5 sites during 21 years:
site<- rep(rep(c("Golden Cave","Ringo's place","Damned Dam","Knockampton","Easy Fuzzy"),each=12),21)
year <- rep(2000:2020, each=12*5)
month <- rep(seq(1,12),21*5)
rabbit_captures <- rnorm(12*21*5, 50, 10)
dataset <- as.data.frame(cbind(site,year,month,rabbit_captures))
dataset$rabbit_captures <- as.numeric(dataset$rabbit_captures)

And the corresponding model:
library(nlme)
library(MASS)

model_lme <- lme(fixed = log(rabbit_captures) ~ site,
                 random = ~ site|year/month,
                 data = dataset, method = "ML",
                 control = lmeControl(opt = 'optim'))

With this simulated distribution of rabbit_captures, the model does not converge, sorry I tried, but it still corresponds to the design of my experiment. In other words, with a more complex dataset, this model runs properly. However, I am uncomfortable doing spatial modeling without considering time autocorrelation...
So here is my question:
How to consider time autocorrelation within the model?
Knowing that:

correlation = corAR1(value = 0.9, form = ~ site|year/month) is an incorrect formulation. Variable "Site" is made of factors and "A covariate for this correlation structure must be integer valued."

correlation = corAR1(value = 0.9, form = ~ 1|year/month) should work (with my example, it won't) but I really don't know if it makes sense statistically.
Answer: Typically, you'd use something like corCAR1(form = ~ date | site) or corAR1(form = ~ months_since_start_of_timeseries | site). This specifies time as an auto-correlation co-variate and groups by site. If your time series are perfectly regular and the data sorted by time, you could use corAR1(form = ~ 1 | site).
However, I suggest that you do not use a mixed model and include time as a fixed effect. Since this would not be a linear effect, you could use a GAM.
#never use `as.data.frame(cbind(...))`, do this instead:
dataset <- data.frame(site,year,month,rabbit_captures)

library(mgcv)
dataset$site <- factor(dataset$site)
dataset$yearmon <- (dataset$year - min(dataset$year)) * 12 + dataset$month

fit <- gam(log(rabbit_captures) ~  site + s(yearmon, by = site), data = dataset)
gam.check(fit)
abline(0, 1)

summary(fit)
#Family: gaussian
#Link function: identity
#
#Formula:
#log(rabbit_captures) ~ site + s(yearmon, by = site)
#
#Parametric coefficients:
#                   Estimate Std. Error t value Pr(>|t|)
#(Intercept)        3.886624   0.013556 286.714   <2e-16 ***
#siteEasy Fuzzy    -0.005637   0.019171  -0.294    0.769
#siteGolden Cave    0.008913   0.019171   0.465    0.642
#siteKnockampton   -0.023536   0.019171  -1.228    0.220
#siteRingo's place  0.001926   0.019171   0.100    0.920
#---
#Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#
#Approximate significance of smooth terms:
#                               edf Ref.df     F p-value
#s(yearmon):siteDamned Dam    5.826  6.984 1.116   0.338
#s(yearmon):siteEasy Fuzzy    1.000  1.000 2.710   0.100 .
#s(yearmon):siteGolden Cave   2.114  2.638 1.390   0.316
#s(yearmon):siteKnockampton   5.523  6.667 1.597   0.138
#s(yearmon):siteRingo's place 1.000  1.000 0.814   0.367
#---
#Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#
#R-sq.(adj) =  0.0132   Deviance explained = 2.84%
#GCV = 0.047072  Scale est. = 0.046307  n = 1260

plot(fit, pages = 1)

The time smoothers should take care of auto-correlation. You could also use an additional time smoother independent of site to model a general, site-independent trend.


how to check messages exists

Question: I want to check is there any messages in my template
in drupal 7 there was a global variable in page template variables messages
now in drupal 8 I want to check  is there any message  but messages variable in page--front.html.twig is always empty, alternative way is check  region highlightes( because  status messages block is in highlighted region)
but below  statement always return TRUE,
    {% if page.highlighted %}
      {% block highlighted %}

{{ page.highlighted }}

      {% endblock %}
    {% endif %}

is there any way to check that messages exists in page in drupal 8?
Comment: Just a suggestion...cant we create a custom  variable.... and set it to true or false depending on message block ???
Comment: @ShreyaShetty, that would be difficult, because the page template can't look in the future.
Answer: While the page.highlighted variable contains the block data (mytheme_messages) for message display, the block rendering data doesn't contain the messages themselves, so there's no way to check for message existance there.
However, you can inject a custom variable based on the current message queue from drupal_get_messages().
For example, in your MYTHEME.theme:
/**
 * Implements hook_preprocess_page() for page.html.twig.
 */
function MYTHEME_preprocess_page(array &$variables) {
  // Inject a flag based on the current message queue.
  $variables['available_messages'] = !empty(drupal_get_messages(NULL, FALSE));
}

And within your page.html.twig:
{% if available_messages == true %}

{% endif %}

Answer: In Drupal 9, use the Messenger service.
function MYTHEME_preprocess_page(array &$variables) {
  $variables['available_messages'] = !empty(\Drupal::messenger()->all());
}

Comment: This works great in general. But in my case, it didn't catch some webform validation messages. Maybe because I'm using a webform referenced from a paragraph.

how to check messages exists


How to use the REQUIRES parameter in a MODULE creation - angularjs

Question: The syntax for usage of a module is:
angular.module(name, [requires], [configFn]);

The Type for the [requires] parameter is:
!Array.<string>=

I have no idea what that means.  What does the exclamation point signify, that it's NOT an Array?  Why is there an equal sign at the end?  What is the period for?
Looking at another Stackoverflow question, the [requires] parameter is used to list other modules that this module uses.  So, is this sort of like an Include statement, that puts another module inside of this module?
Answer: It is the Google Closure Compiler Type Expressions syntax.

Array.<string> means an array of strings
! at the beginning means it cannot be null
= at the end means it is optional

Combine them together and we got: !Array.<string>= means it can be

an array of strings
an empty array
undefined (optional)

But it cannot be null


Is there a way to isolate multiple index points in a dataset and isolate a time window around that index point in R?

Question: I have a dataset with patients who were under observation for 72 hours. Patient's oxygen levels were measured every 4 seconds but some observations had to be removed due to issues with accuracy of the data. As a result, patients have a varying number of observations.
While patients were observed, they underwent various interventions. The objective of my analysis is to assess whether interventions affected the patient's oxygen levels or not. Therefore, I am doing a comparison of the oxygen levels pre-intervention, during intervention and post-intervention.
While the actual analysis isn't too difficult, I am having a hard time subsetting the data. For example, I only want observations 300 seconds prior to the start of an intervention and 300 seconds post intervention. I have to take into account the fact that an individual may have multiple interventions over the course of the time window and there are multiple subjects.
I have provided some sample code below to generate a dataset but please let me know if I'm missing anything.
id <- rep(c(1,2,3), each = 1000)
intervention <- c(rep(0,200), rep(1,10), rep(0,153), rep(0,5), rep(0,284), rep(0,20), rep(0,159), rep(0,23), rep(0,146),
                  rep(0,123), rep(1,23), rep(0,356), rep(1,8), rep(0,234), rep(1,23), rep(0,233),
                  rep(0,345), rep(1,12), rep(0,48), rep(1,15), rep(0,74), rep(1,4), rep(0,233), rep(1,82), rep(0,187))
final <- data.frame(id, intervention)
final <- final %>%
  group_by(id) %>%
  mutate(time = row_number() * 4)

So far, I have tried this method but I was only able to isolate single observations 5 mins pre and post an intervention and not all observations between those time windows (ie, the single observation 5mins prior to start of an intervention and the single observation 5 mins post an intervention but not all the observations in between these three points)
data <- final4 %>%
  filter(intervention == 1) %>%
  mutate(five_mins_after = time + 300, #5 mins after intervention
         five_mins_before = time - 300) %>% #5 mins before intervention %>%
  filter(id == "1")

data2 <- final4 %>%
  filter(intervention == 0,
        id == "1")

data_after <- data %>%
  dplyr::select(five_mins_after)

data_before <- data %>%
  dplyr::select(five_mins_before)

data3 <- merge(data2, data_after, by.x = "time", by.y = "five_mins_after")
data4 <- merge(data2, data_before, by.x = "time", by.y = "five_mins_before")

final <- final %>%
  dplyr::bind_rows(data3) %>%
  dplyr::bind_rows(data4)

Please let me know if you need any additional information and thanks for your time!
PS: Apologies if I missed anything, first time asking for help here
Answer: Here is the answer. Although long, it worked fine for gathering times 300 seconds before and 300 after the beggining of an intervention.
Let me know if you'd like further explanation or if I have misunderstood anything.
library(magrittr)
library(tidyverse)

### Sample code
id <- rep(c(1,2,3), each = 1000)
intervention <- c(rep(0,200), rep(1,10), rep(0,153), rep(0,5), rep(0,284), rep(0,20), rep(0,159), rep(0,23), rep(0,146),
                  rep(0,123), rep(1,23), rep(0,356), rep(1,8), rep(0,234), rep(1,23), rep(0,233),
                  rep(0,345), rep(1,12), rep(0,48), rep(1,15), rep(0,74), rep(1,4), rep(0,233), rep(1,82), rep(0,187))
final <- data.frame(id, intervention)
final <- final %>%
  group_by(id) %>%
  mutate(time = row_number() * 4)

### Start of data processing to get wanted observations

# Order it by id and time
final %<>% arrange(id, time)

# Loop over the unique ids
obs_to_keep <- list()
for(i in unique(final$id)) {

  # Get starts of treatment
  time_zero_intervention <- final %>%
    filter(id == i & intervention == 0) %>%
    select(time)

  # Obtain all times after zero interventions, that could be intervention == 1
  time_plus_4 <- time_zero_intervention$time + 4

  # Where in the times after 0 intervention there is a 1 intervention
  starts_of_interventions <- final %>%
    filter(id == i & time %in% time_plus_4) %>%
    filter(intervention == 1)

  # Loop over each one of the times where intervention starts
  all_times <- list()
  for(n in 1:length(starts_of_interventions$time)) {
    # Gather 300 secs prior and post
    time_300_before <- starts_of_interventions$time[n] - 300
    time_300_after <- starts_of_interventions$time[n] + 300

    # Filter for observations in this interval
    all_times[[n]] <- final %>%
      filter(id == i) %>%
      filter(time >= time_300_before & time <= time_300_after)
  }
  if(length(all_times) == 1){
    obs_to_keep[[i]] <- as.data.frame(all_times)
  }
  else {
    obs_to_keep[[i]] <- do.call(rbind, all_times)
  }
}

# Make a data frame from the list
df <- do.call(rbind, obs_to_keep)

# Order it by id and time
df %<>% arrange(id, time)

Comment: Hi, thanks a lot for your help! Was just wondering, the way you have it coded, would it find the first instance of when the event goes from 0 to 1 or is it coded so it captures the first instance of 0?
Comment: It will find the first instance of 1, and also all of the switches from zero to 1, which I had understood as the starts of the multiple interventions an individual has. Is that correct, or do the multiple interventions would be numbers different than 1 (2, for example)?


Selecting two columns from different tables in one result using MySQL

Question:

Selecting two columns from different tables in one result using MySQL.
how can make it in MySQL and CakePHP
Comment: [JOIN](https://en.wikipedia.org/wiki/Join_%28SQL%29).
Comment: What version of CakePHP are you using and what have you tried so far? As it stands there is not enough information for a proper answer.
Comment: Query in CakePHP : $options = array( 'fields' => array( 'Usercat.user_id',), 'joins' => array( array('conditions' => array('Userarea.area_id' => $area_id,), 'table' => 'user_area', 'alias' => 'Userarea', ), ), 'conditions' => array( 'Usercat.cat_id' => $cat_id, ), 'group' => array( 'user_id', ), 'contain' => array( 'Userarea', ), ); $data = $this->Usercat->find('all', $options); @drmonkeyninja
Comment: @GhalySaqqal in future specify the version of CakePHP you are using. From your last comment I assume it is either CakePHP 1.x or 2.x, but you would write a very different query for CakePHP 3.x. Your question is a bit vague and as a result you haven't been supplied with the best answers, although appear to have solved it yourself. It would perhaps be a good idea to post your own solution and mark it as the answer seeing as it is better than the one you've currently accepted!
Comment: @drmonkeyninja this query in CakePHP 2.x
Answer: The SQL query would be
$sql = "SELECT t1.user_id FROM table1 AS t1
                   JOIN table2 AS t2
                   ON t2.area_id = 3
                   WHERE t1.cat_id = 11"

$this->query($sql);

See here for the documentation
Comment: and you can make it in CakePHP :
$options = array(
    'fields' => array( 'Usercat.user_id',),
     'joins' => array(
             array('conditions' => array('Userarea.area_id' => $area_id,),
              'table' => 'user_area',
              'alias' => 'Userarea',
             ),
         ),
         'conditions' => array(
             'Usercat.cat_id' => $cat_id,
         ),
         'group' => array(
               'user_id',
         ),
         'contain' => array(
             'Userarea',
          ),
 );
$data = $this->Usercat->find('all', $options);
Comment: Please don't write queries like this in CakePHP. This could all be done using CakePHP's ORM which would lead to much safer code.
Answer: You can use INNER JOIN between tables, the query will be:
SELECT t1.*, t2.* FROM table1 t1 INNER JOIN table2 t2 ON t1.user_id = t2.user_id

CakePHP has his owns ORM, you can check it out here:
Simple query.
http://book.cakephp.org/3.0/en/orm/query-builder.html#selecting-rows-from-a-table
Associations:
http://book.cakephp.org/3.0/en/orm/query-builder.html#loading-associations
Regards.
Comment: SELECT t1 . * , t2 . *
FROM table1 t1
INNER JOIN table2 t2 ON t1.user_id = t2.user_id
WHERE cat_id =2
AND area_id =3

I got empty result when I use it
Comment: Yeah, that is because your user_id does not have a cat_id 2. Try LEFT JOIN ... but i think the problem is not the query. The problem is the table structure.
Answer: SELECT user_id
FROM table1, table2
WHERE table1.cat_id = 2 and table2.area_id = 3

but i am a litle bit confused what you want... :S


Ao tentar criar um objeto usando `this` o console diz que a propriedade é undefined

Question: Estou aprendendo a usar objetos em JavaScript, e me deparei com um comportamento imprevisto no meu objeto. No caso quando tento armazenar um elemento via DOM dentro de uma propriedade o console diz que a propriedade é indefinida:
TypeError: this.seletorTimer is undefined [Learn More]

Imagem do erro no console
let Inicial = {
  seletorTimer: document.getElementById("TimerInicial"),
  letrasRegex: /[A-Z]+/,
  maximo: 59,
  SemLetras: this.seletorTimer.addEventListener('keyup', () => {
    if (document.getElementById("TimerInicial").value.match(this.letrasRegex)) {
      this.seletorTimer.value = "";
    }
  }),
  numeroLimite: document.getElementById("TimerInicial").addEventListener('keyup', () => {
    if (this.seletorTimer.value > this.maximo) {
      this.seletorTimer.value = 59;
    }
  }),
};

Código online no jsfiddle
O que eu estou tentando fazer nesse código é basicamente não permitir que o usuário digite letras ou símbolos (apenas números) e quando o usuário colocar um valor maior que 59, o evento reescreva o valor do input para 59.
Confesso que não sei se é possível fazer um objeto dessa forma porque a questão do escopo fica um pouco confusa para mim.
Comment: Ola @Daniel, você poderia melhorar sua pergunta ( *eu não consegui entender* -- [ask] ), vale apena você colocar o erro que apareceu no console, isso pode ajudar a comunidade a solucionar sua dúvida. =P -- não esqueça de dar uma olhada em nosso [Tour]  =D
Comment: @IcaroMartins não tenho o hábito de perguntar no Stack, tentei dar uma caprichada na pergunta, espero que esteja mais legível kkkkk
Comment: Também adicione seu código aqui como texto e não como imagem ou link
Comment: Verdade obrigado @Costamilam!
Comment: @DanielKenzi, alterei o titulo da sua pergunta, pois acredito que dessa forma possa ajudar outras pessoas. Se sentir que eu alterei o sentido da sua pergunta original acesse esse [link *(revisions)*](https://pt.stackoverflow.com/posts/381442/revisions) e clique em **reverter** na versão que você achar melhor. =D
Comment: @IcaroMartins tá excelente hahaha. Depois da explicação que você deu realmente faz mais sentido, brigadão!
Answer: Parte de seus problemas e o entendimento de algumas coisas em seu código, o primeiro que eu vejo é o seu entendimento do this.
O this é o objeto de contexto de execução, sua utilização é um pouco complicada de entender mesmo, mas uma vez que você entenda sua utilização ele passa a ser muito útil, então vou tentar te explicar abaixo usando comentários no código:

/// ; neste momento o `this` === `window`
console.log( 'this é igual a window ? ',  this === window );

var elTimerInicial = document.getElementById("TimerInicial");

let Inicial = {
  seletorTimer: elTimerInicial,
  letrasRegex: /[A-Z]+/,
  maximo: 59,

  /// ; Neste momento o this ainda é `window` e nele não existe
  /// ; a propriedade `selectorTime`, você esta pensando que nesse
  /// ; momento o `this` era o objeto `Inicial` e por isso estava chamando
  /// ; `this.seletorTimer`.

  /// ; Outra coisa que acontece é que neste momento o objeto
  /// ; `Inicial` ainda não foi criado, ou seja, você também não
  /// ; conseguiria acessar usando `Inicial.selectorTime`

  SemLetras: elTimerInicial.addEventListener('keyup', function(){

      /// ; Agora complica um pouco, note que eu troquei seu
      /// ; `Arrow Function` por um `Function normal`.
      /// ; Quando esse evento é disparado o `this` aqui vai ser o
      /// ; elemento que disparou o evento ou seja, nesse caso o `elTimerInicial`

      console.log( 'Function Normal,  this é igual a window ? ', this === window, this );

  }),

  numeroLimite: elTimerInicial.addEventListener('keyup', () => {

      /// ; Aqui eu mantive o `Arrow Function` para você ver a diferença
      /// ; ele NÃO cria contexto, então aqui dentro
      /// ; o `this` é o mesmo que era fora dele quando ele
      /// ; foi criado, ou seja, o `window`.
      /// ; Esse é um dos motivos que levaram ao desenvolvedores
      /// ; a criar esse novo tipo de função, veja um
      /// ; exemplo prático na sessão 'alguns problemas comuns com this'.

      console.log( 'Arrow Function,  this é igual a window ? ', this === window );

  }),
};
<!DOCTYPE html>

    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <title>Guessing game</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
    <link rel="stylesheet" type="text/css" media="screen" href="style.css">
    <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.8.1/css/all.css" integrity="sha384-50oBUHEmvpQ+1lW4y57PTFmhCaXp0ML5d60M1M7uH2+nqUivzIebhndOJK28anvf" crossorigin="anonymous">

PROJETO TIMER

        <input type="number"  placeholder="HH" > :
        <input type="number"  placeholder="MM" > :
        <input type="number"  placeholder="SS" >

Outro problema é que em seu código você esta tentando pegar value do elemento #TimerInicial que é uma
 e não o <input> que esta recebendo o valor.
Então no código abaixo vamos tentar usar o this e mudar seu objeto.

var elTimerInicial = document.getElementById("TimerInicial");

/// ; ao invés de criar um objeto vamos usar o próprio `elTimerInicial`

elTimerInicial.letrasRegex = /[A-Z]+/;
elTimerInicial.maximo = 59;
elTimerInicial.semLetras = function( input ){

    /// ; Quando você chamar essa função desta forma `elTimerInicial.semLetras( )`
    /// ; `this` aqui sera o próprio `elTimerInicial`

    console.log( '2- semLetras - this:' , this, ' input:' , input );

    if( input.value.match(this.letrasRegex) ){
        input.value = "";
    }

};
elTimerInicial.numeroLimite = function( input ){

     console.log( '3- numeroLimite - this:' , this, ' input:' , input );

     if ( input.value > this.maximo ) {
          input.value = 59;
     }

};

/// ; Como só os `input`s são filhos do `elTimerInicial` vamos usar a propriedade
/// ; `children` do `elTimerInicial` para pegar seu filhos e adicionar o evento neles
/// ; depois vamos usar o `this` novamente dentro da função:

for( var i = 0, max = 3; i < max ; i++ )
{

    elTimerInicial.children[ i ].addEventListener('keyup', function(){

        /// ; Quando essa funcao for chamada o `this` aqui dentro sera o `input`
        /// ; que disparou o evento.

        /// ; vamos usar o `parentNode` para pegar o pai desse `input` nesse caso o
        /// ; `elTimerInicial` e vamos usar as funções que
        /// ;  adicionamos a esse elemento `elTimerInicial`

        console.log( '1- keyup this:' , this, 'this.parentNode', this.parentNode );

        this.parentNode.semLetras( this );
        this.parentNode.numeroLimite( this );
        /// ;    ^          ^          ^ this === input
        /// ;    |          função que adicionamos ao `elTimerInicial`
        /// ;    elTimerInicial

    });

}
<!DOCTYPE html>

    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <title>Guessing game</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
    <link rel="stylesheet" type="text/css" media="screen" href="style.css">
    <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.8.1/css/all.css" integrity="sha384-50oBUHEmvpQ+1lW4y57PTFmhCaXp0ML5d60M1M7uH2+nqUivzIebhndOJK28anvf" crossorigin="anonymous">

PROJETO TIMER

        <input type="number"  placeholder="HH" > :
        <input type="number"  placeholder="MM" > :
        <input type="number"  placeholder="SS" >

Alguns problemas comuns com o this
Fato é que quando você esta usando this a forma como você chama uma função pode alterar o seu contexto.
Imagine que eu tenho um objeto com o nome mObject e nesse objeto existe o método minhafuncao que utiliza o this esperando que ele seja o mObject.
Imagine que por algum motivo ao invés de chamar o mObject.minhafuncao() assim, ou seja de forma direta, você precise chamar essa função dentro de um setTimeout, passar ela como uma callback ou associar essa função a uma variável para poupar tempo digitando etc... , como no exemplo abaixo:

var mObject = {
   minhafuncao:function(){
       console.log( 'this é:', this===window ? 'window' : this );
   }
};

/// ; Exemplo callback
function chamaCallback( callback ){
    callback();
}

chamaCallback( mObject.minhafuncao );

/// ; Exemplo associar a função a uma variável
var func = mObject.minhafuncao;
func();

/// ; Exemplo setTimeout
setTimeout( mObject.minhafuncao, 1 );

Se isso ocorrer o você ira perceber que o this que esta dentro da função não vai ser o mObject ele será o window ou algum outro contexto não esperado, agora você já deve ter notado que isso ocorreu porque o contexto da chamada mudou.
Isso pode ser solucionado de forma simples com o uso do apply, call, bind para manipular o contexto ou no caso do exemplo do setTimeout uma simples function já resolveria, veja o código abaixo:

var mObject = {
   minhafuncao:function(){
       console.log( 'this é:', this===window ? 'window' : this );
   }
};

/// ; Exemplo associar a função a uma variável
var func = mObject.minhafuncao;

func.call( mObject );
/// ; func.call( mObject, parametro1, parametro2, ...  );
/// ;            ^        ^ os parâmetros da função começa aki
/// ;            contexto, isso sera o `this` dentro da função

/// ; Exemplo setTimeout
setTimeout( function(){

    mObject.minhafuncao();
    /// ; o contexto dessa chamada voltou a ser o `mObject`

}, 1 );

 /// ; Exemplos com o `bind`
 var novaFunc = mObject.minhafuncao.bind( mObject );
 /// ; bind retorna a função com o contexto sendo o parâmetro

 /// ; Agora quando eu chamar a função `novaFunc` ela vai manter o contexto
 /// ; Então essa chamada o `this` sera o `mObject`
 novaFunc();

 /// ; Chamando pela callback
 function chamaCallback( callback ){
     callback();
 }
 chamaCallback( novaFunc );

 /// ; O mesmo ocorre com o `setTimeout` o contexto da função vai ser mantido
 /// ; por causa do `bind` então o `this` sera o `mObject`
 setTimeout( novaFunc, 1 );

 /// ; Ou seja com `bind`, `call`, `apply` você vai conseguir manipular o
 /// ; contexto e modificar como quiser veja esse outro exemplo:
 novaFunc = mObject.minhafuncao.bind( {name:"### Mudei o contexto totalmente ###"} );
 setTimeout( novaFunc, 10 );

Um outro exemplo, imagina que você tem uma
 em que você colocou um evento onClick, e você vai utilizar o this dentro do evento para alterar o style, className, etc... dessa
 e depois de x tempo vai voltar ao normal, ou seja, utilizar um setTimeout dentro do evento.

/// ; criar a div
var div = document.createElement('DIV');

/// ; adicionar ele ao body da pagina
document.body.appendChild( div );

/// ; colocar o texto e mudar style `bgcolor`
div.innerHTML = 'MEU TESTE';
div.style['backgroundColor'] = 'red';

/// ; adicionar o evento de onclick
div.onclick = function(){

    /// ; `this` quando esse evento for disparado sera a `<div>`

    this.style['backgroundColor'] = 'green';
    this.innerHTML = "eu cliquei aki";

    /// ; agora dentro da função do setTimeout o `this` será a `window`
    /// ; para solucionar esse problema alem das soluções já mostradas
    /// ; podemos associar o `this` a uma variável local dentro do
    /// ; evento e utilizar essa variável dentro do `setTimeout` veja abaixo

    var self = this;

    setTimeout( function(){

       console.log( 'settimeout this é:', this === window ? 'window':'outra coisa');

       self.style['backgroundColor'] = 'red';

    }, 600 );

    /// ; Outra solução seria o uso do `Arrow function`
    /// ; isso porque como eu já disse antes um dos
    /// ; objetivos dele é de NÃO criar contexto, então
    /// ; o `this` dentro dele será o mesmo que era fora dele

    setTimeout( ()=>{

       console.log( 'arrow function this é:', this === window ? 'window': this);

        this.innerHTML = "MEU TESTE";

    }, 600);
}

Outros links que podem ajudar

Links SOpt:
"this" uma referencia confiavel?
Qual a diferença entre $(this) e $this e this?
Para que serve o paramêtro THIS?
Links Externos:
codigosimples.net - Saiba mais sobre o "this" utilizado javascript
Referencias da Mozilla Developer Docs:
Arrow functions, this, children, apply, call, bind
Comment: Nossa me esclareceu muito sobre o uso do this! E não sabia que dava para inserir propriedades dessa forma, muito legal. A única dúvida que me surgiu é se o loop no final é uma boa prática, porque eu estava tentando usar objetos justamente para evitar de criar variáveis globais ou deixar addeventListeners "soltos". Bom muito obrigado pela resposta!
Comment: O `for` no fim pode ser melhorado, uma melhoria seria retirar a criação de função de dentro dele `addEventListener('keyup', function(){ ... });`, criar a função fora do `for` por exemplo `function eventoKeyUp(){ ... }` ou `elTimerInicial.eventoKeyUp=function(){ ... }`, e mudar para `addEventListener('keyup', eventoKeyUp)` ou `addEventListener('keyup', elTimerInicial.eventoKeyUp)`


Can I use "simr" package for power analyses on ANOVA models?

Question: simr is an R package for conducting power analyses on linear mixed effects models. One might compute power for a model with:
library('simr')
fit = lmer(data=data, formula = x ~ y + (y | z)
powerSim(fit, nsim=200)

My question is this: does it make sense to use this package assess the power of models that were not built with lmer? For example, on a simple ANOVA:
fit2 = aov(data=data, formula=x~y)
powerSim(fit2, nsim=200)

Just to clarify, I know that this CAN be implemented in R (i.e. powerSim(fit2) does not cause an error, and returns output similar to that of powerSim(fit) ), but I'd like to know if using powerSim on an ANOVA model is theoretically flawed.
Comment: This is off topic here, but I can't think of any reason it shouldn't work. There may be other packages that work faster or have more options or something for regular ANOVA.
Answer: The article SIMR: an R package for power analysis of generalized linear mixed models by simulation by Green and MacLeod, which is devoted to the simr package,  states the following:

"simr is designed to work with any linear mixed model (LMM) or GLMM that can be fit with either lmer or glmer from lme4. This allows for a wide range of models with different fixed and random effect specifications. Linear models and generalized linear models using lm and glm in base r are also supported, to allow for models with no random effects."
"Version 1.0 of simr is designed for any LMM or GLMM fitted using lmer or glmer in the lme4 package, and for any linear or generalized linear model using lm or glm, and is focussed on calculating power for hypothesis tests."

Since the help function for the aov function in R states that this function "provides a wrapper to lm for fitting linear models to balanced or unbalanced experimental designs", it makes sense to use the simr package to perform power calculations for tests of hypotheses involving parameters in models which can be fitted with either the aov or the lm functions in R.
You can find the article here: https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12504.
Comment: This is exactly what I was looking for. Thanks so much!
Comment: You're welcome, @Lyam!


1 node appearing less in recursive implementation of Tree in C++

Question: I have created an implementation of a complete binary-tree in C++ using class and recursion. At depth = 6 it is supposed to have 63 nodes. But in the output I can see only 62 nodes. Can someone please point out the problem?
class node{
public:
    node *left, *right, *root, *it;
    int data,m;

void create(){
        root = new node;
        it = root;
        root->data = 1;
        const int n = 6;          // Depth to be passed
        addchildren(root,n); }

void addchildren(node *x,int z)   // Recursion (z = No. of levels)
    {

        if (z==1)
            return;
        else
        {
            it->left = new node;
            it->left->data = 1;
            it->right = new node;
            it->right->data = 1;
            addchildren(it->left,z-1);
            addchildren(it->right,z-1);
            return;
        }
    }

void display(node *x,int z)
    {

        if (z==1)
            return;
        else
        {
            cout<<it->left->data;
            cout<<it->right->data;
            display(it->left,z-1);
            display(it->right,z-1);
            return;
        }
    }
};
int main()
{
    node A;
    A.create();
    A.display(A.root,6);
}

Output : 11111111111111111111111111111111111111111111111111111111111111 (62 1s)
Comment: [How to debug small programs](https://ericlippert.com/2014/03/05/how-to-debug-small-programs/)
Answer: Take a look at your display function:
void display(node *x,int z) {
        if (z==1)
            return;
        else
        {
            cout<<it->left->data;
            cout<<it->right->data;
            display(it->left,z-1);
            display(it->right,z-1);
            return;
        }
    }
};

Let's Rubber Ducky this function:

If we're on the lowest row of the tree, just return instead of displaying it.
Else, display out two children on the row below us, and then have them display their children.

This is an...interesting way of writing a display for a tree. I'd strongly suggest just printing the node you're currently on and then letting its children print themselves. This convoluted way of printing children yourself is causing your problem. After all, who is the root node a child of? Nobody! So who prints the root node? Nobody!!
In addition to this, you really really really need to set the left and right pointers in your nodes. As of now you have a bunch of wild pointers. I'd suggest adding a constructor to your node class to do this for you.
And lastly: why are you testing with a depth of 6? If I had to count that many 1's every time I ran the program to test I'm pretty sure I'd be pulling my hair out in the space of an hour. Start with a tree of depth 1 (which should fail with your current code!). If that works, go to a depth of 2. At most I'd test with a depth of 3. After all, the case of depth 6 vs. 3 is probably not at all different!
Comment: Thank you for the very detailed answer sir. I didn't get the part about wild left and right pointers. Are you talking about the last level of nodes?
Comment: @SiriusBlack See [the Wikipedia article on Wild Pointers](https://en.wikipedia.org/wiki/Dangling_pointer) for more. Essentially, uninitialized pointers cause UB which is baaaaaad. I'd also suggest looking at some implementations of a BST print to see how they do it without passing in depth.


angularJS make binary data image from base64 and send as image file to server

Question: I'm using this tool to crop images in my AngularJS app: https://github.com/fengyuanchen/cropper
and then I try to store data to server, but now I send base64 code... How could I send normal cropped image binary code, not as base64? is it real?
my code for uploading is such:
var addImage = function(files) {
      var deferred = $q.defer();
      var fd = new FormData();

      var blob = new Blob([files], { type : "image/png"});
      fd.append("file", blob);

      $http.post(settings.apiBaseUri + "/files", fd, {
          withCredentials: true,
          headers: {
            'Content-Type': undefined
          },
          transformRequest: angular.identity
        })
        .success(function(data, status, headers, config) {
          url = data.Url;
          deferred.resolve(url);
        })
        .error(function(err, status) {

        });
      return deferred.promise;
    };

and here I send base64 code:
  $scope.photoChanged = function(files) {
    $scope.oldImg = angular.element(document.querySelector('#avatar-id'));
    $scope.files = files;
    var reader = new FileReader();
    reader.onload = function(e) {
      $scope.imagecontent = e.target.result;
    };
    reader.readAsDataURL(files[0]);
  };

and
  $scope.setCroppedData = function(data) {
    $('#avatar-id').attr('src', data);
    $scope.nImg = new Image();
    $scope.nImg.src = data;  // data - is base64 image code
  }
  ...
  uploaderService.addImage($scope.nImg.src).then(function(url) {
    $scope.person.AvatarUrl = url;
    $scope.updateContactQuery();
  });
  ...

how could I send like I do without any croping: send image file, so that i could go via link in browser and see image, but not a byte64 code???
Comment: Wouldn't it be easier to send base64 and use Image manipulation functions at server end to create image file and save?
Comment: @geekman no - server will not be changed...
Comment: I did not understand what exactly you want to do. you want to save the image to server or simply show it on page?
Comment: Found this genius solution : http://blog.danguer.com/2011/10/24/base64-binary-decoding-in-javascript/
Answer: You can wrap the base64 data in ArrayBuffer then convert to binary Image data like:
var binaryImg = atob(base64Image);
var length = binaryImg.length;
var arrayBuffer = new ArrayBuffer(length);
var uintArray = new Uint8Array(arrayBuffer);
for (var i = 0; i < length; i++) {
    uintArray[i] = binaryImg.charCodeAt(i);
}

I suppose uintArray is what you want.
Comment: Please add query example
Comment: what do you mean by "query example"? It seems your question is `how to send normal cropped image binary`
Comment: i mean post http query. Will it be ok to send as param uintArray ?
Comment: @brabertaser1992 sorry I'm not sure. But you can refer to this [library](https://github.com/danialfarid/angular-file-upload/blob/master/dist/angular-file-upload.js) (line 30 - 70) about how to send file binary to backend server. I'll update my answer if I have any idea.
Comment: Failed to execute 'atob' on 'Window': The string to be decoded is not correctly encoded.
Comment: i do it so `var binaryImg = atob('/9j/4QNaRXhpZ***OkpaanqKwDAQACEQMRAD8Ar/7G7/n/ANv7/jvf5/U8Vf/Z');
      var length = binaryImg.length;
      var arrayBuffer = new ArrayBuffer(length);
      var uintArray = new Uint8Array(arrayBuffer);
      for (var i = 0; i < length; i++) {
          uintArray[i] = binaryImg.charCodeAt(i);
      }
      //var base64Image = files.split("data:image/jpeg;base64,")[1];

      fd.append("file", blob);
      //var blob = new Blob(['data:image/jpeg;base64'], {type: 'image/jpeg'});
      console.log(blob);
      $http.post(settings.apiBaseUri + "/files", fd, {`


Fourier Transform, Schwarz Space Problem

Question: Suppose $f \in L^2(0,2\pi)$ is such that there exists $v \in L^2(\mathbb{R})$ with
$$\int_{\mathbb{R}} f \phi' = \int_{\mathbb{R}} v \hat{\phi}$$ for all $\phi$ in Scwharz space on $\mathbb{R}$, and $\hat{\phi}$ is the fourier transform. Using the density of Schwarz in $L^2(\mathbb{R})$, show that there exists a bounded continuous function $g$ with $f=g$ a.e.
I'm a bit unclear as to how to begin this problem. The condition seems odd to me, I've been trying to somehow use an integration by parts, or the fact that this is a statements about inner products on $L^2$, but i'm unsure how to begin.
Comment: Is $f$ in $L^{2} (0,2\pi)$ or in $L^{2} (\mathbb R)$?
Answer: I have not worked out the entire problem, but I believe it should go something like this. You should use the fact that $$\int_{\mathbb{R}}v\hat{\phi}=\int_{\mathbb{R}}\hat{v}\phi.$$
This can be proven by writing the Fourier transform out as $\hat{v}(x)=\int_{\mathbb{R}}e^{-iyx}v(y)dy$ and switching the integrals (use Fubini). Next, perform an integration by parts. You will also need the fact that antiderivatives are continuous. Doing this should get you a continuous function $g$ such that $\langle f,\phi'\rangle=\langle g,\phi'\rangle$ for each $\phi$ in the Schwarz space. Using density then should get you that $f=g$ a.e.
If you would like further clarification just ask.


Choosing a regression model

Question: How can one objectively (read "algorithmically") select an appropriate model for doing a simple linear least-squares regression with two variables?
For example, say the data seem to show a quadratic trend, and a parabola is generated which fits the data quite well. How do we justify making this the regression? Or how do we eliminate the possibility of there existing a better model?
What I'm really worried about is this: we could just keep adding polynomial terms until we had a perfect fit for the data set (an interpolation of the points), with no error whatsoever. But this would be useless as far as predicting or extrapolating, because there would be no reason to think that the "model" was actually appropriate. So how does one balance the needs of accuracy and intuitive appeal?
(Also, please alert me if this has been asked before, I assumed it would have been but did not find anything.)
Comment: The question itself really begs the question that you need to ask yourself which is why am I building this regression model? What will you use the model for? What do you hope to learn from the model? These are huge questions that will surely guide the steps you take in building the model.
Comment: I think the figure in the [Wikipedia page for overfitting](http://en.wikipedia.org/wiki/Overfitting) speaks by itself.
Answer: It's likely to be the case that you are not going to be able to find a polynomial that constitutes a correct description of the relationship no matter how much data you have.
This problem may extend to almost any class of models.
However, usually we're interested in getting a good description that suffices for some purpose (a model), rather than discovering the (possibly over-complicated) actual process driving the observations.
Indeed, even where the true process is from some hypothesized class of potential models, it may be counterproductive to discover the true model (which might be of high order, for example, but the high order terms might be very very small). It may be that a simpler (i.e. wrong) model is much better for our purposes.
For example, imagine we were trying to predict the next few values in a somewhat noisy series. Any model we fit has some error in the parameter estimates, and that error will be magnified by the forecasting. It doesn't take much to have a low-order model (which is necessarily biased) with much better mean square prediction error (say) performance than the 'true' model order.
One common tool for evaluating model performance is at out-of-sample prediction (not necessarily over time). Cross-validation is one common way to choose models or compare model performance.
Rob Hyndman wrote a nice little introduction here.
Answer: You can look at AIC, BIC or any of the other similar measures.
You could use your eyes and sense of the field.
Or you could avoid some of the problem by using splines.
Comment: Thanks. I'm just a beginner in statistics, may I ask if the AIC,BIC measures are "objective" (as in derived from something), or are they basically "made-up" from statistical experience?
Comment: AIC, BIC, etc are derived from two important qualities of the model under consideration: the number of parameters (k) and the likelihood of the model (L). Consider the $AIC \equiv 2k-2\ln(L)$ - here we see a tradeoff between the number of parameters and the likelihood.
Answer: I would say very often people align themselves with one of three different approaches:

frequentists, which make use of tests such as the F-test
bayesians, which make use of bayesian inference
information theory guys, which use the BIC and AIC, just like other examples cited above.

Frequentist analysis is probably both the most straightforward and the most criticized for its shortcomings. Information theory on the other hand, underwent a boom recently, drawing the attention of more and more people as time goes by.
I think you should try to understand a bit and draw some ideas from each of the three approaches.
If you have no idea about what the data should contain, then the frequentist approach is a good way to start; on the other hand If you have some information on the underlying model, take a look at bayesian inference. And I would always keep the number of free parameters low, and that's what AIC and BIC try to balancing information with parameters.
Answer: I would use restricted cubic splines which allow you to better approximate the curve. As an added refinement, may use AICc (or BIC) to chose the number of knots.
Comment: It really depends on the specific problem and data set, sometimes a straight line is the best difficult to say anything without seeing the data.

Choosing a regression model


Sample weighting vs. (e.g. one-hot encoded) categories

Question: I have seen recommendations to use sample weighting when the training dataset is not evenly balanced over known categories, so that an imbalance in the number of elements in each category does not skew the training towards the most frequently represented types (i.e. the #samples x sample weights is made the same for all categories)
If, however, training specifically includes the categories (and within which there is either no further skewing effect or it is immaterial) is it necessary or desirable to include weighting as well?
Naively, one might assume that one-hot encoded categories effectively switch between different parameter sets.
However, the extent to which the input 1's and 0's are then reflected in internal 1, 0 weights is surely a function of the degree of training.
In other words, is it true that, whilst in theory sample weighting over categories is not required as #epochs -> infinity, in practice, sample weighting ensures that the model is effectively as good as it can be for #epochs < infinity, even when the weighting is over categories explicitly included as input?
Update
Thanks to @Tim's linked paper, in section 1.1 we find:

Despite the popularity of importance sampling in combination with deep
neural networks, how and when it works remain open questions. Unlike
linear models, deep neural networks are generally over-parameterized,
capable of fitting training datasets to perfect accuracy (Zhang et
al., 2017). Moreover, it is now recognized that for many tasks deep
neural networks continue to improve generalization error past the
point of achieving zero training error (Soudry et al., 2017). Thus
they are not only capable of separating the training set (given enough
epochs) but actually are trained to do so in common practice. Since
neural networks are capable of shattering the training set (and often
do), it is not clear that any trade-offs must be made among
classifying each of the training points. Thus any effects of
importance weighting depend crucially on how they impact the dynamics
of optimization, an actively-studied but still poorly-understood
topic.

(emphasis added)
It seems that the authors are of similar opinion to me; I suspect that because nn's can shatter* the dataset, whether or not they do so perfectly at any time depends on lots of things, but in general the shattering can be perfected given sufficient training (provided in practice one doesn't run out of precision)

Definition: A set S of examples is shattered by a set of functions H
if for every partition of the examples in S into positive and negative
examples there is a function in H that gives exactly these labels to
the examples Intuition: A rich set of functions shatters large sets of
points (from ...Shattering and VC Dimensions)

But - is there anything else to add to the idea?
It seems that the impact of importance weighting diminishes, but if one has limited time one would get better results by including it.
Answer: There was an interesting paper What is the Effect of Importance Weighting in Deep Learning? by Byrd and Lipton, where they've shown the following:

We demonstrate the surprising finding that for regularized neural networks optimized by stochastic gradient descent (SGD), the impact
of importance weighting diminishes over epochs of training.
We show that L2 regularization and batch normalization, (but not dropout)  interact with importance weights, restoring (some) impact on
learned models.
We replicate our results across a variety of networks, tasks, and datasets.
Our results call into question the standard application of importance weighting when applied to deep networks, a finding with
practical consequences on the fields of causal inference, domain
adaptation, and off-policy reinforcement learning.

So it works for traditional machine learning models (say, logistic regression), there are mixed results when using it for neural networks. Notice that the results show something opposite to what you suggested. Moreover, this has nothing to do with one-hot-encoding.
Comment: Thanks. Off to read that paper now; given my line of reasoning I'm curious to see why they find it surprising. (I know one hot encoding per se is not strictly relevant, but I thought surfacing the ideas of 1s and 0s motivated the rationale better.)

Hmmm... deep learning case different? "while **the effect of importance weighting is well characterized for low-capacity misspecified models**, little is known about how it impacts overparameterized, deep neural networks"


Matrices times Kronecker Delta

Question: What would
$\Lambda^{\alpha}_{\lambda '}\Lambda^{\gamma '}_{\beta}\delta^{\lambda '}_{\gamma '}$ be equal to (where $\delta$ is the Kronecker Delta)? Is it $\Lambda^{\alpha}_{\beta}$? I don't quite get it, maybe someone can help me.
Answer: The summation convention is commutative and associative (since multiplication of scalars is, and you can interchange the order of summation); since the first factor doesn't involve $\gamma'$, you can compute this as
$$ \Lambda^{\alpha}_{\lambda '}\Lambda^{\gamma '}_{\beta}\delta^{\lambda '}_{\gamma '} = \Lambda^{\alpha}_{\lambda '} \left( \Lambda^{\gamma '}_{\beta}\delta^{\lambda '}_{\gamma '} \right)
= \Lambda^{\alpha}_{\lambda '}\Lambda^{\lambda '}_{\beta}$$
Alternatively, the middle factor doesn't involve $\lambda'$ so
$$ \Lambda^{\alpha}_{\lambda '}\Lambda^{\gamma '}_{\beta}\delta^{\lambda '}_{\gamma '}
= \Lambda^{\gamma '}_{\beta} \left( \Lambda^{\alpha}_{\lambda '}\delta^{\lambda '}_{\gamma '} \right)
= \Lambda^{\gamma '}_{\beta} \Lambda^{\alpha}_{\gamma '} $$
(note that both results are the same; notationally they only differ in the order the factors are written and the choice of dummy variable)

Matrices times Kronecker Delta


Moving a CakePHP 1.3.2 Application to a New Server

Question: I have a very old CakePHP 1.3.2 application that used to work on a server under a given domain. I have the original files and now I'm trying to set it up and get it running on a new server under a different domain.
At first I had to fix the following line in the webroot/index.php:
define('CAKE_CORE_INCLUDE_PATH', '/home/SOME_NAME/cakephp-1.3.2');

as the application could not even find the framework. It does find it now but I'm getting a 404 Page Not Found error. I'm completely clueless what is causing it and I even can't determine if the database connection is successful using the new user/pass/host parameters. The only indication of something going on is the following log file that I found on the server and it gets populated with the following lines every time I try to access the application through the browser:
2015-06-09 10:00:24 Warning: Warning (512): /home/SOME_NAME/public_html/OLD_DOMAIN_NAME/tmp/cache/ is not writable in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/cache/file.php, line 267]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): array_keys() expects parameter 1 to be array, boolean given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1936]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): array_keys() expects parameter 1 to be array, boolean given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1949]
2015-06-09 10:00:24 Warning: Warning (2): array_filter() expects parameter 1 to be array, null given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1953]
2015-06-09 10:00:24 Warning: Warning (2): array_values() expects parameter 1 to be array, null given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1953]
2015-06-09 10:00:24 Warning: Warning (2): array_unique() expects parameter 1 to be array, null given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 2045]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): array_keys() expects parameter 1 to be array, boolean given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1936]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): array_keys() expects parameter 1 to be array, boolean given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1949]
2015-06-09 10:00:24 Warning: Warning (2): array_filter() expects parameter 1 to be array, null given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1953]
2015-06-09 10:00:24 Warning: Warning (2): array_values() expects parameter 1 to be array, null given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1953]
2015-06-09 10:00:24 Warning: Warning (2): array_unique() expects parameter 1 to be array, null given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 2045]
2015-06-09 10:00:24 Warning: Warning (2): array_unique() expects parameter 1 to be array, null given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1229]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): implode() [function.implode]: Invalid arguments passed in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1448]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): array_keys() expects parameter 1 to be array, boolean given in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo_source.php, line 1936]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]
2015-06-09 10:00:24 Warning: Warning (2): Invalid argument supplied for foreach() in [/home/SOME_NAME/cakephp-1.3.2/cake/libs/model/datasources/dbo/dbo_mysql.php, line 127]

Any suggestions on how to proceed from here to narrow down the error to something more specific?
Comment: Have you tried searching the app's files for `OLD_DOMAIN_NAME`? It would appear there is a reference to it somewhere in your code and you need to change it.
Comment: The files on the new server are still in a folder OLD_DOMAIN_NAME so the path exists and the files are still there. I'll make a search though to see what else will pop up.
Answer: Your first error is because /home/SOME_NAME/public_html/OLD_DOMAIN_NAME/tmp/cache/ is not writeable. Make sure that tmp has the correct file permissions; it needs to be writeable by the web server user.
Otherwise your issue appears to be that the database config is wrong so the app cannot connect to the database. Check app/config/database.php and make sure the database is setup correctly and make sure that bootstrap.php isn't altering the connection (perhaps dependent on the server the app is installed).
You can check which database settings are being used in your code using:-
Configure::read('Session.database');

Test this near one of your errors.
If you can clear up those two issues you should be good.
Comment: The first one is cleared now, thank you. As for the database issue, it seems that it is setup correctly and I don't see anything in bootstrap.php that would alter the connection. Any ideas how to verify that it is working as intended?
Comment: Hm, might be irrelevant but I have the following two lines in the webroot/index.php:
$Dispatcher = new Dispatcher(); $Dispatcher->dispatch();
The error that I'm getting is generated after the call to dispatch().
Comment: The `Dispatcher` code is ok as that comes from Cake itself (https://github.com/cakephp/cakephp/blob/1.3/app/webroot/index.php). I've updated my answer with a suggestion about checking which database settings are in use. Still suspect this is the cause of your issue.
Comment: It does make sense but what I got from Configure::read(...); per your suggestion was just 'default'. I guess that knocks out bootstrap out of the picture at least. I've read somewhere that this might be caused by missing model so is there a way to check what exactly is causing the issue in more detailed error message? I've check the routes and the controller/model/view used there seem to be ok...
Comment: Cake's error logs should really be telling you if models are missing (`tmp/logs/error.log`). Your issue appears to arise from this line of code `$cols = $this->query('SHOW FULL COLUMNS FROM ' . $this->fullTableName($model));` (https://github.com/cakephp/cakephp/blob/1.3/cake/libs/model/datasources/dbo/dbo_mysql.php#L125) it might be worth checking what the query being generated is and whether it runs on your database (wouldn't normally recommend hacking core code, but may be the quickest way of reaching an answer).
Comment: So at the end it turns out that there were several tables missing from the database and it seems to be working now when they were added. What annoys me beyond boiling point though is that there was no human readable easy to understand message indicating that. I mean, come on, that should be one of the first errors that one could think of and it's not covered by the framework? Anyway, thank you very much for your time and dedication, marking this as correct as it was the database indeed and upvoting. Thanks!
Comment: Yes, it is an old version indeed but unfortunately I had no choice when a client decided to dig up a legacy project to run it again. Thanks again and good luck!
Answer: You have your work cut out for you. For starters, I would create a phpinfo to see the configuration differences between the old and new servers.
<?php

// Show all information, defaults to INFO_ALL
phpinfo();

// Show just the module information.
// phpinfo(8) yields identical results.
phpinfo(INFO_MODULES);

?>

phpinfo
Comment: I have no access to the old server. The site was live on it more than an year ago and now it's long gone.
Comment: I strongly suggest that you develop site locally or in a non-production environment. I would still run the phpinfo on the current server. I would also change/add the following settings in your php.ini file:   ini_set('display_errors',1);
ini_set('display_startup_errors',1);
error_reporting(-1);
Comment: Also, do you have cakewalk configured on your current host? Does your current host support the use of cakewalk?


Error Start Apache "php value"

Question: After updating my vps, I get the follow error:

Failed to start apache :
  Starting httpd: Syntax error on line 31 of /etc/httpd/conf.d/php.conf:
  Invalid command 'php_value', perhaps misspelled or defined by a module not included in the server configuration
  [FAILED]

Enable modules on PHP:

[PHP Modules]
  bz2
  calendar
  Core
  ctype
  curl
  date
  dom
  ereg
  exif
  fileinfo
  filter
  ftp
  gd
  gettext
  gmp
  hash
  iconv
  imap
  intl
  ionCube Loader
  json
  libxml
  mbstring
  mcrypt
  mhash
  mysql
  mysqli
  openssl
  pcntl
  pcre
  PDO
  pdo_mysql
  pdo_sqlite
  Phar
  readline
  Reflection
  session
  shmop
  SimpleXML
  sockets
  SPL
  sqlite3
  standard
  suhosin
  tokenizer
  wddx
  xml
  xmlreader
  xmlrpc
  xmlwriter
  xsl
  zip
  zlib
  [Zend Modules]
  the ionCube PHP Loader

Line relative to error:

php_value session.save_handler "files"

What does the error mean and how can I fix it?
Answer: I recently ran into this exact problem using Plesk 9.5 on CentOS.
I cannot say for sure whether it was caused by an update to Plesk, or not. The customer doesn't think any changes were made recently, but Apache failed to start with this error.
After an assesment of the system to ensure it wasn't due to a breach, I did some troubleshooting and determined that mod_php had been removed from the Apache config. After checking Plesk settings, every vhost on the box was using FastCGI and SuExec.
When using FastCGI and SuExec, you cannot change PHP directives in php.conf (FastCGI) and .htaccess (SuExec).
The customer had originally commented out the offending lines, but this broke session support for everything. The only way I was able to resolve it was to manually add mod_php back to httpd.conf.
Add the following line to the section with the other LoadModule's. Make sure the path (../modules/) matches the rest of the modules in there. Chances are good that it already exists on your system and was simply removed from the config during the update.
bash# vi /etc/httpd/conf/httpd.conf
LoadModule php5_module ../modules/libphp5.so

bash# apachectl restart

This caught me off guard, and I cannot say for sure it is the upgrade that caused the issue or whether this is the best fix. I am open for comment, but highly advise against commenting out the php directives in /etc/httpd/conf.d/php.conf as it will break stuff.
Comment: thanks David, saved the day.
My yum log shows an auto update to php 5.3.26 but don't really understand why httpd.conf was over written. Regards
Comment: Thank you! I went to httpd.conf and I saw that the line LoadModule php5_module pathfilename had been commented out,I just removed the # and everything is back to normal.
Comment: I obtained this error after an upgrade from Ubuntu 16.04 to 18.04.1. I don't know the cause and there is no modules directory, instead a mods-available and mods-enabled directory. I added symlinks for php7.2.conf and php7.2.load to mods-enabled pointing to mods-available and then apache2 was able to start.
Comment: Hi David, it resolved the issue and introduced a new one. "Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP" How can I resolve this?
Answer:

Invalid command 'php_value', perhaps misspelled or defined by a module not included in the server configuration

The Apache httpd-2.4 can failed with above error if you by mistake configured it to use other mpm module than prefork (as only prefork mpm works fine with php on Linux (CentOs7/RHEL7). Correcting Apache httpd configuration to use prefork mpm will solve the issue.

/etc/httpd/conf.modules.d/10-php.conf

  LoadModule php5_module modules/libphp5.so

/etc/httpd/conf.modules.d/00-mpm.conf
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
Answer: As commented by David previously, this is likely because the PHP mod has been disabled from Apache2.
To me, it appeared as an error when (re)starting Apache:

Job for apache2.service failed because the control process exited with error code.
  See "systemctl status apache2.service" and "journalctl -xe" for details.

Then, inside journalctl -xe:

AH00526: Syntax error on line 31 of /etc/apache2/sites-enabled/host.conf
  Invalid command 'php_value', perhaps misspelled or defined by a module not included in the server configuration

This happened when upgrading from Ubuntu 17.10 to 18.04 (and in consequence from PHP7.1 to 7.2), which apparently disabled libapache2-mod-php* completely.
On Ubuntu, the following should fix it:
sudo a2enmod php7.2
sudo systemctl restart apache2

Answer: This could happen if you configured Apache to use other MPM than mod_prefork.
Check what module is in use by httpd -V command. It should show you something like:
...
Server MPM: prefork
...

If it's not, check
/etc/httpd/conf.modules.d/00-mpm.conf

like described here and make sure the next line is present in it and uncommented:
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

In case you need to use other module than mod_prefork you will have to comment or delete these lines.
Answer: Just comment those lines out.
I had
#php_value session.save_handler "files"
#php_value session.save_path    "/var/lib/php/session"

Then run the "reconfigure if your in Plesk, then restart apache" seemed to work fine. Had this happen after plesk update #53 10mins ago.
Comment: Just had this happen. Commenting out the offending lines will allow Apache to start, but removes session support among other things. Chances are mod_php was disabled. I am using Plesk also.
Comment: Hi,
I'am not using Plesk, using Zpanel. What I need reconfigure?

Thanks
Comment: I basically removed those lines and then an apache restart may fix it. But i had to re-configure the vhost in plesk. It is just a button, not sure the equivalent in cpanel.
Comment: This is a bit disturbing. This is part of the core php - you need to jump through hoops at compile time to disable session support. And according to the manual, they are both changeable in PHP_INI_ALL (but it might be worth trying php_admin_value). Does your session handler work? Are you using mod_php?
Answer: Before running the upgrade from Atomic make a backup of file php.conf. Then after the upgrade completes overwrite the new file with the old one.
Answer: I had the similar problem, realized that php5 module was commented,
which means could not load and in phpmyadmin conf it was giving me errors because of having   php_admin_value which probably was working with php5_module and it was not in place.
I went to httpd.conf and uncommented:
LoadModule php5_module "c:/wamp/bin/php/php5.5.12/php5apache2_4.dll"

Answer: Edit: After implementing my comments below, I found that there was another reason I was running into this issue. The server has PHP-FPM implemented. Part of the implementation of PHP-FPM meant that /etc/httpd/conf.d/php.conf was disabled (i.e. renamed to php.conf.bak). The recent update brought in a new php.conf. As a result the PHP-FPM configuration was conflicting and causing the errors mentioned. The fix was to simply rename php.conf to something like php.conf.bak.feb2019. As a result, php.conf does not get loaded and therefore there is no conflict with php-fpm.
/// Before edit: /////
The solution for me was a combination of the answers provided here.
I noticed this error after updating an Oracle Linux 7 server that hand't been updated in a long
As suggested by @dasharathmasirkar, @davidhoude @alexlanger, the problem stems from mpm prefork config being overridden after the update.
To address this:
check /etc/httpd/conf.modules.d/00-mpm.conf and ensure that the following line is uncommented:
# prefork MPM: Implements a non-threaded, pre-forking web server
# See: http://httpd.apache.org/docs/2.4/mod/prefork.html
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

Also, make sure that the other worker mpm is commented-out so it does not conflict:
# worker MPM: Multi-Processing Module implementing a hybrid
# multi-threaded multi-process web server
# See: http://httpd.apache.org/docs/2.4/mod/worker.html
# LoadModule mpm_worker_module modules/mod_mpm_worker.so

Then restart apache: apachectl restart

Error Start Apache "php value"


Angular with wordpress jwt auth

Question: I'm trying to make wordpress as a backend to my angularjs app, so I'm using the plugin rest-api with the jwt-auth
so when trying to login I get the following error
XMLHttpRequest cannot load http://localhost/back/wp-json/jwt-auth/v1/token. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://imider.ma' is therefore not allowed access. The response had HTTP status code 500.

I know i have to add CROS access, but I'm not familiar with wordpress, so any help?
Answer: https://docs.google.com/document/d/17zgUHZrvL5KVG2yKQE8NWgxNZn6V08xr65DBox5WVZ0/edit?usp=sharing
here is my tutorial of how to access the api
then you can use this to get the token
$http({
                method:'post',
                url:'',
                data: {
                    username: '',
                    password: ''
                }
            }).then(function(results){
                console.log(results);
            })

then you can use this
$http({
                    method:'get',
                    url:'',
                    headers: { 'Authorization': 'Bearer <myTokenId>' }
                }).then(function(results){
                    console.log(results);
                })

Answer: I created a video detailing the process of installing and setting up the plugin. If you follow the steps I outlined there you should be good.
https://youtu.be/Mp7T7x1oxDk
The idea is that you also need to modify .htaccess and wp-config.php to make the plugin work with the existing API endpoints as well.
Just by installing the plugin and adding a SECRET_KEY, used to sign the token will make the JWT setup work but it will not allow you to use the tokens generated through that API with the existing REST API endpoints.

Angular with wordpress jwt auth


How Block size varies from Cluster1 to Cluster2, if we use DistCp command?

Question: I am processing "DistCp" command to move few critical files form My Cluster1 to Cluster2. These critical files were residing with Blocksize 64MB, before. And now moved to Cluster2 [it got 128MB blocksize).
After the DistCp move, how does the does the critical files performance will increase with new blocksize in Cluster2..performance increase or decreases..???
Answer: It depends on your files. Hadoop files are supposed to be read sequentially and if your files are big(let's say Gbs or Tbs) it would increase performance if you increase your blocksize, because it would decrease the number of tasks to be performed. Copying with Distcp will not maintain the block properties of your file since block configurations varies from cluster to cluster.


How to make a div accessable only for authenticated users

Question: I have a  inside page.aspx, and I have created a role called Premium. I want all users to be able to see page.aspx, but only Premium users can see the div.
My web.condig
<location path="?">
<system.web>
    <authorization>
      <allow roles="Premium" />
      <deny users="*" />
    </authorization>
</system.web>

I don't now how to write it so only users in the role Premium can see the
Comment: When you say div, you mean a part of your page warped with `div` - correct ?
Comment: You should read some basics about authentication, authorization, validation (in WebServer/Controllers/Code-Behind) and security filtering(in aspx-files etc.), and why you should use all at once.
Answer: The simplest way is to use an if statement to control output of that part of the page:
<% if(User.IsInRole("Premium")) { %>

 blah

<% } %>

You could, of course, make a UserControl or similar that contains this logic instead, thus pushing the conditional expression out of the host page and having the page simply render the control always.
Answer: You could make the premium section a partial view which has it's own method in the controller,  for which you could require the role Premium.
Comment: The term 'partial view' is an MVC thing, really - and the question appears (based on what little information is available!) to be a webforms question.
Answer: in your page.aspx, do the following:
<%if (Request.IsAuthenticated){ %>
    <%if HttpContext.Current.User.IsInRole("premium"){ %>

premium content

    <%} %>
<%}%>


curl from localhost gets 404

Question: We have a cPanel host running on centos that has got me stumped.  I am trying to use curl from the command line to trigger a specific web page.  However I keep getting 404 errors for the specific page whereas the main page works fine.
curl http://example.com

This works fine.
curl http://example.com/dev/buil?flush=all

This returns a 404 every time.  I can access this exact page from a web browser and indeed running this command from another remote Linux box works fine.  I have run a packet capture on the firewall that it sits behind and can find no blocked packets, only DNS requests for example.com.
I am really stumped as to where to look next as I was certain I was going to find some kind of rule conflict on the firewall blocking a port that curl needed but alas I have found nothing.
Comment: What does curl -v ... say ?
Comment: When you visit http://example.com/dev/buil?flush=all in your browser, is your browser conveniently redirecting you to http://WWW.example.com/dev/buil?flush=all ? Perhaps it is, and your Apache virtual host is not configured with an example.com ServerAlias (only www.example.com).
Answer: A 404 error has nothing to do with a firewall.  404 represents "Page not found" and is generated by the web-server itself.  It is not uncommon for web-servers to be configured to support "virtual-hosting" in which, different websites are offered on different network interfaces.
In this case, this could mean that you have a different virtual host on 127.0.0.1 (the loopback interface) than the wan interface.  You may need to adjust your hosts file to point that "hostname" to the wan interface, instead of the "localhost" interface.
Comment: He browsed to the same host, so virtualhosts don't matter.
Comment: Same host does not mean same interface.  It is very likely that he is getting the default site on 127.0.0.1 instead of the host on the expected interface.
Comment: harsh down vote.  Especially as you are bang on.  Added the WAN IP and name to the hosts file and now it starts resolving and processing correctly.  Thank you very much for the suggestion.

curl from localhost gets 404


Annuities, lottery and interest rate

Question: I am having trouble doing this question.

Mary Jones won $\$4,000,000$ in a state lottery.
She will receive a cheque for $\$200,000$ now and a similar one for each year for 19 years. To provide these 20 payments, the State Lottery Commission purchased an annuity due at the interest rate of $10\%$ compounded annually. How much did the annuity cost the Commission?

The answer key is utilizing this formula:
$$A =R\cdot (1+r)\cdot \frac{1 – (1+r)^{-n} }{ r}$$
However, I am wondering, where did $R\cdot (1+r)$ come from?
Comment: The factor $(1+r)$ comes from the fact that Mary receives her first cheque now and not at the end of the period.
Comment: Please give a reply if you have any further questions or not.
Comment: The $R$ is the amount of each payment.  The cost of the annuity has to scale with that.
Comment: You are having trouble doing this question? What do you mean? Are you trying to find out where did R x (1 + r) come from?
Answer: Correct me if Im wrong. R=Rent, r=number of payments of R in a year, n=total time in years.
Then the resolution might be A =200,000\cdot (1+1)\cdot \frac{1 – (1+1)^{-20} }{ 1}
A = 399,999.62

Annuities, lottery and interest rate


getting all notifications

Question: I can easily get the unread notifications with this query:
https://graph.facebook.com/USER_ID/notifications?access_token=ACCESS_TOKEN
Is it possible to all the notifications (marked as read) too?
Answer: Yes, you can specify the parameter include_read as 1 to indicate you wish to receive read notifications.
https://graph.facebook.com/USER_ID/notifications?include_read=1&access_token=ACCESS_TOKEN

Answer: this link may be help you
http://developers.facebook.com/docs/howtos/notifications-api/
Using The API
Applications can generate notifications by issuing a HTTP POST request to the /user_id/notifications Graph API, with a app access_token.
POST /{recipient_userid}/notifications?access_token= … &template= … &href= …
Comment: This has nothing to do with the user's question and merely references irrelevant documentation.  The user does not want to push notifications from games, he wants to pull in read notifications from his user...

getting all notifications


Acoustic guitar with humbucker pickup sounds like an electric guitar, not acoustic

Question: Last week I had played the  first time with a PA. My acoustic guitar Epiphone ej160e using mini acoustic humbucker sounds more like a clean electric guitar, not like an acoustic.
But why? Can someone can give me a hint which setup I should use ( additional Pickup, footswitch...)?
Comment: Please add some info about the type of pickup the guitar has, the complete signal chain, etc..
Answer: So this guitar is quite a unique model, the John Lennon signature. Its pickup is not an acoustic-guitar pickup at all but, well, basically a standard high-impedance humbucker in small format.
It's no big surprise then that it sounds more like an electric guitar: such a pickup, together with the cable capacitance, forms a 2nd order lowpass filter, and that gives the characteristic electric-guitar sound. (Why did you buy such a guitar if you don't want its characteristic sound??)
To get a more acoustic sound, the best option is of course to use a microphone. Unfortunately that brings lots of problems in a live setting. Magnetic pickups actually remain the best in terms of feedback robustness etc.. They can also give a decent acoustic sound, but it's crucial that you avoid the resonant lowpass-filter effect. Properly designed modern acoustic PUs achieve this through a low-impedance spec, i.e. the coil uses thicker wire  with fewer windings, which leads too a much smaller inductance. Unfortunately it also makes the signal voltage much weaker, but that can actually be compensated easily with a preamp.
If you want to keep the PU as it is in the guitar, there's another parameter you can change to keep the filter in check: you can lower the capacitance. It's mostly the cable that's responsible for this, so you want a decoupling circuit right after the pickup. This can be a very simple buffer:

(created with CircuitLab)
After that, the frequency response should be much clearer.
Of course make sure you don't use an electric-guitar amp but an acoustic amp or go straight into the PA via a DI, best an acoustic-optimised one like a Radial Tonebone or LRBaggs ParaDI. But with the active circuit in your guitar, a cheap passive DI like the Palmer PAN01 should also be fine.
Comment: Wow, thank you for your very technical and detailed reply. I'm impressed! But back to your answer.  That mean I have to solder the circuite  based on your draft. Plug my guitar to the filter,  the output to the acoustic-optimiser, those output to the passiv DI box and this output into the PA.  You mark the Output in your draft, means that, ouput --> + and ground keeps ground at the conection wire. Is this correct?
Comment: Yes, ground needs to be connected to everything that's currently ground. — I didn't include the volume / tone pots in the circuit; I suppose you want to use these too. You can re-wire the tone pot to a simple 1st order lowpass, and put the volume pot after it. Also keep the usual stuff in mind: a ≈10μF electrolytic capacitor parallel with the battery for smoothing, some way of disconnecting the battery (best with a stereo jack, so disconnecting the cable automatically disconnects the battery), and some protective resistor before the output.
Comment: Well, so much great information. It's great!  Does you have a book recommendation about this topic. It is easier to me to read a circuite than a technical explaination. But I try to see what your meaning :)
Comment: https://www.amazon.de/Elektrogitarren-Technik-Sound-Helmuth-Lemme/dp/3895761117. (He actually has basically the same circuit on page 203.)
Answer: If you have plugged straight into the PA and not via a DI box  that could account for your different sound. A cheap passive DI (and an XLR-XLR cable) is a good thing to carry around with your guitar gear.
Comment: The output from the pickup on your electric acoustic guitar is not optimised for the inputs on a PA. The DI unit matches  up the impedances. https://en.wikipedia.org/wiki/DI_unit . just buy the cheapest passive DI unit you can find. They don't need batteries or power, but you will need an XLR (microphone) lead to connect to the PA input as well as a guitar lead to connect your guitar to the DI.
Answer: The amplified sound comes from the humbucker pickup positioned between the sound hole and the end of the fretboard. Your guitar has steel strings and the pickup behaves exactly the same as the pickup in a solid-body guitar - which explains why the amplified sound is that of a solid body electric.
To get the acoustic sound you'll need to experiment with an acoustic microphone pointing at the sound hole on your guitar.
Comment: Thank you for your repyl, some people means, that there will be some problems with feedback, but I also strongly belive that this should be a good technique. Some means also to put a aditional piezo Pickup. So you have experiance about it?
Comment: Frequently a pickup like a piezo will be used along with a microphone so the two signals can be filtered and mixed together for the "best of both worlds" - the electric pickup reproduces clean attacks and the microphone captures the characteristic resonant sound of an acoustic guitar.
Answer: I have never found a true acoustic sound with humbuckers. Single coil pickups like Fender is the only way I have gotten a true acoustic sound.
Comment: No magnetic pickups give a true acoustic sound. A microphone is really your only option. A piezoelectric pickup is closest if you must use a pickup.


best practices for setting development environment

Question: I use Linux as primary OS. I need some suggestions regarding how should I set up my desktop and development. I do work on mostly .Net and Drupal, but some time on other lamp products and C/C++, Qt.  I'm also interested in mobile (android..) and embedded development.
Currently I install everything on my main OS, even I use it a little. I use VMs a little (for lamp server).
Should I use separate VM for each kind of development (like one for .Net/Mono, another C++, one for mobile and one for db only, one for xyz things etc)
Keep primary development environment on main os and move others in VM.

main os should be messed up
keep things easy to organize (must)
performance should be optimal (optimal settings for best performance of components)

I'm interested to know how others' are doing.
Comment: Seems like more of a StackOverflow question to me
Comment: this is better suited to our sister site, [Stack Overflow](http://stackoverflow.com/).  please don't crosspost; it will be migrated there shortly.
Answer:
Optimal performance != using VMs
if you need to use VMs anyway, then yes: it could be better to use a separate VM for each thing that need one, unless you need more than one at once
Comment: by optimal performance I mean optimal performance of individual component (VM), (i.e. with many packages installed server may slow down).
Answer: There are both pros and cons with VM's.
Pros:

portability: you can move image to
different server
easy backup (but lengthy)
replication (new member joins team)

Cons

performance
hardware requirements
size of backups (20-40 GB per VM ...)
management of backed up images (what is the difference is not obvious)
keeping all images up to date
(patching / Windows updates)

For your scenario, I would create base VM with core OS and shared components (Web server, database), replicated it and installed specific tools into separate VM. If you combine tools within VM, you may end up with same mess as in case of using base OS - the advantage is that it is much easier to get rid of it ;-)
Answer: Now that OCI containers are stable and well supported, using those through docker, podman or other similar tool is an increasingly popular option.
They are isolated, but under the same kernel, so:

they are almost as portable as virtual machines,
like virtual machines they can have their own virtual IP addresses, so they can run services not visible from the outside and without occupying port on the host, but
they don't reserve any extra space on disk or in memory like virtual machines and
they are not slowed by any virtualization layers and
mounting directories from the host is easy and does not require any special support.

The usual approach is to have the checkout in the developer's normal home directory and mount it into containers for building, testing and running.
Also building in containers is now supported by Remote Development extension for Visual Studio Code


Javascript breaking text across multiple lines while maintaining indentation

Question: I have the following function
const taskErrorMessage = (task, reason) => {
    const account = task.account;
    return {
        message: `ERROR. ACTION: ${task.action}. ACCOUNT: ${account._id}, \
${account.name}. REASON: ${reason}. DATE: \
${new Date()}`
    }
}

It doesn't fit on one line so I am breaking it up using \ however it seems that if I try to add indentation the final string will be incorrect with unwanted spaces. Is there a way to break a string into multiple lines whilst maintaining indentation?
Comment: Does this have to be solved in javascript? Or can we use HTML and CSS?
Answer: For bare template literals, spaces at the start of lines are always interpreted literally, so they can't be used for code indentation.
If you want all whitespace collapsed, you could imitate this functionality with a tagged template literal:

const $_ = (strings, ...vals) => {
    let result = ''

    strings.forEach((str, idx) => {
        result += str
        if (typeof vals[idx] !== 'undefined')
            result += vals[idx]
    })

    return result.trim().replace(/\s+/g, ' ')
}

console.log(
    $_`hooray
          indents for    every
                               ${1}
    `
)

However, note that this will run at run-time, not compile-time. It's unlikely to be a performance bottleneck unless you're creating hundreds of thousands of strings this way, but worth considering all the same.
Comment: I tried literal line breaks and it's breaking the line which I don't want
Comment: I've edited my answer with a possible solution, if you don't mind the performance overhead.


how to show user name in menu

Question: I have a {{user_name}} variable in my app.component.ts and I want to use it for displaying the user name which I am getting from local storage after the user logs in.
But this loads only at app start and I want to reload it after the user logs in. When I close and reopen app after log in it works fine and user name displayed. But at first, it shows only "hi" and doesn't show the user name.
<ion-menu [content]="content" >
    <ion-header>
      <ion-toolbar>
        <ion-title>

            HI! {{user_name}}

</ion-title>
      </ion-toolbar>

    </ion-header>

    <ion-content>

      <ion-list >

        <button menuClose ion-item *ngFor="let p of pages" (click)="openPage(p)">
          {{p.title}}

        </button>
        <button  menuClose ion-item (click)="logout()" >
          Log Out
        </button>
      </ion-list>

    </ion-content>

  </ion-menu>

  <ion-nav [root]="rootPage" #content swipeBackEnabled="false"></ion-nav>

I know that i can publish an event at user loged in , but how to  reload the menu after receiving the event
Comment: Using of **Ionic Events** may help you. Refer to this link https://ionicframework.com/docs/v3/api/util/Events/
Comment: show the code for getting from localStorage
Comment: This is what state management of variables in Angular does, You need to use observable/Subscribe for this case, as angular is SPA
Answer: I had a pretty similar case, in my opinion events are the best way to go:
app.component.ts
   import { ..., Events } from 'ionic-angular'; //First import events

   constructor(
    ...
    public events: Events,
    ...
    ){
        .....
        events.subscribe('user:created', (user) => {

          this.user_name=user.name;

        });

     ...
    }

Now your able to change that value anywhere in the app using     this.events.publish('user:created', user);
For example:
login.ts
    import { Your other imports...,Events} from 'ionic-angular';

    constructor(
      ...
      public events: Events,
      ...
      ) {}

     // let's assume you did your login and have an user array which has user.name in it,
     // pass that data like this

     updateUserInfo(user){
      this.events.publish('user:created', user);
     }

It will work as expected. Hope it helps...
Comment: `this.events.publish` will reload the data on Ion menu if you do what I explained on my answer. If you create a stackblitz will be much easier to help.
Comment: yes i have published a event now , but how come i can reload the menu if i publish a event that out of my reach , please leave your valuable suggestion. @Whatatimetobealive
Comment: I m not really sure what do you mean by that? Are you curious about how events work?
Comment: i am sorry , but i know how events works , I just want to know that when a event took place and i have subscribed it. that at that time when that event took place i can catch it and able to show an alert. but instead of that alert only i want that  the menu should reload cause its not showing the users name.
Comment: thanks a lot You have provided your valuable help after a little addition into the logic . i did it with your help

how to show user name in menu