Using the Sites module with a multiligual site

Question: My multilingual site has three sets of (anonymous or logged-in) users: public, set1 and set2. They will all get the same site structure, but set1 or set2 will see slightly different content on some pages. Each group will be given a URL: http://example.com/mysite for public, and http://example.com/mysite/set1 or http://example.com/mysite/set2 for the others.
I can use the Views module with the Context module to help decide what content to show on the pages, but I need a way for the URL to retain, for example, "set1" in the URL path as they navigate around.
I tried Sites (which uses PURL), but it doesn't work with Internationalization: it prefixes the site name before the language, so when clicking the language selector back and forth, you can end up with page not found and a broken menu at http://example.com/mysite/set1/en/fr/fr/fr/en/en/en/en!
Is it possible to use Sites with i18n?
Or should I use another approach?
Answer: You cannot use the PURL "path" method with i18n (Internationalization) because it conflicts with i18n outbound url rewrites - specifically the detection and maintaining of region and language using the path.
I'm not sure if PURL domain and query string methods will still work.
I too am looking at this because I have a project where we've used PURL to maintain a brand identifier with the commerce module, but we want to also use i18n in future. I've used i18n and Domain Access modules before, both separately and together and they play well. Domain Access though is not what you want.
My suggestion would be to treat your "set1" and "set2" like regions. The i18n module lets you specify a region and language that is detected from the path. This is so you can have things like

Germany with German (DE-de)
Germany with French (DE-fr)
Germany with English (DE-en)
United States with Spanish (US-es)
United States with English (US-en)
etc.

So, in these examples we have a country-code and then a language. The i18n module allows you to detect first the country code as a region and the language code for language and then customise the settings based on both.
I would suggest you use this method to specify custom languages where you have a "set1" for region and a standard language code for language, defaulting to English where no language is defined (ie. LANGUAGE_NONE).
I'm going to give this a go myself for my project. I'll let you know how I go.


Probability, that when we send a $0$ down the network we will get back a $0$

Question: We can send a $0$ or a $1$ over a network of $1,2...$ nodes. Unfortunately on each node with probability $p$ the message is not made different, and with probability $1-p$ the message is XOR'ed. Find recurrence relation that determines with what probability if we send a $0$ over a network of $n$ nodes we will get $0$ after $n$'th node.
Base cases.
Obviously $$a_1=p$$ and $$a_2=p^2 + (1-p)^2$$ Because we can either succeed two times or fail two times and be ok. But what about the rest?
$$a_3=p^3+{{3}\choose{2}}(1-p)^2p$$ $$a_4=p^4+ (1-p)^4+{{4}\choose{2}}(1-p)^2p^2$$ And I fail in seeing a reccurent relation between the subsequent $a_n$'s.
Answer: I assume when you say XOR, it is XOR with 1.  Take the case that you send 0. And after node $n$, call the value $x_n$.  Then
$$\begin{align}
\text{Prob}(x_n=0) &= \text{Prob}(x_{n-1} = 0)*p + \text{Prob}(x_{n-1} = 1)*(1-p) \\
&= \text{Prob}(x_{n-1} = 0)*p + (1-\text{Prob}(x_{n-1} = 0))*(1-p) \\
&= (2p-1)\text{Prob}(x_{n-1} =0) + 1 - p
\end{align}
$$
You can figure out the rest.
Comment: Thanks, I totally forgot about the Law of Total Probability :)


Using MID and FIND to extract data from a String into an Excel Cell

Question: I have to Extract information from a json String for that I am trying to use Excel to extract the tag and the value of the tag within quotations into separate cells and eventually make a table, which will be easier for editing.
For Example: After I paste the data in an excel Cell:
"record_data":"[{\"fixed_leg\":[{\"calc_period_bus_center\":[\"Tgt\"],\"calc_period_bus_day_conv\":\"MOD \",\"calc_period\":\"12M\",\"calc_period\":\"8\",\"dcf\":\"30/360\",\"effective_date\":\"2021-04-08\",\"effective_date_ \":[\"Tgt\"],\"effective_date_bus_day_conv\":\"FOLLOWING\",\"final_principal_exchange\":\"FALSE\",\"fixed_rate\":\"0.0311\",\"initial_principal_exchange\":\"FALSE\",\"ntl_amount\":\"96\",\"ntl_ccy\":\"EUR\",\"pay_dates_relative_to\":\"CalculationPeriodEndDate\",\"pay_offset\":\"2D\",\"pay_offset_bus_center\":[\"Tgt\"],\"pay_offset_bus_day_conv\":\"MOD\",\"pay_period_bus_center\":[\"Tgt\"],\"pay_period_bus_day_conv\":\"MOD\",\"pay_period_freq\":\"12M\",\"payer_party\":\"US\",\"receiver_party\":\"GFIL\",\"resettable_ntl\":\"FALSE\",\"termination_date\":\"2026-04-08\",\"termination_date r\":[\"Tgt\"],\"termination_date_bus_day_conv\":\"MOD\"}],]"
}

I want to extract the Tag name and value to paste in other excel cells but due to limited knowledge about excel, I am struggling.
I have used
=MID(A2,FIND("calc_period_bus_center:",A2,1),A2)
but that does only return an error #VALUE
Comment: vba would be better see: https://stackoverflow.com/questions/6627652/parsing-json-in-excel-vba
Comment: Third argument for `MID` is the number of characters to return.
Comment: Have a look [here](https://stackoverflow.com/q/57761298/9758194). Looks like the same type of question to me :)
Answer: 2 points:  First, the third argument of MID is a number (how many character to return).  If you change this from A2 to LEN(A2), it will pull everything until the end of the string.
Secondly, you are searching for the string "calc_period_bus_center:".  However, this does not exist in the string you are searching:

"record_data":"[{\"fixed_leg\":[{\"calc_period_bus_center\":[\"Tgt\"],\"calc_period_bus_day_conv\":\"MOD \",\"calc_period\":\"12M\",\"calc_period\":\"8\",\"dcf\":\"30/360\",\"effective_date\":\"2021-04-08\",\"effective_date_ \":[\"Tgt\"],\"effective_date_bus_day_conv\":\"FOLLOWING\",\"final_principal_exchange\":\"FALSE\",\"fixed_rate\":\"0.0311\",\"initial_principal_exchange\":\"FALSE\",\"ntl_amount\":\"96\",\"ntl_ccy\":\"EUR\",\"pay_dates_relative_to\":\"CalculationPeriodEndDate\",\"pay_offset\":\"2D\",\"pay_offset_bus_center\":[\"Tgt\"],\"pay_offset_bus_day_conv\":\"MOD\",\"pay_period_bus_center\":[\"Tgt\"],\"pay_period_bus_day_conv\":\"MOD\",\"pay_period_freq\":\"12M\",\"payer_party\":\"US\",\"receiver_party\":\"GFIL\",\"resettable_ntl\":\"FALSE\",\"termination_date\":\"2026-04-08\",\"termination_date r\":[\"Tgt\"],\"termination_date_bus_day_conv\":\"MOD\"}],]"
  }

Remove the Colon (:):  =MID(A2,FIND("calc_period_bus_center",A2,1),Len(A2))  (Or add the \" before the colon instead)
Now, presuming you want the bit in square brackets after that, you can use a double-FIND to find the first appearance of "]" after "calc_period_bus_center", like so:
FIND("]", A2, FIND("calc_period_bus_center",A2,1))

You can then use this to trim down further with LEFT:
=MID(LEFT(A2, FIND("]", A2, FIND("calc_period_bus_center",A2,1))), FIND("calc_period_bus_center",A2,1), Len(A2))

This will give you "calc_period_bus_center\":[\"Tgt\"]", so SUBSTITUTE the first bit:
=SUBSTITUTE(MID(LEFT(A2, FIND("]", A2, FIND("calc_period_bus_center",A2,1))), FIND("calc_period_bus_center",A2,1), LEN(A2)),"calc_period_bus_center\"":","")

[\"Tgt\"]
Comment: Thank you it did work but I am not sure it was clear or i couldn't explain properly, i want to get both calc_period_bus_center and the value "Tgt" in separate cells for example:
CELL 1                                CELL 2
calc_period_bus_center          Tgt
calc_period_bus_day_conv   MOD
Comment: calc_period_bus_center Tgt
calc_period_bus_day_conv MOD
calc_period 12M
calc_period_roll 8
dcf 30/360
Comment: @xack That sounds like something that you should probably be trying to do with VBA instead...  (e.g. the `SPLIT` function can use `","` to break up rows, and `":"` to split up the columns)


How to send form data to web api via Angular

Question: I have a form in Angular 6 and I have a table which gets data from a web api. Is it possible to send that form data to that web api? I have a mock web api url which is a get request. My code so far is:
Html form:
    <form class="form-horizontal" [formGroup]="serviceForm" (ngSubmit)="onSubmit()">

      On behalf of this user: *

      <select formControlName="u_caller_id" class="form-select" required>
      <option *ngFor="let user of users" [ngValue]="user">
      {{ user.id }}
      </option>
      </select>

This field is required

      Your location: *

      <select formControlName="u_destination_country" class="form-select">
      <option *ngFor="let state of countries" [ngValue]="state">
      {{ state.name }}
      </option>
      </select>

This field is required

      Phone number: *

      <input class="form-input" type="text" id="u_requester_phone_number" placeholder="Enter phone number" type="number"
         formControlName="u_requester_phone_number" required>

This field is required

      Device/Asset: *

      <select formControlName="u_serial_number" class="form-select" required>
      <option *ngFor="let device of devices" [ngValue]="device">
      {{ device.id }}
      </option>
      </select>

This field is required

      Subject: *

      <input class="form-input" type="text" id="subject" placeholder="Enter subject" type="text" formControlName="subject"
         required>

This field is required

Please note maximum length is 80 characters.

      Please describe your issue:

      <textarea class="form-input" id="describe" placeholder="Describe your issue" rows="3" formControlName="issue"></textarea>

The more information you can provide here, the easier time the organization will
         have in diagnosing and resolving your issue.

      <button class="btn btn-link">Cancel</button>

      <button class="btn btn-primary" type="submit">Submit</button>

</form>

TS file
onSubmit(serviceForm) {
    console.log("Service form submitted");
      var data = "u_caller_id=" + serviceForm.u_caller_id + "&u_destination_country=" + serviceForm.u_destination_country + "&u_requester_phone_number=" + serviceForm.u_requester_phone_number;
      this.http.post("https://demo1049220.mockable.io/api/incident", data).subscribe((data) => {});
      console.log("data has gone");
  }
}

service.ts (to get the web api data)
serviceApiUrl: string = 'https://demo1049220.mockable.io/api/incident';

  constructor(
    private http: HttpClient,

    ) { }

  public getAll() {
    return this.http.get(this.serviceApiUrl);
    }
  }

So I am a little confused on how to achieve this, any help would be great.
Answer: It seems you are trying to call onSubmit() but on your .ts file, you have onSubmit(serviceForm), since you are using a formGroup, you can get the values from that with "this.serviceForm.value", not pass to onSubmit function as an argument. So you can use it like this:
onSubmit() {
var data = "u_caller_id=" + this.serviceForm.value.u_caller_id + "&u_destination_country=" + this.serviceForm.value.u_destination_country + "&u_requester_phone_number=" + this.serviceForm.value.u_requester_phone_number;
this.http.post("https://demo1049220.mockable.io/api/incident", data).subscribe((res) => {});
console.log("data has gone");
}

This is the correct way to pass form values to your http request. Also name "data" can shadow result (data) from your subscription so change its name to something like (res) like I did.
Answer: On top of ts file, import "HttpHeaders" from "@angular/common/http":
import { HttpHeaders } from @angular/common/http;

Then update your onSubmit method like below:
onSubmit(){
   this.http.post("https://demo1049220.mockable.io/api/incident",
      this.serviceForm.value,
      {
         headers : new HttpHeaders().set("Content-Type","application/json")
      }
   ).subscribe((response : any)=>{
       console.log(response);//On success response
   },(errorResponse : any)=>{
       console.log(errorResponse);//On unsuccessful response
   });
}

If the data has been sent successfully and server sends a successful response with 200 Ok, you will get a successful response in the first callback function, or else the second callback function will be called and your response will be in the "errorResponse" variable.


identificando provedor de email na lista

Question: acredito que não exista pergunta irrelevante então lá vai:
Estou começando a estudar python para data analytics e me deparei com um desafio de ler um .csv com alguns emails e separa os provedores deste arquivo em variaveis. A lógica eu até consigo compreender, mas por falta de conhecimento, estou apanhando para a sintaxe da linguagem.
Segue a minha tentativa muito falha: https://drive.google.com/drive/folders/1u2sGyeOw9rZlxStlslE3euot8j1zn_i3?usp=sharing
Comment: Por favor, edite a pergunta para limitá-la a um problema específico com detalhes suficientes para identificar uma resposta adequada.
Answer: Utilize a função split do python para manusear uma string.


Formulário Ajax, Javascript e PHP

Question: Como posso fazer um formulário com um campo UF e um campo de seleção com label Cidades que será preenchido através de AJAX e que ao selecionar a UF a função JS faz a conexão com o servidor e retorna as cidades daquela UF, mas sem pesquisa no Banco de Dados? Fui orientada a fazer com uma estrutura switch-case ou algo semelhante que retorne valores a partir de um valor chave, mas não faço ideia de como isso funciona. Só tenho uma ideia de como fazer utilizando o banco de dados.
OBS: Resolvi fazer com o banco de dados mesmo, por achar que é mais prático.
Já fiz o formulário:
<form name="formCadastro" action="buscaCidade.php" method="post">

    Nome
    <input type="text" name="name" placeholder="Digite seu nome" required>

    E-mail
    <input type="email" name="email" placeholder="Digite seu email">

    UF
    <select name="uf">
    </select>

    Cidade
    <select name="cidade">

    </select>

    <input type="submit" name="submit" value="Enviar">

</form>

E também fiz o banco, que seria algo mais ou menos assim:
DROP TABLE IF EXISTS `cidades`;
DROP TABLE IF EXISTS `estados`;

CREATE TABLE `cidades` (
  `estados_cod_estados` int(11) DEFAULT NULL,
  `cod_cidades` int(11) DEFAULT NULL,
  `nome` varchar(72) COLLATE utf8_unicode_ci DEFAULT NULL,
  `cep` varchar(8) COLLATE utf8_unicode_ci DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

CREATE TABLE `estados` (
  `cod_estados` int(11) DEFAULT NULL,
  `sigla` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
  `nome` varchar(72) COLLATE utf8_unicode_ci DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Agora, estou finalizando o PHP.
Comment: O que você ja fez?
Comment: Fiz o HTML, o BD e agora estou finalizando o PHP.
Answer: Jamile será que um uso de uma API com os estados e cidades iria te ajudar? não sei se iria resolver mas aqui está o que estava pensando, desse jeito você iria ter a opção de cidade e estados sem precisar fazer um select no banco e nem mesmo ter esses valores guardado neles

//jquery var do select dos estados
var $selectUf = $("#selectUf");

//jquery var do select das cidades
var $selectCidades = $("#selectCidade");

//será atribuido um valor nessa variável sempre que ele escolher um UF
var estadoSelecionado;
var cidadeSelecionada;

//uma função que pega os estados da API e o parametro dela recebe uma função que lida com essa resposta
var getEstados = function(responseFunction){
  $.ajax({
    method: "GET",
    url: "http://api.londrinaweb.com.br/PUC/Estados/BR/0/10000",
    contentType: "application/json; charset=utf-8",
    dataType: "jsonp",
    async:false,
    success: responseFunction
  });
}

//mesma ideia da função de cima porém nessa pegaremos a cidade de acordoo com a UF escolhida
var getCidades = function(estadoSelecionado,responseFunction){
  $.ajax({
    method: "GET",
    url: "http://api.londrinaweb.com.br/PUC/Cidades/"+estadoSelecionado+"/BR/0/10000",
    dataType: "jsonp",
    success: responseFunction
  });
}

//função que lida com a resposta da api e lista os estados no select
var populandoSelectUf = function(response) {
  $.each(response,function(i,item){
    $selectUf.append('<option value="'+item.UF+'"> '+item.UF+' </option>');
  })
}

//essa função só é chamada quando escolhe algum estado da UF
var populandoSelectCidades = function(response){
  //limpando o html do select
  $selectCidades.html("");
  $.each(response,function(i,item){
    $selectCidades.append('<option value="'+item+'">'+item+'</option>')
  });
}

getEstados(populandoSelectUf);

//sempre quando escolher uma UF coloca o valor selecionado na variavel estado selecionado
//e depois disso lista as cidades correspodente a UF Escolhida
$selectUf.change(function(){
  estadoSelecionado = $(this).val();
  alert("voce selecionou: "+estadoSelecionado);
  getCidades(estadoSelecionado,populandoSelectCidades);
});

$selectCidades.change(function(){
  cidadeSelecionada = $(this).val();
  alert("voce selecionou: "+cidadeSelecionada);
});

<form name="formCadastro" action="buscaCidade.php" method="post">

  Nome
  <input type="text" name="name" placeholder="Digite seu nome" required>

  E-mail
  <input type="email" name="email" placeholder="Digite seu email">

  UF
  <select name="uf" id="selectUf">
  <option selected="true" disabled>UF</option>
  </select>

  Cidade
  <select name="cidade" id="selectCidade">

  </select>

  <input type="submit" name="submit" value="Enviar">

</form>
Comment: Reynnan, até ajudaria e eu agradeço muito sua gentileza em me ajudar. Mas eu tenho especificidades a seguir, são elas: Um campo de seleção com label Cidade que será preenchido através de AJAX com, pelo menos, 5 cidades de cada UF. Ao selecionar a UF a função JS faz a conexão com o servidor e retorna as cidades daquela UF. Não é necessário fazer pesquisa a banco de dados, pode utilizar uma estrutura switch-case ou algo semelhante que retorne valores a partir de um valor chave. Um botão do tipo button com value "Exibir". No onclick, exibir um alert com a mensagem mostrando a UF e a cidade.
Comment: @JamileLima então você poderia fazer algo tipo a resposta de um JSON como na API, ao selecionar a UF, vc daria um $.get para a  url desejada enviando o valor da UF escolhida assim nessa url vc verificava qual a uf escolhida if $_GET['uf'] == "SE" echo "Aracaju, Itabaiana..." e com a resposta dessa URL você listaria as cidades da UF escolhida, desculpa se ficou um pouco atrapalhada essa comentário mas basicamente iria trabalhar bem parecido como a API.
Comment: Obrigada pela ajuda, Reynnan! Vou tentar aqui!

Formulário Ajax, Javascript e PHP


symantec backupexec 11d slow backup

Question: I have a IBM blade center H with modules HS21 type 8853 with Windows 2003R2 (32 bits). My backup is very slow (transfer starts with 16MB/min and slow down o 3 MB/min). I have update all firmware, all driver, nothing help. Any sugestions what can be wrong? Size of a backuped data no matter. It can be 100MB or 10GB, effect always is the same
Comment: What kind of tape drive and SCSI interface are you using?
Comment: tape drive is Overland ARCvault and SCSI interface is a Adaptec
Answer: This might not apply to 11d, but in the past, I've noticed slower than usual backup speeds when not using the remote agents.
Comment: I'm using remote agents on all servers. But I notice interesting thing. When I use disk no LTO to store backup speed is OK. My LTO is Overland ARCvault 24
Answer: Most speed issues I have seen have to do with the SCSI settings for the tape drive.  You will want to determine the SCSI ID of the tape drive (not the library) and enter the SCSI BIOS.  In the BIOS, enable "Disconnection", disable "Sync Negotiation", and then set the transfer rate to the maximum for your device (maybe 160MB/s).  Also, make sure your SCSI chain is correctly terminated.
Comment: I set "enable disconetion" to 'NO'(default was 'YES' and max speed 320) but I don't have "sync negotiation" option. Max speed sets automaticly to 160MB. Still not good
Answer: Problem was in LTO driver. I changed driver from symantec to Microsoft native and speed increase to max.

symantec backupexec 11d slow backup


Custom Ubuntu/Apache docker image in Azure App Service is using IIS?

Question: I have a mystery. So I set up an Azure Web app on Linux that pulls from a custom docker image stored in Azure container repo. The dockerfile for the image is pretty simple - ubuntu:latest as the base, installs apache, php, and wordpress using apt-get, sets config files, etc. I can bring up the Wordpress site just fine and do all the expected operations in the administration of it. Except none of the images load (png/jpg/svg/woff/woff2), even if I upload a new image in the wp-admin interface. Every image gives a 502 Bad gateway error.
Looking a bit deeper at the server response, the response headers for images say "Server: Microsoft-IIS/10.0." Wth? How did IIS get on here? If I look at the response for the base WP home page (or any text/html/php file), it says "Server: Apache/2.4.41 (Ubuntu)" as expected.
The bad gateway error makes me think somehow IIS has been set up as a proxy for certain file types, but I haven't set up any proxies in my configs (I may set up nginx in the future, but that's a story for another day). I'm fairly new to Azure web apps, so I wouldn't be surprised if I'm missing something obvious. Any ideas?
Answer: App service has a number of components that sit in front of your actual web app to deal with load balancing etc. I would not be surprised if some of these are running IIS.
Answer: I think this may have been a remnant of old configuration. I deleted my app service container, waited overnight, rebuilt, and now I don't have that problem anymore. Sorry to bother.


Add a new feature to existing rails project

Question: I have recently downloaded a new project(open source),i found certain features missing like blog,forum ,chat etc.. ..so I like to add those features to the project .My problem if run rails forum it will create a new rails project but i want to add to the existing project. I have found business logic. . . .
I had created models
ruby script/generate model forum
ruby script/generate model topic
ruby script/generate model post
rake db:migrate
ruby script/generate migration add_foreign_to_topics forum_id:integer
ruby script/generate migration add_foreign_to_post topic_id:integer
rake db:migrate

Then i ran
ruby script/generate controller forum

it was asking should i overwrite or not,so i am stuck up here,i need to create a controller and view for this feature.I am following this tutorial http://net.tutsplus.com/tutorials/other/building-a-forum-from-scratch-with-ruby-on-rails/ and i have already user table etc..
Answer: As far as i can read, you are not following the tutorial, as it does a scaffold which generates the controller and models at the same time.
Either you do something like
ruby script/generate scaffold Forum title:string contents:text

and it generates the model, controller, routes and views for you. In the tutorial they use nifty_scaffold and i think it mostly improves the view.
If you create the models seperately, you need to do something like
ruby script/generate controller Forum index show create edit update new destroy

and then you will have to fill in all those actions yourself. You will also have to set your routes correctly. That is not bad and not at all difficult. But when you are starting, using the scaffold is much easier.


SQL Order by from another table

Question: I'm trying to display my posts by upvote on my page.
My table posts:
<code>(id, post_name, post_description, post_category, post_subcategory)
</code>
and table votes:
<code>(id, post_id, vote_rank)
</code>
I can display all my post with
<code>$bdd->query('SELECT * FROM posts WHERE post_subcategory = "'.$_GET['s'].'" ORDER BY post_name');
</code>
I tried with JOIN but nothing appears.
<code>$bdd->query('SELECT *
            FROM posts AS p
               JOIN votes as v ON v.post_id = p.id
            WHERE p.post_subcategory =  "'.$_GET['s'].'"
            ORDER BY v.vote_rank');
</code>
Any idea?
Comment: Please do not compose queries with concats: use parameters such that the content is escaped.
Comment: Edit your question and provide sample data and desired results.
Answer: <code>$bdd->query('SELECT p.*
        FROM posts p LEFT JOIN votes v ON v.post_id = p.id
        WHERE p.post_subcategory =  "'.$_GET['c'].'"
        ORDER BY v.vote_rank');
</code>
Try It
Answer: Try using the inner join
<code>$bdd->query('SELECT p.*
        FROM posts p inner join votes v on v.post_id = p.id
        where p.post_subcategory =  "'.$_GET['c'].'"
        order by v.vote_rank');
</code>
Comment: Doesn't work with `inner join` but works with `left join`

SQL Order by from another table


How to change the header color of csv in java using ICsvBeanWriter

Question: I am using ICsvBeanWriter to generate the CSV file over the REST API, my code looks like
// uses the Super CSV API to generate CSV data from the model data
    ICsvBeanWriter csvWriter = new CsvBeanWriter(response.getWriter(),
    CsvPreference.STANDARD_PREFERENCE);

    String[] header = { "Title", "Description", "Author", "Publisher",
                    "isbn", "PublishedDate", "Price" };

    csvWriter.writeHeader(header);

I want to set the header color to some different color rather than the default one. Please tell me how i can do this?
Answer: As per definition, CSV is comma-separated values stored as plain text. It's a plain text file after all. So no, there is no way of adding colour via Java nor via any editor.
Answer: There are no colors in CSV.
The CSV is a textual, data-only format that does not allow any styling information.


How to get meta properties of a website and store the values to variables?

Question: I am trying to get a website meta properties such as price, availability, etc and store them to database.
I got the values by array but can't extract it.
What I want to do is, Extract the data from the array and assign them to separate variables.
Below I shared the code that I used to extract the data from the website.
<code>$content = file_get_contents('http://184.108.40.206/product/oneplus-7-pro-8gb-ram-256gb-price-in-qatar/');

$doc = new DOMDocument();

// squelch HTML5 errors
@$doc->loadHTML($content);

$meta = $doc->getElementsByTagName('meta');
foreach ($meta as $element) {
    $tag = [];
    foreach ($element->attributes as $node) {
        $tag[$node->name] = $node->value;
    }
    $tags []= $tag;
}
print_r($tags);
</code>
This is the result I am getting from the website by using the above code.
<code>Array ( [0] => Array ( [charset] => UTF-8 ) [1] => Array ( [name] => viewport [content] => width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no ) [2] => Array ( [http-equiv] => X-UA-Compatible [content] => ie=edge ) [3] => Array ( [name] => description [content] => Buy Oneplus 7 Pro 8GB RAM + 256GB at Low Price in Qatar and Doha - Get Free Home Delivery Inside Doha - Also, Enjoy Free Shop Pickup to See Product Directly. ) [4] => Array ( [name] => robots [content] => noindex,follow ) [5] => Array ( [property] => og:locale [content] => en_US ) [6] => Array ( [property] => og:type [content] => article ) [7] => Array ( [property] => og:title [content] => Buy Oneplus 7 Pro 8GB RAM + 256GB Price in Qatar - AlaneesQatar.Qa ) [8] => Array ( [property] => og:description [content] => Buy Oneplus 7 Pro 8GB RAM + 256GB at Low Price in Qatar and Doha - Get Free Home Delivery Inside Doha - Also, Enjoy Free Shop Pickup to See Product Directly. ) [9] => Array ( [property] => og:url [content] => http://184.108.40.206/product/oneplus-7-pro-8gb-ram-256gb-price-in-qatar/ ) [10] => Array ( [property] => og:site_name [content] => Al Anees Qatar ) [11] => Array ( [property] => article:publisher [content] => https://www.facebook.com/alaneestrading/ ) [12] => Array ( [property] => og:image [content] => http://220.127.116.11/wp-content/uploads/2019/07/one01-min-1.png ) [13] => Array ( [property] => og:image:width [content] => 570 ) [14] => Array ( [property] => og:image:height [content] => 706 ) [15] => Array ( [name] => twitter:card [content] => summary_large_image ) [16] => Array ( [name] => twitter:description [content] => Buy Oneplus 7 Pro 8GB RAM + 256GB at Low Price in Qatar and Doha - Get Free Home Delivery Inside Doha - Also, Enjoy Free Shop Pickup to See Product Directly. ) [17] => Array ( [name] => twitter:title [content] => Buy Oneplus 7 Pro 8GB RAM + 256GB Price in Qatar - AlaneesQatar.Qa ) [18] => Array ( [name] => twitter:site [content] => @alaneesqatar ) [19] => Array ( [name] => twitter:image [content] => http://126.96.36.199/wp-content/uploads/2019/07/one01-min-1.png ) [20] => Array ( [name] => twitter:creator [content] => @alaneesqatar ) [21] => Array ( [name] => generator [content] => WordPress 5.2.2 ) [22] => Array ( [name] => msapplication-TileImage [content] => http://220.127.116.11/wp-content/uploads/2017/11/cropped-alanessqatar-icon-1-270x270.png ) [23] => Array ( [property] => og:type [content] => og:product ) [24] => Array ( [property] => product:plural_title [content] => Oneplus 7 Pro 8GB RAM + 256GB ) [25] => Array ( [property] => product:availability [content] => InStock ) [26] => Array ( [property] => product:price:amount [content] => 2999.00 ) [27] => Array ( [property] => product:price:currency [content] => QAR ) [28] => Array ( [property] => product:retailer_item_id [content] => 15545 ) )
</code>
I want to store the product:price:currency values to a variable called $price and the product:availability to a variable $availability like that.
Thank you.
Answer: This will help you
<code>$content = file_get_contents('http://126.96.36.199/product/oneplus-7-pro-8gb-ram-256gb-price-in-qatar/');

$doc = new DOMDocument();

// squelch HTML5 errors
@$doc->loadHTML($content);
$meta = $doc->getElementsByTagName('meta');
$price = $availability = NULL;
foreach ($meta as $element) {
    $tag = [];
    foreach ($element->attributes as $node) {
        $tag[$node->name] = $node->value;
    }

    if(isset($tag['property']) && isset($tag['content'])) {
        switch($tag['property']) {
            case "product:price:amount":
                $price = $tag['content'];
                break;
            case "product:availability":
                $availability = $tag['content'];
                break;
        }
    }
}
</code>
Comment: I have updated the answer. You need to check if the `property` and `content` is set or not using `isset`
Comment: I got the price and availability but I am also expecting some error like this ```Notice: Undefined index: property in C:\xampp\htdocs\test\tracker.php on line 17``` - How to avoid this error?


Get the OS of an instance from an AMI

Question: Is it possible to get the OS type from an AWS AMI via Boto?
ie. MSWinSQLWeb, RHEL, SLES, LINUX/UNIX, etc....?
-RBL
Answer: Not exactly sure what you are trying to achieve, but this may be it:
boto.ec2.get_only_instances()

The returned Instance objects will have some informative properties, like:

platform – Platform of the instance (e.g. Windows)
kernel – The kernel associated with the instance.
architecture – The architecture of the image (i386|x86_64).
product_codes – A list of product codes associated with this instance.
Comment: Platform and Kernel are quite often 'None' and product_codes is quite often '[ ]'. I think information needs to be retrieved from the AMI

Get the OS of an instance from an AMI


How can I know If a webcomponent is on screen in litelement?

Question: Im doing a small proyect with litelement, but I want to fire an event when a webcomponent is shown, not only the first time, but every time that the component shows, how can I do that?
thanks in advance
Answer: Intersection Observer API will help you find if the element is on the screen. For more information, visit MDN docs.
<code>// 1. Observer options
let options = {
  // The scroll area - mostly body or document
  root: document.querySelector('#parent'),
  rootMargin: '0px',
  threshold: 1.0
};

let callback = () => { /* Do something */ };

// Observe Element
let observer = new IntersectionObserver(callback, options);

// Start observing
const targetElm = document.querySelector('#target');
observer.observe(targetElm);
</code>
The API is supported on all browsers except IE but a polyfill is already available:
https://github.com/w3c/IntersectionObserver


Every bounded set $E\subseteq\mathbb{R}^2$ is contained in a disc of minimal radius

Question: Consider the following proof from the book "The Banach-Tarski Paradox" by Stan Wagon:

In the second paragraph he seems to implicitely use that every bounded set $E\subseteq\mathbb{R}^2$ is contained in a unique disc of minimal radius. Now the uniqueness of such a disc is actually easy to see, but I don't see why it must necessarily exist.
Comment: What's your definition of bounded?  Is your problem that there will be A disc or that there will be a MINIMAL-radius disc?
Comment: The minimal part. In other words why is there a circle which realizes the inf of radii of enclosing circles
Answer: Let's suppose that $D(p_0,r_0)$ is a closed disc with center $p_0$ and radius $r_0$ that contains $E$.
Let $CR \subset \mathbb R^2 \times [0,\infty) \subset \mathbb R^3$ be the set of ordered pairs $(q,s)$ such that the closed disc $D(q,s)$ contains $E$ and such that $s \le r_0$.
Once you believe that the set $CR$ is closed and bounded, it follows that $CR$ is compact, and therefore the projection function $CR \mapsto [0,\infty)$ has compact image and hence has a minimum.
The set $CR$ is bounded, because if $(p,r) \in CR$ then $r \le r_0$, and then by picking any $q \in E$ one sees that $d(p,p_0) \le d(p,q) + d(q,p_0) \le r + r_0 \le 2 r_0$.
The set $CR$ is closed, because if $(p,r) \in \mathbb R^2 \times [0,\infty)$ is a limit of a sequence $(p_i,r_i) \in CR$ then for each point $q \in E$ we have $d(q,p_i) \le r_i$, and since distance is continuous we may pass to the limit to deduce that $d(q,p) \le r$.
Answer: First note, that C and its interior is a closed disc.
This is a little bit messy, but:  You take the closure.  (Call this F.)  This must be a subset of the smallest (closed) disc, if one exists.  Now the closure is closed and bounded, so it's compact, and there are two points that realize the diameter.  (Call these x and y.)  If you take the closed disc for which this diameter is the diameter of the disc, I claim it would be such a smallest closed disc.
First if you had a smaller closed disc, then it's subset, F, would have a bigger diameter, a contradiction.  Second, if there were points outside of this disc in F, you could use triangle inequality to argue that that point along with either x or y form a longer diameter in F.


Tensorflow: Proper queue closing between sessions

Question: I have two graphs (one for training and one for evaluation), which use the same inference network, but different input data. The inputs are created by reading binary data files. The graphs are run one after another and in seperate sessions. It actually seems to work, but I can't get rid of a warning that occurs everytime I close a session:
W tensorflow/core/common_runtime/executor.cc:1102] 0x7fb7ac082980 Compute status: Cancelled: Enqueue operation was cancelled
 [[Node: input_producer/input_producer_EnqueueMany = QueueEnqueueMany[Tcomponents=[DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](input_producer, input_producer/RandomShuffle)]]
I tensorflow/core/kernels/queue_base.cc:286] Skipping cancelled enqueue attempt

My question is, how do I properly close all queues and threads in order to avoid the warning. I wrote a little example, which shows what I'm doing and hopefully lets you reproduce my problem.
import tensorflow as tf
import struct
import numpy as np

file_name = 'test.bin'

def write_binary_file():
    with open(file_name, 'w') as f:
        for i in range(20):
            # image = np.ones([3, 3])*i
            image = np.zeros([3, 3])
            image[:, 0] = i
            image = image.astype('uint8')
            for u in range(3):
                for v in range(3):
                    f.write('%s' % struct.pack('B', image[u, v]))

# write binary file
write_binary_file()
print "size of one entry"
print 9
print "size of file"
print 20*9

# GRAPH DEFINITION
filename_queue = tf.train.string_input_producer([file_name])

reader = tf.FixedLengthRecordReader(header_bytes=0, record_bytes=9)
key, value = reader.read(filename_queue)

value_uint8 = tf.reshape(tf.decode_raw(value, tf.uint8), [3, 3])

# FIRST session
sess = tf.Session()
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)
for i in range(5):
    key_v, value_uint8_v = sess.run([key, value_uint8])
    print key_v
    print value_uint8_v

coord.request_stop() # *HERE* the warning is thrown
coord.join(threads, stop_grace_period_secs=5)
sess.close()

# SECOND session
sess2 = tf.Session()
coord2 = tf.train.Coordinator()
threads2 = tf.train.start_queue_runners(sess=sess2, coord=coord2)
for i in range(5):
    key_v, value_uint8_v = sess2.run([key, value_uint8])
    print key_v
    print value_uint8_v

coord2.request_stop()
coord2.join(threads2, stop_grace_period_secs=5)
sess2.close()

Do you know how to fix the warning or do you know a better way to change the data source?
Comment: Which version of TensorFlow ? In earlier versions this message was harmless and impossible to get rid of, but it may possibly be fixed in head
Comment: I'm using 0.8.0, not sure if that is considered old.
Comment: The changes I'm thinking about are in a later version (compiled from head)
Answer: As Yaroslav said; I believe this warning is fixed in more recent versions of TF.


Unable to list contents/remove directory (linux ext3)

Question: System is CentOS5 x86_64, completely up to date.
I've got a folder that can't be listed (ls just hangs, eating memory until it is killed).  The directory size is nearly 500k:
root@server [/home/user/public_html/domain.com/wp-content/uploads/2010/03]# stat .
  File: `.'
  Size: 458752          Blocks: 904        IO Block: 4096   directory
Device: 812h/2066d      Inode: 44499071    Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 3292/ user)   Gid: ( 3287/ user)
Access: 2012-06-29 17:31:47.000000000 -0400
Modify: 2012-10-23 14:41:58.000000000 -0400
Change: 2012-10-23 14:41:58.000000000 -0400

I can see the file names if I use ls -1f, but it just repeats the same 48 files ad infinitum, all of which have non-ascii characters somewhere in the file name:
La-critic\363-al-servicio-la-privacidad-300x160.jpg

When I try to access the files (say to copy them or remove them) I get messages like the following:
lstat("/home/user/public_html/domain.com/wp-content/uploads/2010/03/Sebast\355an-Pi\361era-el-balc\363n-150x120.jpg", 0x7fff364c52c0) = -1 ENOENT (No such file or directory)

I tried altering the code found on this man page and modified the code to call unlink for each file.  I get the same ENOENT error from the unlink call:
unlink("/home/user/public_html/domain.com/wp-content/uploads/2010/03/Marca-naci\363n-Madrid-150x120.jpg") = -1 ENOENT (No such file or directory)

I also straced a "touch", grabbed the syscalls it makes and replicated them, then tried to unlink the resulting file by name.  This works fine, but the folder still contains an entry by the same name after the operation completes and the program runs for an arbitrarily long time (strace output ended up at 20GB after 5 minutes and I stopped the process).
I'm stumped on this one, I'd really prefer not to have to take this production machine (hundreds of customers) offline to fsck the filesystem, but I'm leaning toward that being the only option at this point.  If anyone's had success using other methods for removing files (by inode number, I can get those with the getdents code) I'd love to hear them.
(Yes, I've tried find . -inum <inode> -exec rm -fv {} \; and it still has the problem with unlink returning ENOENT)
For those interested, here's the diff between that man page's code and mine.  I didn't bother with error checking on mallocs, etc because I'm lazy and this is a one-off:
root@server [~]# diff -u listdir-orig.c listdir.c
--- listdir-orig.c      2012-10-23 15:10:02.000000000 -0400
+++ listdir.c   2012-10-23 14:59:47.000000000 -0400
@@ -6,6 +6,7 @@
 #include <stdlib.h>
 #include <sys/stat.h>
 #include <sys/syscall.h>
+#include <string.h>

 #define handle_error(msg) \
        do { perror(msg); exit(EXIT_FAILURE); } while (0)
@@ -17,7 +18,7 @@
    char           d_name[];
 };

-#define BUF_SIZE 1024
+#define BUF_SIZE 1024*1024*5

 int main(int argc, char *argv[])
 {
@@ -26,11 +27,16 @@
    struct linux_dirent *d;
    int bpos;
    char d_type;
+   int deleted;
+   int file_descriptor;

    fd = open(argc > 1 ? argv[1] : ".", O_RDONLY | O_DIRECTORY);
    if (fd == -1)
        handle_error("open");

+   char* full_path;
+   char* fd_path;
+
    for ( ; ; ) {
        nread = syscall(SYS_getdents, fd, buf, BUF_SIZE);
        if (nread == -1)
@@ -55,7 +61,24 @@
           printf("%4d %10lld  %s\n", d->d_reclen,
                   (long long) d->d_off, (char *) d->d_name);
           bpos += d->d_reclen;
+          if ( d_type == DT_REG )
+          {
+              full_path = malloc(strlen((char *) d->d_name) + strlen(argv[1]) + 2); //One for the /, one for the \0
+              strcpy(full_path, argv[1]);
+              strcat(full_path, (char *) d->d_name);
+
+              //We're going to try to "touch" the file.
+              //file_descriptor = open(full_path, O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666);
+              //fd_path = malloc(32); //Lazy, only really needs 16
+              //sprintf(fd_path, "/proc/self/fd/%d", file_descriptor);
+              //utimes(fd_path, NULL);
+              //close(file_descriptor);
+              deleted = unlink(full_path);
+               if ( deleted == -1 ) printf("Error unlinking file\n");
+              break; //Break on first try
+          }
        }
+       break; //Break on first try
    }

    exit(EXIT_SUCCESS);

Comment: If you are willing to remove the whole thing, what does "rm -fr 03" do when done from in the parent directory "/home/user/public_html/domain.com/wp-content/uploads/2010"? You may wish to strace that command, too.
Comment: Unfortunately that has the same result, it hangs infinitely.  An strace shows that it's looping over the same getdents calls as an ls.
Comment: Can you provide a `df -h` listing and the output of `mount`?
Answer: I am presumming you are doing this on an ACTIVE filesystem.  Thus it is possible that while you are doing the find and such, the file got deleted before it could be processed by find.  This would be ok.
What I might do to get a list of files is NOT to use ls.  ls tries to do a sort and with a directory that size it would take pretty long just to get the list and then get it sorted.
What I do in such cases is (as the root user):
 find dirname -ls >outfile

If you want to delete something based on times:
 find dirname -type f -mtime +60 -print0 | xargs -0 rm -f

BTW, the -0 and -print0 are options in Linux so that filenames with "special" characters are passed properly to xargs.  The above of course removes files that have been modified GREATER THAN 60 days before now.
Comment: Of course, you could be right and the filesystem is in need of repair, in this case, it would be better to do a FULL fsck on the filesystem.
Comment: Hi mdpc, I appreciate you taking a look.  The C code above is performing essentially the same operation you've showed in the find here, including unlinking the file.  It's entirely possible that the files don't, in fact, exist but regardless I need to find out why the parent directory thinks they do and fix that problem.  find will not be able to solve the problem (I've tried `find -exec rm -f {} \;`, which should be more effective than using `xargs -0`.  The problem is that getdents() returns data forever, so the find will never actually finish getting info.  Even if it did, you can't unlink.
Answer: Here's an easy way to determine if the filesystem needs repair, or at least understand how extensive the damage is...
Download the (free) R1Soft/Idera Hot Copy snapshot utility. This is an RPM and kernel module that provides copy-on-write snapshots of Linux filesystems without the need to have LVM, etc. in place.
Let's say your filesystems look like:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              12G  4.4G  7.0G  39% /
tmpfs                  14G     0   14G   0% /dev/shm
/dev/sda1             291M  166M  110M  61% /boot
/dev/sda3             9.9G  2.5G  7.0G  26% /usr
/dev/sdb1             400G  265G  135G  67% /home

You could use Hot Copy to snapshot /home without mounting the resulting filesystem... Then run a fsck to see if there are any issues.
hcp --skip-mount /dev/sdb1
fsck -a /dev/hcp1

This can save you the time of the reboot and help you assess the severity before scheduling client downtime.
Long-term, I'd just use XFS as the filesystem... But that's another topic...
Comment: Hi ewwhite, thanks for this!  I'm going to try this on our development servers tonight and if all goes well I'll give it a go.  Seems like the only solution so far that has a chance of at least telling me that there is a problem.
Comment: @RedKrieg Did it all work out?
Answer: Don't use "find ... -exec rm -fv {} \;" use "find ... -delete" instead


MSSQL -> MySQL or PostgreSQL DB replication

Question: I find myself in a situation that someone else must have got stuck in at some point.
Our company runs an in house MSSQL based database, our website then sits on an external server and currently runs MySQL.  The problem comes that both databases need to communicate with each other increasingly often.
Our current solution involves SSIS packages but they are tempromental and often slow.  We end up very often having to drop tables on our live production server before then rebuilding them from the in house server.  Of course this is leading to data inconsistencies so we try and perform these operations outside of peak hours.  However, this leads to 24hr delays in some data being transferred from MySQL to MSSQL and then an additional 24hr delay in getting it back to MySQL.  None of this is ideal.
What I am looking for is a solution that will allow replication between our in-house MSSQL server and any database that can sit upon a linux box and play nicely with PHP.  As I said, we currently run MySQL so we have a preference to any solution that would include that but all other suggestions would be greatfully received.
I've read somewhere that PostgreSQL is a vialbe solution but I am now struggling to find where I read that.  I've come accross a few tools that might help such as DBMoto and Slony-I, does anyone have any experience of either in such a situation?
Alternatively, are we simply barking up the wrong tree altogether and should we be trying to integrate MSSQL into our PHP environment?
Any thoughts or suggestions would be gratefully recieved.
Many thanks
Rob
Comment: the information you provide is not enough to help... any bidirectional replication between two different DB is bound to have some problems... some result from differences in the DB, some from available replication option... any reason you don't use the same for both (MSSQL for both OR Postgresql for both OR Oracle for both...) ?
Comment: MSSQL+MySQL is the status quo here.  We don't have the option to change the in-house database but I would happily investigate the option of changing our web server to MSSQL if this is what you suggest.  I see that PHP and MSSQL talk relatively happy but would installing MSSQL on CentOS require some virtualisation solution?
Comment: MSSQL will need a Windows server... PHP can run on Windows too... so one option is switch to a Windows web server with MSSQL...
Comment: While I guess that is an option, I'd like to only use that as a final last resort.  We're rather happy with our linux server at the moment and switching to a windows server would be a big upheaval.
Comment: then I wish you luck that someone can give you a production-quality answer for your scenario...
Answer: The best solution that i know and i have used for this kind of job is SymmetricDS .

SymmetricDS is an asynchronous data replication software package that
  supports multiple subscribers and bi-directional synchronization.
Comment: Looks like a good option, is it pretty easy to set up and configure?
Comment: It is not too complicated to configure and there is a good documentation on the official site.
Comment: I've just noted that its compatability is limited to SQL Sever 2005, we run 2008 R2, any idea if that is likely to be compatable too?
Comment: I've used it also with SQL Server 2008 R2 with JTDS JDBC Driver .
Comment: Thanks, aleroot.  I'll be testing this in earnest next week.  I'll let you know how I get on.
Answer: You can access MSSQL servers from PHP using FreeTDS and the mssql_* family of functions (or classes built thereon, or alternative freetds-using extensions). We use this for our admin application, with a Linux/PHP website talking to a MSSQL database.
Comment: I came accross the mssql_* family of functions but as the two servers are geographically remote, I fear that the latency between the two would cause significant issues.  Is this true for your setup too?
Comment: No, our web server and database server are close together. (we do some communications over the longer distance from office to prod, but not as often, and latency does show up). But if you're already handling a mssql<->mysql replication over a slow link, a mssql<->mssql replication should be less troublesome? although I believe that will only work in a single-master mode.


Sample size difference between online calculator and R

Question: I am trying to find a proper sample size for a project I'm working on and just realized that the values I have calculated are significantly different from an online calculator (which is supposed to be the same).
If I calculate the sample size in this online calculator, the result is 1.6 million, whereas if I use the same parameters in R using a power analysis of two independent proportions, the estimated sample size I get is around 1,130:
power.prop.test(p1=0.15, p2=0.10, power=0.95, alternative='two.sided', sig.level=0.05)

Any ideas on why there is such a difference?
Comment: You also used 0.15% in the calculator and 15% in the R function. Mind your decimals.
Comment: The link you included also has a FAQ with an answer to the quetsion: "Why is your calculator different from other sample size calculators?". It doesn't answer your question completely I think. But my perspective: there are different ways to compute sample sizes for different types of tests. Clearly, these computations have very different assumption.s
Comment: I don't have time to check the documentation for the R function you've used, but I suspect that while the "p1" argument corresponds to the "baseline conversion rate", "p2" does not correspond to the "minimum detectable effect". Rather, the minimum detectable effect may be the difference between p1 and p2 (or some measure of [effect size](https://en.wikipedia.org/wiki/Effect_size))
Comment: In addition to that, the website doesn't say what the beta level it's using is (i.e. "power"), so can't be sure that it's .95, as is the case in that R function.
Comment: It looks like you specified the effect to the web calculator as 10. Is that what you meant to do?
Comment: yes, i specified 10 as the % of detectable change (relative)
Answer: The second parameter is the second proportion. So you need to define according to the MDE used in the online calculator. As other said, there is not information on the power.
With the following parameters we get same approximation
power.prop.test(p1=0.15, p2=0.15*(1-0.10), power=0.80, alternative='two.sided', sig.level=0.05)

     Two-sample comparison of proportions power calculation

              n = 8524.013
             p1 = 0.15
             p2 = 0.135
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

See this for more information https://signalvnoise.com/posts/3004-ab-testing-tech-note-determining-sample-size
Comment: This together with Pierre Lafortune's comment made me realize my issue. The sample size is indeed very big.


How does JSLint's function-within-a-loop error apply when using jQuery's .each()?

Question: I'm using a jQuery .each() call, which uses an anonymous function, from inside a for loop. JSLint comes up with the warning "Don't make functions within a loop".
Here's a code snippet from a larger function - in essence it is doing a check of whether each player of the game is still "alive" (has at least one piece on the board).
for( i=0 ; i<PLAYERLIMIT ; ++i ) {
    if( player[i].status !== 0 ) { //skip already dead players
        var stillAlive = false;
        $board.find("td").each( function() { //this func causes JSLint warning
            if( $(this).data("owner") === player[i].number ) {
                stillAlive = true;
                return false;
            }
        });
        if( !stillAlive ) {
            //... action to take for dead players
        }
    }
}

I can see how to remove this warning - just declare the function seperately and call it. But this is a very small one-shot function and essentially I consider this the body of a nested for loop, (I'm essentially reading the .each() call as something like for $("td") in $board {})
Is this JSLint providing one of it's style warning, or is this more serious?
Basically, is it best for me to fix this?
I'd like to understand the reason for the warning, so any comments RE the why the warning exists would be useful (again I want to know if it's practical or style).
Answer: It's just a style thing mainly, possibly some efficiency gained in IE, though newer JS engines will inline the function via their tracing engine anyway.  You can get rid of the JSLint "error" tough, like this:
function setupPlayer(player) {
  var stillAlive = false;
  $board.find("td").each( function() {
      if( $(this).data("owner") === player.number ) {
          stillAlive = true;
          return false;
      }
  });
  if( !stillAlive ) {
      //... action to take for dead players
  }
}

for(var i=0 ; i<PLAYERLIMIT ; ++i ) {
    if( player[i].status !== 0 ) { //skip already dead players
        setupPlayer(player[i]);
    }
}

Should you?  Personally I would ignore it in this case for the sake of maintainability, your current version is much easier to read/maintain, to me.  Your call here, which version suits you better?  Either way is likely to have little impact on performance.

Bit of a tangent but related:
What would have a bigger impact on performance is caching that  selector, DOM traversal is quite expensive.  I'd do this just before your for loop:
var cells = $board.find("td");

And use cells inside the loop, no need finding the same elements again :)
Comment: Good, style-wise I've happy with it as it stands, but given I've been unable to find anything that really *explains* the JSLint warnings it's difficult to know which I may safely disregard. Also, good catch on the `cells` suggestion - I'm doing that elsewhere already, but failed to utilise it here for some reason...  Thanks.
Comment: It _is_ actually more than just a style warning -- see @Sean McMillan's alternative answer below. The _only_ reason the original code example works correctly is that `$.each()` is guaranteed to run the function you pass it synchronously, before `$.each()` returns (and never run it after that). In any case where the function might be called _after the current iteration of its containing for loop_, then this form of code is dangerously bug-prone -- for the reasons described below.
Answer: So, I believe JSLint is concerned about something like this:
for (var i=0; i<10; i++) {
    $("#my-elem" + i.toString()).click(function() { alert(i)});
}

That will cause all elements with IDs my-elem0 - my-elem9 to alert "10" when clicked -- i is scoped to the containing function, not to the for loop. JSLint tries to protect you from this by telling you not to make functions in a loop. It's not smart enough to know that each will call your function now, and not later.


How to perform regression on panel-data with timelag in SPSS / PASW?

Question: For my thesis, I have gathered search volume data (svi) from Google and message data from Twitter (tweets is the number of daily tweets) for serveral companies (comp). The variable tradevol is the trading volume in the stock of a company, as taken from Yahoo! Finance.  svi and tweets are my independent variables, tradevol is dependent.
For argument's sake, say I have collected data over 3 days for each of 3 companies (in reality, I have data for  100 companies gathered during 200 days), as follows:
comp  date   svi  tweets  tradevol
-------------------------------------
1     02-12  1.07  223    2,209,425
1     02-13  1.03  200    2,021,502
1     02-14  1.10  196    2,124,555
2     02-12  0.55  110    1,942,211
2     02-13  0.45  211    1,532,453
2     02-14  0.41  104    1,432,655
3     02-12  1.05  303    1,765,273
3     02-13  1.08  250    1,932,672
3     02-14  1.09  277    1,597,892

A dataset like this with measurements over time goes beyond what has been tough during my studies. So I need to understand how to analyze this. Therefore, I have some questions analyzing this dataset in SPSS / PASW.

How can I, from this dataset, measure the correlation between svi and tradevol for each company? I would then somehow have to tell SPSS to split the datafile on comp, calculating the correlation for each unique comp
My thesis-coach calls this dataset a "panel dataset". However, searching for "paneldata analysis SPSS" I don't find much usefull information. If I want to perform a regression, measuring the effect of svi and tweets on tradevol, how is this then called? A multilevel regression?
Regarding regression, my coach wants me to account for a timelag. For instance, today's svi and tweets may not have an effect on today's tradevol but perhaps there is an effect (or: a bigger effect) of today's svi and tweets on tomorrows tradevol. In this case, I would have to perform the regression for lag t-2, t-1, t, t+1 and t+2. Is this operation possible to perform with SPSS (18) and if so, please send me somthing to go by :-)
Comment: `but perhaps there is an effect (or: a bigger effect) on today's svi and tweets and tomorrows tradevol` this was not quite clear meaning to me. Maybe there're typos?
Comment: "...but perhaps there is an effect (or: a bigger effect) of _today's_ svi and tweets on _tomorrow's_ tradevol" :-)
Comment: I've edited these typos for you
Answer: You may want to apply the Fama-MacBeth regression technique outlined in their 1972 paper (link is a PDF, couldn't find a regular citation page for it.) This is a crude method since it doesn't do much residual clustering or analysis, but it's easy to implement and almost certainly already exists in SPSS.
Instead of risk premium, you'll be calculating some sort of trade volume premium on your factors. The huge problem you're going to run into is between-company correlations, both in trade volume and in things like being newsworthy enough to tweet about. Since you only have a small number of observations per company, I don't think it's possible to estimate the correlations with much accuracy, without pre-specifying some sort of model that depends on the correlations and allows you to infer them.
Answer: For easy to running your data analysis, try the eviews software or stata, etc. In SPSS, if you want to running the time lag data analysis, you must perform the data, like 1 to 2  etc. For example:
x : 12, 13,12,17,9,12

You can create the new variable x1 : 13, 12, 17, 9 and 12. The first number data variable for x not available... and so on.
Comment: iam sorry, that i mean not for x first number data is not available, but for x1


Sortable example with Shopify Draggable

Question: I need to create a view similar to trello main view. Where I have categories and tasks sortable
I want to sort booth thru Sortable API.
When I turn categories Sortable active, when I try to sort tasks, it tries to also sort associate categories. I understand why it do this, but how to exclude? Is this possible to do with Shopify Dragable api?
https://shopify.github.io/draggable/
I am new to Draggable api, code example with my problem:

//task sortable
const sortable = new Draggable.Sortable(document.querySelectorAll('ul'), {
            draggable: 'li',
        });

<link href="https://adminlte.io/themes/v3/dist/css/adminlte.min.css" rel="stylesheet"/>
<link href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.5.0/css/bootstrap.min.css" rel="stylesheet"/>

Cat 1

Task 1

Task 2

Task 3

Task 4

Cat 2

Task 1

Task 2

Task 3

Task 4

Cat 3

Task 1

Task 2

Task 3

Task 4

Cat 4

Task 1

Task 2

Task 3

Task 4

Now with Tasks and Category Sortable

//task sortable
const sortable = new Draggable.Sortable(document.querySelectorAll('ul'), {
            draggable: 'li',
        });

//cat sortable
const sortableCat = new Draggable.Sortable(document.querySelectorAll('.main-div'), {
          draggable: '.category-swap',
});

<link href="https://adminlte.io/themes/v3/dist/css/adminlte.min.css" rel="stylesheet"/>
<link href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.5.0/css/bootstrap.min.css" rel="stylesheet"/>

Cat 1

Task 1

Task 2

Task 3

Task 4

Cat 2

Task 1

Task 2

Task 3

Task 4

Cat 3

Task 1

Task 2

Task 3

Task 4

Cat 4

Task 1

Task 2

Task 3

Task 4

Answer: The handle option can specify where to drag. You can set handle: ".card-header" like the snippet below to exclude sort the associate categories.
See: https://github.com/Shopify/draggable/tree/master/src/Draggable#options

//task sortable
const sortable = new Draggable.Sortable(
    document.querySelectorAll("ul"),
    { draggable: "li" }
);

//cat sortable
const sortableCat = new Draggable.Sortable(
    document.querySelectorAll(".main-div"),
    { draggable: ".category-swap", handle: ".card-header" }
);

<link
href="https://adminlte.io/themes/v3/dist/css/adminlte.min.css"
rel="stylesheet"
/>
<link
href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.5.0/css/bootstrap.min.css"
rel="stylesheet"
/>

Cat 1

Task 1

Task 2

Task 3

Task 4

Cat 2

Task 1

Task 2

Task 3

Task 4

Cat 3

Task 1

Task 2

Task 3

Task 4

Cat 4

Task 1

Task 2

Task 3

Task 4

Sortable example with Shopify Draggable


Intemporal budget by lagrange

Question: Assume that a representative agent lives forever and receives an endowment, denoted yt, in each period. The entire endowment sequence is known
with certainty on date 0. The representative agent discounts future utility at
rate $\beta$ and has a period utility function given by
$u(c_{t}) = (c_{t})^{1-\sigma}/(1-\sigma)  $
(d)write the intertemporal budget constraint of the representative
agent by aggregating the flow budget constraints.
Write down the Lagrangian using the constraint in (d). How many
Lagrange multipliers are there? Find the first order conditions and write down
the Euler equation.
I have found the equation
$ L = \beta^tU(c_{t}) + (\lambda)_{t}(1/(1+r)^t)(c_{t}-y_{t}) $
then I have found the equation by taking derivative of L with respect to $c_{t}$ and $c_{t+1}$ but since there are no savings(b) I could not take derivative of L with respect to b so cannot find the ratio of lamdas :
$ \frac{U^{'}(c_{t})}{U^{'}(c_{t+1})\beta} = \frac{(\lambda)_{t}}{(\lambda)_{t+1}(r+1)}$
Comment: `\lambda` and `\sigma` in $\LaTeX$ give $\lambda$ and $\sigma$
Comment: thank you do you have any idea about problem @Henry
Comment: @Pumpkin In the first equation a closing bracket is missing. Are the two $\sigma$-signs correct in the first equation ? What is the correct term ? You should also explain the meanings of $\lambda$ and $\sigma$.
Comment: $\lambda$ is lagrange multiplier other has no importance just a parameter @calculus
Comment: @Pumpkin And $\sigma$ ?
Comment: it is just a parameter for utility func. does not affect the solution
Comment: You're not supplying all the information. What exactly is the flow budget constraint for this problem? There is no savings? Then why would the agent not simply consume his endowment every period?
Comment: With savings, you would guess that the Euler equation looks like the usual trade-off between marginal utilities $U'(c_t) = \beta (1+r) U'(c_{t+1})$.
Comment: @Michael how did you find $\frac{(\lambda)_{t}}{(\lambda)_{t+1} = 1 $ ?
Answer: The lagrange-function should be look like this:
$\mathcal L=\sum_{t=0}^{n} \left( \beta ^t\cdot \frac{1}{1-\sigma}\cdot c_t^{1-\sigma}+ \lambda_t \left(  \frac{y_t}{(1+r)^t}- \frac{c_t}{(1+r)^t} \right) \right) $
$ y_t- c_t  =0\Rightarrow \text{No savings in period t}$
First order condition:
$\frac{d L}{d c_t}=\beta ^t\cdot \frac{1}{1-\sigma} \cdot (1-\sigma) c_t^{1-\sigma}-\frac{\lambda _t}{(1+r)^t}=0 \Rightarrow \lambda_t =\beta ^t\cdot \frac{(1+r)^t}{1-\sigma} \cdot (1-\sigma) c_t^{1-\sigma} \quad (1) $
$\frac{d L}{d c_{t+1}}=\beta ^{t+1}\cdot \frac{1}{1-\sigma} \cdot (1-\sigma) c_{t+1}^{1-\sigma}-\frac{\lambda _{t+1}}{(1+r)^{t+1}}=0 \Rightarrow \lambda_{t+1} =\beta ^{t+1}\cdot \frac{(1+r)^{t+1}}{1-\sigma} \cdot (1-\sigma) c_{t+1}^{1-\sigma} \quad (2) $
Dividing (2) by (1) gives $\frac{\lambda _{t+1} }{\lambda _{t}}=\beta \cdot (1+r) \frac{c_{t+1}^{1-\sigma}}{c_{t}^{1-\sigma}}$
It is the same as $\frac{\lambda _{t+1} }{\lambda _{t}}=\beta \cdot (1+r) \frac{U'(c_{t+1})}{U'(c_{t})}$
My relation is a little bit different from yours.
Comment: If agent is infinitely lived, why is your $n$ finite? If $y_t = c_t$ and $y_t$ is exogenous, what are you solving for?
Comment: n can go to infinity. The second point is a good point. But I haven`t seen other way to get the relation. If you have a better way, then it would be helpful to provide an answer.
Comment: $n$ can go to infinity? Really?
Comment: Sure, if the representative agent lives forever.

Intemporal budget by lagrange


UltraGridBagLayoutPanel "Stackable Controls"

Question: I am wondering if anyone has any idea how to make the controls stack and fill up the available space left by a control if the control is hidden or removed from the UltraGridBagLayoutPanel.
Example:
<code>[Textbox 1]
[Textbox 2]
[Textbox 3]
[Textbox 4]
[Textbox 5]
</code>
If I hide [Textbox 3] as of now, it will disperse the space left equally and pad the remaining text boxes with the space.  However, I would like it to do this...
<code>[Textbox 1]
[Textbox 2]
[Textbox 4]
[Textbox 5]
</code>
Where all the text boxes will move up and [Textbox 4] will completely consume the space left by [Textbox 3].
Any ideas?
Answer: You can add a panel for each textbox that you might hide to the GridBagLayoutPanel after the TextBox controls and set it's visibility to be the opposite of the TextBox visiblity.  As long as the Panel is the same size as the TextBox then it will reserve the same amount of space and the text boxes will move as you would like.


Javascript workaround to nest an array of objects

Question: I get some data in a flat array and I want to nest one inside another.
The attribute which defines the depth of an object is the DEPENDENCY where I split it in so as to get the different levels of nesting.
This is the data I get
const initArray = [
  { NAME: "initArray", DEPENDENCY: "0", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray2", DEPENDENCY: "0.0", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray3", DEPENDENCY: "0.1", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray4", DEPENDENCY: "0.0.0", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray5", DEPENDENCY: "0.0.1", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray6", DEPENDENCY: "0.0.2", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray7", DEPENDENCY: "0.1.1", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray8", DEPENDENCY: "0.1.2", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray9", DEPENDENCY: "0.1.3", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray10", DEPENDENCY: "0.1.4", VAR1: "VAR1", VAR2: "VAR2" },
  { NAME: "initArray11", DEPENDENCY: "0.1.5", VAR1: "VAR1", VAR2: "VAR2" },
];

and this the expected result
const myArray = [
  {
    NAME: "initArray",
    DEPENDENCY: "0",
    VAR1: "VAR1",
    VAR2: "VAR2",
    CHILDREN: [
      {
        NAME: "initArray2",
        DEPENDENCY: "0.0",
        VAR1: "VAR1",
        VAR2: "VAR2",
        CHILDREN: [
          {
            NAME: "initArray4",
            DEPENDENCY: "0.0.0",
            VAR1: "VAR1",
            VAR2: "VAR2",
          },
          {
            NAME: "initArray5",
            DEPENDENCY: "0.0.1",
            VAR1: "VAR1",
            VAR2: "VAR2",
          },
          {
            NAME: "initArray6",
            DEPENDENCY: "0.0.2",
            VAR1: "VAR1",
            VAR2: "VAR2",
         },
       ],
     },
     {
       NAME: "initArray3",
       DEPENDENCY: "0.1",
       VAR1: "VAR1",
       VAR2: "VAR2",
       CHILDREN: [
         {
           NAME: "initArray7",
           DEPENDENCY: "0.1.1",
           VAR1: "VAR1",
           VAR2: "VAR2",
         },
         {
           NAME: "initArray8",
           DEPENDENCY: "0.1.2",
           VAR1: "VAR1",
           VAR2: "VAR2",
         },
         {
           NAME: "initArray9",
           DEPENDENCY: "0.1.3",
           VAR1: "VAR1",
           VAR2: "VAR2",
         },
         {
           NAME: "initArray10",
           DEPENDENCY: "0.1.4",
           VAR1: "VAR1",
           VAR2: "VAR2",
         },
         {
           NAME: "initArray11",
           DEPENDENCY: "0.1.5",
           VAR1: "VAR1",
           VAR2: "VAR2",
         },
       ],
     },
   ],
  },
];

This can be done by the following code
let myArray = Array();

Object.keys(initArray).map((key) => {
  const dependencyTree = initArray[key].DEPENDENCY.split(".");

  if (dependencyTree.length === 1) {
    myArray[Number(dependencyTree[0])] = initArray[key];
  } else if (dependencyTree.length === 2) {
    if (!myArray[Number(dependencyTree[0])].hasOwnProperty("CHILDREN")) {
      myArray[Number(dependencyTree[0])].CHILDREN = Array();
    }

    myArray[Number(dependencyTree[0])].CHILDREN[Number(dependencyTree[1])] = initArray[key];
  } else if (dependencyTree.length === 3) {
    if (!myArray[Number(dependencyTree[0])].CHILDREN[Number(dependencyTree[1])].hasOwnProperty("CHILDREN")) {
      myArray[Number(dependencyTree[0])].CHILDREN[Number(dependencyTree[1])].CHILDREN = Array();
    }

    myArray[Number(dependencyTree[0])].CHILDREN[Number(dependencyTree[1])].CHILDREN.push(initArray[key]);
  }
});

but I would like a solution for nesting regardless the depth. Now I have to add manually the if statement for each depth level. Is there any workaround?
Comment: My answer on a similar question: [tree from array of dot-separated strings](https://stackoverflow.com/questions/62490323)
Comment: Your solution works, but it just adds a nested label. I need to add nested objects.
Comment: You just need to tweak a little. Instead of splitting the string, you need to split the property: `for (const obj of data) { let splits = obj.DEPENDENCY.split('.')` And instead of `const o = { label }`, you need `const o = { ...obj }`
Comment: Here's a fiddle: https://jsfiddle.net/adigas/7etafd25/
Answer: Nesting a flat list with arbitrary depth has common recursive algorithm.
The code below is a working example with your dataset.
Another thing to keep in mind is that this code does NOT mutate your original flat list
 function generateNesting(originalList, nestedList) {
      return nestedList.map((item) => {
        const { DEPENDENCY: parentDependency } = item;
        const childItems = originalList.filter((item) => item.DEPENDENCY.startsWith(parentDependency) && item.DEPENDENCY.length - 2 === parentDependency.length);

    return { ...item, CHILDREN: generateNesting(originalList, childItems) };
  });
}

function makeNested(list) {
  const rootItems = list.filter((item) =>
    item.DEPENDENCY.length === 1
  );

  return generateNesting(list, rootItems);
}

function searchByName(name, nestedList = []) {
    for(let i = 0; i < nestedList.length; i++) {
        const { NAME, CHILDREN } = nestedList[i]

        if(NAME === name) {
            return nestedList[i]
        }

        const found = searchByName(name, CHILDREN)

        if(found) {
            return found
        }
    }
}

const result = makeNested(initArray) // initArray is your flat list
console.log(searchByName('initArray3', result))
console.log(searchByName('initArray10', result))
console.log(searchByName('initArrayISNOTPRESENT', result)) // prints undefined

Comment: Ok now one more thing. How do I search for a name on the nested array and extract it to a single object? For example, let's say I search for the object with NAME = initArray4. Which is inside initArray2 object, which is inside initArray object. How to filter the whole nested array and keep only the initArray4 object?
Comment: @iNemesis
I added the searchByName function.
Searching (and/or traversing) is core to a nested/tree-like data structure.

I suggest you read about tree/nested data structures and related algorithms
Comment: Thank you so much for your replies. I definitely will read about tree/nested data structures for better understanding your code. Once again, i really appreciate your time


How often does the IFSC actually hold a world championship?

Question: The Wikipedia article says

The IFSC Climbing World Championships are the biennial world championships for competition climbing ...

but there was a world championship in 2018 and now again in 2019.
Is this information out of date? Why would the IFSC make this change, if they have done so?
Answer: You can currently find the IFSC rules on the official website, the current version as well as the 2018 version.
The first point in the introduction of chapter "world championship" states in the 2018 version (paragraph 12.1.1):

a World Championship competition shall be organised every second, even-numbered, year.

...while the same paragraph states in the 2019 version (paragraph 13.1.A):

the IFSC shall arrange for the organisation of the World Championship(s) every second, odd-numbered year

So that's why you saw world championships on two consecutive years (and why there shouldn't be any in 2020).
That answers your first question: this information is still correct.
I could not find information to the "why" part but my guess is that this is to "make room" for the 2020 Olympic games. Climbing will be represented there for the very first time. So the IFSC did not need to make this change earlier.


SQL Server: Insert record if not found in table from a function in Select

Question: I am using OPENROWSET for importing a .csv file into SQL Server 2014.
This is my query
SELECT
    ufnGetChargeTypeId([Charge Type]) AS ChargeTypeId, [Description]
FROM
    OPENROWSET ('MSDASQL',
                'Driver={Microsoft Access Text Driver (*.txt, *.csv)}; DBQ=D:\Files\;',
                'SELECT * from [myfile.csv]' )

[Charge Type] is column header in the .csv file. ufnGetChargeTypeId is a function which returns ChargeTypeId if found in table ChargeType which is a reference table.
Working fine if [Charge Type] exist. I want to insert [Charge Type] in table if not found and return ChargeTypeId. User defined functions don't support INSERT/UPDATE in table.
One solution is to use a stored procedure but I am calling function in Select statement and you can't exec stored procedure in Select statement.
Would you suggest any solution to my problem.
Comment: Dump to one staging table and then do comparisons
Answer: You could first insert all missing types, then run your query:
DECLARE @T TABLE([Charge Type] varchar(4000), [Description] text)

INSERT INTO @T
SELECT [Charge Type], [Description]
FROM
    OPENROWSET ('MSDASQL',
                'Driver={Microsoft Access Text Driver (*.txt, *.csv)}; DBQ=D:\Files\;',
                'SELECT * from [myfile.csv]' )

INSERT INTO ChargeTypes ([Charge Type])
SELECT DISTINCT [Charge Type]
FROM @T
WHERE [Charge Type] NOT IN (SELECT [Charge Type] FROM ChargeTypes)

SELECT
    ufnGetChargeTypeId([Charge Type]) AS ChargeTypeId, [Description]
FROM @T


How to get php-fpm to log to stdout / stderr when running in a docker container

Question: I have php-fpm in a docker container and in the Dockerfile I edit the fpm config file (/etc/php5/fpm/pool.d/www.conf) to set up access logs to go to /var/log/fpm-access.log and error logs to go to /var/log/fpm-php.www.log:
# Do some php-fpm config
#  Redirect worker stdout and stderr into main error log
#  Activate the fpm access log
#  Enable display errors
#  Enable the error log
RUN sed -i '/^;catch_workers_output/ccatch_workers_output = yes' /etc/php5/fpm/pool.d/www.conf && \
    sed -i '/^;access.log/caccess.log = /var/log/fpm-access.log' /etc/php5/fpm/pool.d/www.conf && \
    sed -i '/^;php_flag\[display_errors\]/cphp_flag[display_errors] = off' /etc/php5/fpm/pool.d/www.conf && \
    sed -i '/^;php_admin_value\[error_log\]/cphp_admin_value[error_log] = /var/log/fpm-php.www.log' /etc/php5/fpm/pool.d/www.conf && \
    sed -i '/^;php_admin_flag\[log_errors\]/cphp_admin_flag[log_errors] = on' /etc/php5/fpm/pool.d/www.conf

This works fine - I can get a shell into the container to see the logs. But... it is not best-practice.
The problem is when I try to use the docker log collector - I need php-fpm to log to stdout or stderr so docker can capture them and provide them to the docker logs command.
I tried to do this in the Dockerfile (which is a idea I copied from the official nginx docker image):
# Redirect fpm logs to stdout and stderr so they are forwarded to the docker log collector
RUN ln -sf /dev/stdout /var/log/fpm-access.log && \
    ln -sf /dev/stderr /var/log/fpm-php.www.log

This is not working - no access logs are seen from docker logs - I'm trying to figure out why? Did anyone else that uses fpm in docker manage to get logging working to the docker log collector?
Answer: Ok, the way to do this is to send the error and the access logs to the following address:
/proc/self/fd/2

In php5-fpm.conf (or the appropriate configuration file) add:
access.log = /proc/self/fd/2
error_log = /proc/self/fd/2

NOTE: access.log is correct, find in this page https://www.php.net/manual/en/install.fpm.configuration.php
Comment: It appears that it is `access.log`: https://github.com/docker-library/php/blob/6e600f59a4405b5066eb78f5f02180212dffd065/5.6/fpm/php-fpm.conf
Comment: Believe this should be `php5-fpm.conf`, or on a more current PHP, `php-fpm.conf`. Adding those directives to `.log` file would do nothing :D
Comment: There's an error in the answer - it's "access_log" not "access.log"
Comment: It's possible for there to be `/dev/stdin ~ /dev/fd/0 ~ /proc/self/fd/0`, and `/dev/stdout` and `/dev/stderr` variants. Might be easier to remember to use `/dev/stdin`.
Answer: Note that the baked in fpm config for the latest version of the official PHP fpm docker image writes to the standard streams:
error_log = /proc/self/fd/2

...

; if we send this to /proc/self/fd/1, it never appears
access.log = /proc/self/fd/2

Comment: Which docker image are you referring to?  I ran php:7-fpm and it doesn't appear to be logging errors to `stderr`.
Comment: Thanks, that's interesting. I didn't know that there's a official image now
Comment: If you run `php-fpm` inside a container it should report what `error_log` value is. In my case it's `/proc/self/fd/2` as needed. Docker image `php:7.2-fpm`.
Comment: @laimison "Report" `error_log` where?
Answer: PHP-FPM logs will only appear in STDERR - so you can symlink the fpm.log to /dev/stderr if you want.
ln -sf /dev/stderr /var/log/fpm-access.log
ln -sf /dev/stderr /var/log/fpm-error.log

Comment: This solution was given in the question and the asker stated it didn't work. Maybe you can specify how he can load it in his dockerfile to make it work properly or other diagnostics he can perform on his container?
Comment: @AndrewDomaszek file names are different


Rest on wildfly return "Not Found", but jsp works, What's wrong?

Question: I'm start study Rest and I try to do REST to return a JSON of Aluno, when I will test on "postman" or "firefox" I receive this 404 error, but ther isn't no problem to access jsp.
I try many thinks, I changed versions of rest-easy, hibernate, try do another Rest, but only the same answer.
Technology used: Wildfly, RestEasy, EasyCriteria, JPA, Hibernate.
URL: http://localhost:8080/teachmanager/ws/aluno
What's wrong?!
answer:

        <title>Error</title>

    Not Found

Rest:
@Path("aluno")
public class AlunoRest {

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public String busca() {
        Dao<Aluno> dao = new Dao<Aluno>(JPAUtil.getEntityManager(), Aluno.class);
        Aluno aluno = dao.getById(1);
        return aluno.toJson();
    }

}

pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>br.com.ilearning.teachmanager</groupId>
  <artifactId>teachmanager</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>war</packaging>

  <dependencies>

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>6.0.3</version>
    </dependency>

    <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-core</artifactId>
        <version>5.2.10.Final</version>
    </dependency>

    <dependency>
        <groupId>org.hibernate.javax.persistence</groupId>
        <artifactId>hibernate-jpa-2.1-api</artifactId>
        <version>1.0.0.Final</version>
    </dependency>

    <dependency>
        <groupId>org.hibernate.validator</groupId>
        <artifactId>hibernate-validator</artifactId>
        <version>6.0.7.Final</version>
    </dependency>

    <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-ehcache</artifactId>
        <version>5.2.13.Final</version>
    </dependency>

    <dependency>
       <groupId>dom4j</groupId>
       <artifactId>dom4j</artifactId>
       <version>1.6.1</version>
       <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>uaihebert.com</groupId>
        <artifactId>EasyCriteria</artifactId>
        <version>3.0.0</version>
    </dependency>

    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
        <version>2.8.2</version>
    </dependency>

    <dependency>
        <groupId>org.jboss.resteasy</groupId>
        <artifactId>resteasy-jaxrs</artifactId>
        <version>2.2.1.GA</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.jboss.resteasy</groupId>
        <artifactId>resteasy-multipart-provider</artifactId>
        <version>2.2.0.GA</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.jboss.resteasy</groupId>
        <artifactId>resteasy-servlet-initializer</artifactId>
        <version>3.5.0.Final</version>
    </dependency>

  </dependencies>

  <build>
    <sourceDirectory>src</sourceDirectory>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.7.0</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-war-plugin</artifactId>
        <version>3.0.0</version>
        <configuration>
          <warSourceDirectory>WebContent</warSourceDirectory>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

web.xml:
    <web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.jcp.org/xml/ns/javaee" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" id="WebApp_ID" version="3.1">
    <context-param>
        <param-name>resteasy.scan</param-name>
        <param-value>true</param-value>
    </context-param>

    <context-param>
        <param-name>resteasy.servlet.mapping.prefix</param-name>
        <param-value>/ws</param-value>
    </context-param>

    <listener>
        <listener-class>
            org.jboss.resteasy.plugins.server.servlet.ResteasyBootstrap
        </listener-class>
    </listener>

    <servlet>
        <servlet-name>restEasy</servlet-name>
        <servlet-class>org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>restEasy</servlet-name>
        <url-pattern>/ws/*</url-pattern>
    </servlet-mapping>

    <welcome-file-list>
        <welcome-file>index.jsp</welcome-file>
    </welcome-file-list>

</web-app>

Comment: I change web module version for 3.0 and thats works!

But i don't understood why don't work on 3.1!
Comment: *seems* like you don't have your rest framework knowing to expose that AlunoRest class as a resource. Might be a rest-easy question.
Answer: I change web module version for 3.0 and thats works! But i don't understood why don't work on 3.1!
And I discover a new way to do that webservice, without use web.xml.
I create a configuration class:
import javax.ws.rs.ApplicationPath;
import javax.ws.rs.core.Application;

@ApplicationPath("service")
public class RestManager extends Application{

}

and create a normal rest class:
@Path("teste")
public class Teste {

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public String getTeste() {
        return "teste";
    }

    @GET
    @Path("lista")
    @Produces(MediaType.APPLICATION_JSON)
    public String lista() {
        String ret = "";
        try {
            Dao<Aluno> dao = new Dao<Aluno>(JPAUtil.getEntityManager(), Aluno.class);
            List<Aluno> lista = dao.getAll();
            ret = new Gson().toJson(lista);
        } catch (Exception e) {
            ret = new Gson().toJson(e.getMessage());
        }

        return ret;
    }

}

Rest on wildfly return “Not Found”, but jsp works, What's wrong?


Is this a correlation request?

Question: Trying to decode this task:
I have a table of user ids and the up to 12 products they've used over a given period. Two columns, each line being a unique ID and product combo.  Boss wants to know how the use of each product relates to each other product, as in: do users of product a also use product c, d but avoid g, h?
That makes me think I need to check correlation, like with a seaborn pairplot, for example.
Is this the/a correct way to proceed? And if so, how does one measure correlation within one field? Would this require an intermediate step of aggregation, since the data I'm analysing isn't table values, but the aggregate of them?
Hopefully this makes sense, tho I'm happy to clarify where needed.
Any thoughts or pointers would be much appreciated.
M
Comment: To clarify, you have 13 columns of data, one with ID and the rest with ether Y or N (or 1 or 0) for the different products?
Comment: Two columns: user_id, product name. There are unique 12 products and about 12k users. I could expand it out to 13 columns if the analysis method required tho.
Comment: @MatthewJJJ does each (user, product) tuple always appear 0 or 1 times, or is "how many uses?" a valid question?
Comment: @hobbs - each user can be paired with up to all 12 products, tho most of them have used 1-3.
Answer: First, I would expand it to 13 columns.
Then, while you could do some measure of association for each combination of products, that would give you $12\times 11/2 = 66$ measures. Hard to interpret, prone to type 1 error.
I would look into multivariate approaches such as cluster analysis. This thread is a good place to start.
Comment: Thanks for that Peter - I'll give that a thorough look. Is cluster analysis not more suited for data sets where the values are more variable, though? For example where for the 13 columns, the 12 numerical ones had values other than just 0 or 1?
Comment: There are approaches to clustering that work well with dichotmous variables, as in the thread I listed.
Answer: Instead, consider employing association rule mining techniques, such as Apriori or FP-growth algorithms. These methods, commonly used in market basket analysis, can reveal meaningful associations between products. Convert your data into a binary format (used or not used) and use metrics like support, confidence, and lift to interpret the strength of associations.
Comment: One option for an existing implementation of frequent pattern mining is [`mlxtend`](https://rasbt.github.io/mlxtend/api_subpackages/mlxtend.frequent_patterns/).
Answer: Mostly out of curiosity, I would explore a random network approach.
Suppose a random graph $G$ with a fixed vertex set $v(G) = \{ \text{products} \}$ representing your set of products and a random edge set $E(G) \subseteq v(G) \times v(G)$. Your random edge set can be represented as a random adjacency matrix $A$ where $A_{ij}$ representing a Bernoulli variable for whether the $i$th product was paired with the $j$th product for some unique ID. Since this would be an undirected graph, you would really only need to have distinct parameters for the upper/lower triangle of this adjacency matrix (and exclude the diagonal since products always co-occur with themselves).
Here is a just a toy specification to get you started, but I mostly will defer development advice to Gelman et el. 2020.
$$\alpha_{ij} \sim \text{Exponential}(1)$$
$$\beta_{ij} \sim \text{Exponential}(1)$$
$$p_{ij} \sim \text{Beta}(\alpha_{ij}, \beta_{ij})$$
$$A_{ij} \sim \text{Bernoulli}(p_{ij})$$
You'll want to expand this first approximation to account for other aspects of the problem. For example:

You may want to use a multilevel/mixed effect/hierarchical modelling approach for repeated customers or similarly accounting for a taxonomy on the types of products.
There may be further covariance to model between the parameters due to cliques. McElreath 2023 provides some useful guidance on a related problem.
There could also be some non-stationarity over time that turns this into a time series problem (if you can get dates) because customers sometimes change their habits.

After sampling from the posterior distribution, you can interpret you $p_{ij}$ as a tendency for the $i$th product to co-occur with the $j$th product. For something analogous to correlation, you could compute the posterior distribution of normalized independence gaps using the sampled values of $p_{ij}$ and marginalization (to obtain the product of the marginals $p_i p_j$).

Is this a correlation request?


Analytic Function vs Exponential Order function

Question: We say that a function $f$ is of exponential order $\alpha$ if there exist constants: $M$, $\alpha$, $T$ such that for $x>T$
$$f(x)<M\cdot e^{\alpha x}$$
Polynomials are of exponential order. Then, is it true that analytic functions in a neighborhood of the origin are of exponential order?
I want to know this to see if it is legal to calculate the Laplace transform of analytic functions.
Answer: In general no, take for example $e^{x^2}$.
Comment: How do we know this function is analytic?
Comment: It is the composition of analytic functions. (Or you can write down an explicit convergent Maclaurin series for it. Or you can use a complex analysis argument.)


What is this sound effect called?

Question: In this video:

...from 1:13 to 1:20 there is a special sound connection between the clarinet and the accordion. I don't know how to describe it but sounds like a very nice transition (specially if you listening from the very beginning of the song).
My Questions
What is that 'in tune' sound effect called?
Is there any branch of music dedicated to the study of the dynamics of song through it? I mean, how do they know that at that particular time, that effect would sound nice(for me sounds like all the stuff they play before that moment, sound enhance that "in tune" effect).
Comment: The clarinet, accordion and voice (of accordion player) all play/sing a B. Not sure exactly what effect you mean.
Answer: For me it sounds like the notes they are playing are not exactly in tune. They both play a B but the clarinet is slightly sharp.
This results in a so called beating: https://en.wikipedia.org/wiki/Beat_(acoustics)
Comment: Eek, yup, the accordion is flat as a *&^% right there. I'm not sure it's even close enough to beat, it just sounds 'out of tune'.
Comment: @Tetsujin I hate it when accordion players don't tune before a performance!
Answer: Tuning issues aside, the effect you speak of is simply the clarinet staring the B note by himself on the downbeat and then the accordion playing an Amaj9 quietly on beat 4 with the same B on top and doing a crescendo while the clarinet holds his note. The effect is that the sounds of the two instruments merge into one. It doesn't have a specific name,

What is this sound effect called?


drush puts my website in a subdirectory named 'web'

Question: I use shared web hosting with Cpanel, LAMP and Composer.
I have installed Drupal with Drush on a registered web domain (example.com).
My website is accessible via example.com/web/ but I would expect the website to be accessible via example.com only. Why is that the case and how to handle that situation?
Answer: How to fix this
You need to set the web root for the domain (to web).  You need to do this on your web server, not in Drupal.  If you're on a shared host, something like CPanel will have a setting for the web root.
Why does Drush install Drupal in the web subdirectory?
Drupal has a lot of support files, many of which should never be exposed to the public by your webserver.  So, the files that should be exposed get put in the web directory, while the other utility files get put in other directories.


Question about Neural Network Training

Question: What is the benefit of optimizing our neural network error function using back prop rather than just using gradient or stochastic gradient descent directly on the error function?
How come we don't just take the partial derivative of the error function with respect to every weight and do gradient or stochastic gradient descent based on that?
Instead of doing back propagation to come up with these artificial error values for hidden nodes and then for each hidden node multiplying its error times the various partial derivatives of weights associated to its incoming edges, as if that hidden node was actually an output node.
Answer: Backpropagation is basically using the chain rule to efficiently calculate the gradient of the error function with respect to the weights. You need to calculate those gradients for either stochastic gradient descent, conjugate gradients, or any other gradient based optimization method.
It is usually identified with gradient descent, but you can see it independently of it. It allows you to calculate gradients of the error function with respect to the weights with a complexity that scales linearly with the number of weights.
Comment: So what you're saying is that back prop is just a really clever way of doing gradient descent which takes advantage of the fact that the network function has a nice form (it's essentially the composition of the same type of function over and over again) in order to maintain efficiency for a large number of hidden nodes/layers?
Comment: And so if one wanted to they *could* train the network using regular gradient descent rather than back prop and it would optimize in the same exact way, except it would be much less efficient?
Comment: Backpropagation is a clever way to calculate the gradient of the error function with respect to the weights. You can then use conjugate gradients, BFGS, or any other optimization method you prefer.

Question about Neural Network Training


How to click OK in confirm popup opened from a frame (Watin)?

Question: I can't find a way to click confirm popup when it is inside iframe. I could find the iframe and click "Clicked OK" button, then confirm button pops up, at this point I am stuck. I posted my html and c# code below.
Here is file:///C/Temp/IFrame.html

Your browser does not support iframes.

here is the file:///C:/Temp/confirm1.html

    <input id="myButton1" type="button" value="this is a button"
      onclick="confirmMe(); return false;">

Here is what it looks:

I have this code that works without frame but when frame is used, I can't find a way to click the confirm dialog poped up from iframe.
public static void ConfirmPopupFromFrame()
{
    IE ie = new IE();
    ie.GoTo("file:///C:/Temp/IFrame.html");
    ie.WaitForComplete();

    Frame iframe;
    try{
        iframe = ie.Frame("FrameID");
    }
    catch (FrameNotFoundException ex){
        throw ex;
    }

    ConfirmDialogHandler confirm = new ConfirmDialogHandler();
    using (new UseDialogOnce(ie.DialogWatcher, confirm))
    {
        iframe.Button("myButton1").Click(); // At this point confirm dialog pops up but below lines doesn't work since this is frame instance not ie. is there a way to make it work?
        confirm.WaitUntilExists();
        confirm.OKButton.Click();
    }
}

Answer: Change
iframe.Button("btnId").Click();

to
iframe.Button("myButton1").ClickNoWait();

The button id is probably just a typo in your test code, what's important is change Clinck to ClickNoWait. I've just tested - it works for me
Comment: yes ID was typo. but your answer nailed it Thanks :)