Explain this awk command

Question: Please explain what exactly this awk command does:
awk '$0!~/^$/{print $0}'

Answer: It removes blank lines.  The condition is $0 (the whole line) does not match !~ the regexp /^$/ (the beginning of the line immediately followed by the end of the line).
Similar to grep -v '^$'
Answer: It prints non-empty input lines. Note: "Empty" does not mean "blank", in this case.
Answer: Your example could be rewritten as simply:
awk '!/^$/'

or
sed '/^$/d'

Answer: Like Ben Jackson and the others said, it removes completely empty lines. Not the ones with one ore more whitespaces, but the zero character long ones. We will never know if this was the intended behaviour.
I'd like to remark, that the code is at least redundant if not even triple redundant depending on what it's used for.
What it does is that it prints the input line to the output if the input line is not the empty line.
Since the standard behaviour of awk is, that the input line is printed if a condition without a following program block is met, this would suffice:
awk '$0!~/^$/' or even shorter awk '$0!=""'

If you could be sure, that no line would be parsed to zero, even a
awk'$0'

could do the trick.
Answer: Make it readable first...

echo '$0!~/^$/{print $0}' | a2p
==>
$, = ' ';
$\ = "\n";

while (<>) {
    chomp;
    if ($_ !~ /^$/) {
        print $_;
    }
}

And the interpret. In this case, don't print empty lines.
Comment: -1. I like Perl, but "readable" is in the eye of the beholder.  Is `$_` better than `$0`? I think this answer is not very helpful.

Explain this awk command


What is the equivalent for win32com.client.GetObject() in CPP?

Question: What is the equivalent for win32com.client.GetObject() in CPP ?
I am trying to get the SAPGUI object in CPP
SapGuiAuto = win32com.client.GetObject("SAPGUI")

Referring to these quotes from the below article - https://blogs.sap.com/2020/08/02/sap-gui-scripting-api-from-the-past-for-future-automation/

The SAP GUI Scripting API is in the file sapfewse.ocx (FEWSE = Front End Windows Scripting Engine), in the directory C:\Program Files (x86)\SAP\FrontEnd\SapGui. It is a Component Object Model (COM) library and registered in the Windows registry. On this way you can use it from any COM-enabled programming language
Answer: There is no "C++" way of doing it. There are, however, ways of doing it on Windows using Microsoft libraries.
I'd recommend using CComDispatchDriver class. Haven't tested this...but it will get you close.
// .. include standard windows stuff
#include <comdef.h>
#include <atlbase.h>

class ComInit
{
public:
    ComInit()
    {
        CoInitialize(NULL);
    }

    ~ComInit()
    {
        CoUnintialize();
    }

    ComInit(const ComInit& ) = delete;
    ComInit& operator=(const ComInit&) = delete;
};

int main(int argc, char* argv)
{
   ComInit _cominit;

   CComDispatchDriver disp;
   disp.CoCreateInstance(L"Excel.Application");

   _variant_t vName;
   disp.GetPropertyByName(L"Name", &vName);

   return 0;
}


Riemann's $\zeta$ function and the uniform distribution on $[-1,0]$

Question: It seems that the $n$th cumulant of the uniform distribution on the interval $[-1,0]$ is $B_n/n$, where $B_n$ is the $n$th Bernoulli number.
And also $-\zeta(1-n) = B_n/n$, where $\zeta$ is Riemann's $\zeta$ function.
Is there some reason why one should expect these to be the same, as opposed to proofs that convince you that they are?
Comment: Since we're looking at the analytic continuation of $\zeta$, the usual $p$-series concept is out the window - except through formal regularization.
Comment: Also posted to MathOverflow.
Answer: The moment generating function of $U([-1,0])$ is $\mathbb{E}(\mathrm{e}^{t u}) = \int_0^1 \mathrm{e}^{-t v} \mathrm{d} v = \frac{1}{t} \left(1 - \mathrm{e}^{-t}\right) $.
The cumulant generating function is the logarithm of the moment generating function: $\mathcal{c}(t) = \log \left(\frac{1}{t} \left(1 - \mathrm{e}^{-t}\right) \right)$.
Now, find $t \frac{\mathrm{d}}{\mathrm{d} t} \left( \mathcal{c}(t) \right)$:
$$
  t \mathcal{c}^\prime(t) = -1 + \frac{t}{\mathrm{e}^t-1} = \sum_{n \ge 1} \frac{t^n}{n!} B_n
$$
And therefore, integrating term-wise we get:
$$
\mathcal{c}(t) = \sum_{n \ge 1} \frac{t^n}{n!} \frac{B_n}{n}
$$
Of course, this is more of the derivation side...
Comment: This answer doesn't mention the $\zeta$ function.  Is there some way to show the values of that function at negative integers are $-1$ times the cumulants, without deriving the two separately and observing the answers are the same?
Comment: Hi @MichaelHardy: Have you found an appropriate answer for your question? If not, please have a look here: http://arxiv-web3.library.cornell.edu/pdf/1311.4999.pdf
Btw, I have a related question:
http://math.stackexchange.com/questions/570929/references-mentioning-the-relationship-between-cumulants-of-uniform-distribution
I guess you know this, don't you?


Adding CSS styles to a block

Question: UPDATED Added other files just in case the error is somewhere else
I just created a custom module for Drupal 8 that defines a block. I want to add some CSS styles to the content of the block.
My last attempt to achieve this was adding a libraries.yml file to the module linking the block_header.css file. In the render array, I added #prefix and #suffix with the CSS classes (
).
The code doesn't give me any error, but it's not applying the font weight I would expect.
Could you point me to the right direction?
These are the files.
block_header.info.yml
name: Block Header creator
type: module
description: 'Create block with a main header and a text on mouseover'
core: 8.x
package: Fussion Corp - Custom Modules
configure: block_header.settings

block_header.routing.yml
block.content:
  path: '/block'
  defaults:
    _controller: '\Drupal\block_header\Controller\BlockController::content'
    _title: 'Block Header'
  requirements:
    _permission: 'access content'

block_header.libraries.yml
block_header:
version: 1.x
css:
    theme:
        css/block_header.css: {}

BlockController.php
<?php
/**
 * @file
 * Contains \Drupal\block_header\Controller\BlockController.
 */
namespace Drupal\block_header\Controller;
class BlockController {
  public function content() {
    return array(
      '#type' => 'markup',
      '#markup' => t('Sample Text!'),
    );
  }
}

block_header.module
<?php

/**
 * Implements hook_preprocess_HOOK()
 */
function block_header_preprocess_block(&$variables) {
  if ($variables['plugin_id'] == 'block_header') {
    $variables['page']['#attached']['library'][] = 'block_header/block_header';
  }
}

BlockHeader.php
<?php

namespace Drupal\block_header\Plugin\Block;

use Drupal\Core\Block\BlockBase;
use Drupal\Core\Block\BlockPluginInterface;
use Drupal\Core\Form\FormStateInterface;

/**
 * Provides a 'Header' Block.
 *
 * @Block(
 *   id = "block_header",
 *   admin_label = @Translation("Block Header"),
 *   category = @Translation("Block Header"),
 * )
 */

class BlockHeader extends BlockBase implements BlockPluginInterface {

  /**
   * {@inheritdoc}
   */
  public function build() {
    $config = $this->getConfiguration();

    if (!empty($config['block_header_title']) && ($config['block_header_text'])) {
      $title = $config['block_header_title'];
      $descp = $config['block_header_text'];
    }
    else {
      $title = $this->t('
Atención! Titulo no configurado!

');
      $descp = $this->t('
Atención! Descripción no configurada!
');
    }
    $block = array
        (
            'title' => array
            (
             '#prefix' => '

', /* HERE I ADD THE CSS TAGS */
             '#suffix' => '

',
             '#markup' => t('@title', array('@title' => $title,)),
            ),
            'description' => array
            (
             '#prefix' => '

', /* HERE I ADD THE CSS TAGS */
             '#suffix' => '

',
             '#markup' => t('@descp', array('@descp' => $descp,))
            ),
        );
    return $block;

  }

   /**
   * {@inheritdoc}
   */
  public function blockForm($form, FormStateInterface $form_state) {
    $form = parent::blockForm($form, $form_state);

    $config = $this->getConfiguration();

    $form['block_header_title'] = array(
      '#type' => 'textfield',
      '#title' => $this->t('Titulo del Bloque'),
      '#description' => $this->t('Titulo del Bloque'),
      '#default_value' => isset($config['block_header_title']) ? $config['block_header_title'] : '',
    );

    $form['block_header_text'] = array(
      '#type' => 'textarea',
      '#title' => $this->t('Descripción'),
      '#description' => $this->t('Descripción del bloque'),
      '#default_value' => isset($config['block_header_text']) ? $config['block_header_text'] : '',
    );

    return $form;
  }

  /**
   * {@inheritdoc}
   */
  public function blockSubmit($form, FormStateInterface $form_state) {
    parent::blockSubmit($form, $form_state);
    $values = $form_state->getValues();
    $this->configuration['block_header_title'] = $values['block_header_title'];
    $this->configuration['block_header_text'] = $values['block_header_text'];
    $this->configuration['block_header_title'] = $form_state->getValue('block_header_title');
    $this->configuration['block_header_text'] = $form_state->getValue('block_header_text');
  }
}

This is the structure of the directory containing the module.

What am I doing wrong?
Answer: Hooks go inside the .module file.
block_header.module
<?php

/**
 * Implements hook_preprocess_HOOK()
 */
function block_header_preprocess_block(&$variables) {
  if ($variables['plugin_id'] == 'block_header') {
    $variables['#attached']['library'][] = 'block_header/block_header';
  }
}

Just FYI, this did not work:

    $variables['elements']['#attached']['library'][] = 'block_header/block_header';

block_header.libraries.yml
block_header:
  version: 1.x
  css:
    theme:
      css/block_header.css: {}

Comment: Thank you very much!!! this *.module file worked!!! Thank you for your time!
Comment: @Hawk I think Clive's answer is better for your own custom block case.
Comment: I disagree with your prior comment, may I? And if yes: why not?
Comment: `I disagree with your prior comment, may I?` yes, you may.
Comment: I agree with No Sssweat, Clive's answer is more appropriate.  preprocess_block is heavy handed for just attaching a stylesheet to a particular block.  It's going to run for every block in the module.
Answer: The attachment should be on the block itself:
$block = [
        '#attached'=> ['library' => ['block_header/block_header']],
        'title' => array
        (
         '#prefix' => '

', /* HERE I ADD THE CSS TAGS */
         '#suffix' => '

',
         '#markup' => t('@title', array('@title' => $title,)),
        ),
        'description' => array
        (
         '#prefix' => '

', /* HERE I ADD THE CSS TAGS */
         '#suffix' => '

',
         '#markup' => t('@descp', array('@descp' => $descp,))
        ),
    ];

And remember YAML is sensitive to white space, make sure your libraries file is formatted correctly:
block_header:
  version: 1.x
  css:
    theme:
      css/block_header.css: {}

Adding CSS styles to a block


How do I enable and use the password module for Drupal 6?

Question: I've uploaded the module to my site, but it doesn't appear in the list of modules so I can't enable it. Does this module work differently than other modules?
https://drupal.org/project/password
Answer: No, this module works as any other module. See my screenshot as I can see it in my module listing -
Are you sure you have it in sites/all/modules/ ? Are the permissions correctly set?
Comment: Ahh... I see it now.  Thanks.  Silly me, I was expecting it to actually be named Password.
Comment: Yes, it's in the correct location and I'm logged in as the site administrator so permissions shouldn't be an issue.  I've uploaded other modules to the same folder and can see them in the admin list.  Maybe a conflict with other modules?
Comment: No, drupal.org should not allow name conflicts
Comment: Also, I can't see your screenshot - dropbox is forbidden by our network here.
Comment: I updated the screenshot


Qual é a diferença entre criar um objeto a partir da forma literal ou a partir de uma função construtora?

Question: Eu gostaria de saber se possui alguma diferença ou relevância entre as duas formas abaixo na construção de um objeto:
Criar um objeto a partir da forma literal:

let pessoa = {
   nome: 'Pedro'
};

console.log(pessoa.nome);

Criar um objeto a partir de uma função construtora:

let pessoa = new Person();

function Person() {
    this.nome = 'Pedro';
}

console.log(pessoa.nome);
Answer: Em resumo, a diferença está na forma como os objetos são criados e, portanto, o construtor dos objetos.

No seu primeiro exemplo:

let pessoa = {
   nome: 'Pedro'
};

console.log(pessoa.nome);

Você simplesmente está criando um novo objeto através de sua notação literal. Você está atribuindo uma propriedade name diretamente através dessa notação. Não há nada de muito especial além disso.
No seu segundo exemplo:

function Person() {
    this.nome = 'Pedro';
}

let pessoa = new Person();
console.log(pessoa.nome);

Você está instanciando um novo objeto do construtor Person, ou seja, você está criando um novo objeto do "tipo" Person.
Há, então, a diferença, já que quando você usa o operador new, o objeto retornado (instanciado) irá herdar o prototype de sua função construtora. No seu exemplo, pode-se notar a diferença com o seguinte exemplo:

const person1 = {
  name: 'Luiz'
};

function Person() {
  this.name = 'Luiz';
}

const person2 = new Person();

console.log(person1.constructor.name); // "Object"
console.log(person2.constructor.name); // "Person"

Assim, pode-se concluir que, basicamente, a diferença é esta:

No seu primeiro exemplo, como o objeto está sendo construído usando a sua notação literal, ele irá herdar o protótipo do construtor Object (padrão para objetos literais).
No segundo exemplo, contudo, utiliza-se o construtor Person, uma função criada pelo próprio programador. Aí está a diferença.

Isso tudo está muito ligado ao funcionamento de protótipos no JavaScript. Para saber mais, leia esta outra pergunta.

A título de curiosidade, se você quiser criar um objeto que não possua nenhum parente herdado, você pode utilizar Object.create(null), sendo o primeiro argumento o protótipo que o objeto a ser criado herdará.

const example1 = {};
const example2 = Object.create(null);

console.log(example1.constructor.name); // "Object"
console.log(example2.constructor); // undefined
Comment: @felipecardozo Sobre protótipo, um bom ponto de partida é aqui: https://pt.stackoverflow.com/q/15239/112052
Comment: Agora fiquei mais confuso ainda, estou aprendendo sobre OOJS e não entendo muito sobre protótipo, construtor, herdar, polimorfismo, etc eu vou dar mais uma estudada sobre esses conceitos, assim quando eu voltar a rever essa resposta estará mais clara para mim, valeu! :)
Comment: @hkotsubo, obrigado! por compartilhar o link da resposta vou dar uma lida.


Will the fastest time in dry weather be deleted when the weather changes to rainy?

Question: Imagine that in Q3, Valteri Bottas is the first car to release to the track and takes his first lap in dry weather while the other 9 drivers still not haven't got into the track yet. Then, after Bottas finishes his first lap, the weather changes from dry to wet and of course, the remaining drivers that still don't get a chance to get into the track need to change from slick to wet/intermediate tyres.
My question is, will Bottas' time during dry weather get deleted due to changed weather? I mean, Bottas' lap during dry weather could be outqualified by e.g. Verstappen during wet weather even though that is disadvantageous.
Answer: No. Luck of the weather is just one of the hazards.
Teams typically try to plan on weather forecasts so they can get the best time possible, but sometimes they just don't get it quite right. And they have to deal with that - there is no penalisation of a driver/team for managing to be lucky with the weather.


How to use different convolution layers in different branches of tf.map_fn?

Question: I tried to establish a simple multi-head attention layer in tensorflow1.14. Each head contains three different conv1d layers. And I want to use tf.map_fn to compute parallel.
import tensorflow as tf

n_head = 50 # heads counts

conv1d = tf.layers.conv1d
normalize = tf.contrib.layers.instance_norm
activation = tf.nn.elu

f1d = tf.placeholder(shape=(None, 42),dtype=tf.float32) # input feats

f1ds = tf.tile(f1d[None, ...], [n_head, 1, 1]) # n_head copys to apply different attention heads

def apply_attention(f1):
    f1 = activation(normalize(f1[None, ...]))
    q = conv1d(f1, 32, 3, padding='same')
    k = conv1d(f1, 32, 3, padding='same')  # [1,ncol, 32]
    v = conv1d(f1, 1, 3, padding='same')  # [1,ncol, 1]
    attention_map = tf.nn.softmax(tf.reduce_sum(q[0, None, :, :] * k[0, :, None, :], axis=-1) / (32 ** .5),
                                  axis=0)  # [ncol,ncol]
    return attention_map * v[0]

f1d_attention = tf.map_fn(lambda x: apply_attention(x), f1ds, dtype=tf.float32)

But when I detect the variables in this model, it seems like there are only one group of conv1d layers in the whole model.
conv1d/bias/Adam [32]
conv1d/bias/Adam_1 [32]
conv1d/kernel [3, 42, 32]
conv1d/kernel/Adam [3, 42, 32]
conv1d/kernel/Adam_1 [3, 42, 32]
conv1d_1/bias [32]
conv1d_1/bias/Adam [32]
conv1d_1/bias/Adam_1 [32]
conv1d_1/kernel [3, 42, 32]
conv1d_1/kernel/Adam [3, 42, 32]
conv1d_1/kernel/Adam_1 [3, 42, 32]
conv1d_2/bias [1]
conv1d_2/bias/Adam [1]
conv1d_2/bias/Adam_1 [1]
conv1d_2/kernel [3, 42, 1]
conv1d_2/kernel/Adam [3, 42, 1]
conv1d_2/kernel/Adam_1 [3, 42, 1]

What's wrong with my code?
Answer: In that case, you don't ant to use tf.map_fn. tf.map_fn will evaluate your function once, and run your different inputs through the same function, effectively using the same convolution layers for each input.
You can achieve what you want with a simple for loop :
# Creating a different set of conv for each head
multi_head = [apply_attention(f1d) for _ in range(n_head)]
# stacking the result together on the first axis
f1d_attention = tf.stack(multi_head, axis=0)

I've reduce the number of heads to 2 for visibility, but if we look at the variables, we see that 2 groups of convolution has been instantiated.
>>> tf.global_variables()
[<tf.Variable 'InstanceNorm/beta:0' shape=(42,) dtype=float32_ref>,
 <tf.Variable 'InstanceNorm/gamma:0' shape=(42,) dtype=float32_ref>,
 <tf.Variable 'conv1d/kernel:0' shape=(3, 42, 32) dtype=float32_ref>,
 <tf.Variable 'conv1d/bias:0' shape=(32,) dtype=float32_ref>,
 <tf.Variable 'conv1d_1/kernel:0' shape=(3, 42, 32) dtype=float32_ref>,
 <tf.Variable 'conv1d_1/bias:0' shape=(32,) dtype=float32_ref>,
 <tf.Variable 'conv1d_2/kernel:0' shape=(3, 42, 1) dtype=float32_ref>,
 <tf.Variable 'conv1d_2/bias:0' shape=(1,) dtype=float32_ref>,
 <tf.Variable 'InstanceNorm_1/beta:0' shape=(42,) dtype=float32_ref>,
 <tf.Variable 'InstanceNorm_1/gamma:0' shape=(42,) dtype=float32_ref>,
 <tf.Variable 'conv1d_3/kernel:0' shape=(3, 42, 32) dtype=float32_ref>,
 <tf.Variable 'conv1d_3/bias:0' shape=(32,) dtype=float32_ref>,
 <tf.Variable 'conv1d_4/kernel:0' shape=(3, 42, 32) dtype=float32_ref>,
 <tf.Variable 'conv1d_4/bias:0' shape=(32,) dtype=float32_ref>,
 <tf.Variable 'conv1d_5/kernel:0' shape=(3, 42, 1) dtype=float32_ref>,
 <tf.Variable 'conv1d_5/bias:0' shape=(1,) dtype=float32_ref>]

Side Note: Unless you have a really good reason, you should migrate away from TensorFlow 1 and use TensorFlow 2 instead. Support for TF1 is limited.


Lilypond: multiple glyphs in front of a note

Question: I can use a markup to put several glyphs under and over a note with _\markup ^markup but I do not know how to place glyphs in front of a note. Something similar to Natural+flat accidental in front of the note but with arbitrary glyphs.
It would be even better (but not necessary) if the placement were identical to that of accidentals also in case of chords.
Answer: An easy way to get this would be to abuse fingerings:
<code>{
  \set fingeringOrientations = #'(left)

  <c''-\finger\markup \override #'(font-encoding . "latin1") "my mup">
}
</code>
If you do want the collision resolution of accidental columns and you do not need both accidental and custom glyph for any note you can abuse Accidentals:
<code>{
  \override Accidental.stencil = #ly:text-interface::print
  \override Accidental.font-size = #-3
  <\tweak Accidental.text \markup\vcenter "mup1" c'!
   \tweak Accidental.text \markup\vcenter "mup2" d'!
   \tweak Accidental.text \markup\vcenter "mup3" f'!
   \tweak Accidental.text \markup\vcenter "mup4" b'!
   \tweak Accidental.text \markup\vcenter "mup5" c''!>
}
</code>
Else one would need some extra effort for creating some  custom engraver.


Gelfand-Naimark theorem and compactifications

Question: Let $X$ be a Hausdorff space and let $C(X)$ denote the set of all continuous and bounded functions from $X$ to $\mathbb{C}$. It is a well-known fact that $C(X)$ forms a unital and abelian $C^*$-algebra (where the addition, multiplication, and scalar multiplication are defined pointwise, $C(X)$ is endowed with the sup norm, and the $*$ operation is conjugation)
Gelfand-Naimark theorem says the following: any unital and abelian $C^*$-algebra is isomorphic to $C(\mathbf{X}),$ where $\mathbf{X}$ is some compact and Hausdorff space, and $C(X)$ denotes the set of all continuous and bounded functions from $\mathbf{X}$ to $\mathbb{C}.$
Consequently, for any Hausdorff space $X,$ there exists a compact and Hausdorff space $\mathbf{X}$ such that $C(X)$ is isomorphic to $C(\mathbf{X}).$
Question: Is $\mathbf{X}$ a compactification of $X$?
By a compactification I mean the following: $\mathbf{X}$ is a compactification of $X$ if and only if $X$ embeds as an open and dense subspace in $\mathbf{X}$ (I know that the requirement that $X$ is open in $\mathbf{X}$ is usually not a part of the definition of a compactification, but I would like to include it)
Bonus question 1: What conditions do $C(X)$ or $X$ need to satisfy to make sure that $\mathbf{X}$ is a compactification of $X?$
Bonus question 2: If $C_b(X)$ denote the set of all bounded (and not necessarily continuous) functions from $X$ to $\mathbb{C},$ then $C_b(X)$ is a unital and abelian $C^*$ algebra. Is $\mathbf{X}$ from the Gelfand-Naimark theorem a compactification of $X?$ Why or why not?
As always, any help will be appreciated and rewarded.
Thank you.
Comment: $X$ has to be locally compact to have a compactification in your sense.
Comment: because an open subset of a compact Hausdorff space is locally compact and completely regular.
Comment: Both necessary and sufficient.
Comment: @HennoBrandsma And why is that?
Comment: I understand. Is this a sufficient condition or just necessary?
Answer: In general (for any topological space $X$ at all), $\mathbf{X}$ is the Stone-Cech compactification of $X$.  That is, there is a canonical continuous map $i:X\to\mathbf{X}$ such that for every compact Hausdorff space $K$ and every continuous map $f:X\to K$, there is a unique continuous map $g:\mathbf{X}\to K$ such that $gi=f$.  (Explicitly, $\mathbf{X}$ is the space of $*$-homomorphisms $C(X)\to\mathbb{C}$, and $i:X\to\mathbf{X}$ takes any $x\in X$ to the homomorphism given by evaluation at $x$.)
Now, the Stone-Cech compactification is not always a compactification in your sense.  Specifically, it is a compactification (embedding with dense open image) iff $X$ is locally compact and Hausdorff.  This condition is obviously necessary, since any open subspace of a compact Hausdorff space is locally compact Hausdorff.  Conversely, the image of $i$ is always dense, and $i$ is an embedding iff $X$ is completely regular.  In particular, if $X$ is locally compact Hausdorff then $i$ is a embedding with dense image and its image is therefore open since any dense locally compact subspace of a compact Hausdorff space is open.
As for what you call $C_b(X)$, it is just $C(X_d)$ where $X_d$ is $X$ with the discrete topology.  So, the $\mathbf{X}$ you obtain from this is just the Stone-Cech compactification of $X_d$.  This will never be a compactification of $X$ in a canonical way unless $X$ is discrete so that $X=X_d$ (in particular, the canonical map $i:X\to\mathbf{X}$ is not continuous with respect to any topology on $X$ besides the discrete topology, since it is an embedding with respect to the discrete topology).
(Incidentally, what you call $C(X)$ is what is more commonly called $C_b(X)$.  Typically $C(X)$ refers to the algebra of all (not necessarily bounded) continuous functions on $X$.)
Comment: If $X$ is a locally compact Hausdorff space and $A$ is a unital $C^*$-subalgebra of $C_b(X)$ (what you called $C(X)$), then $A$ corresponds to a compactification of $X$ iff $A$ contains all functions with compact support (and every compactification can be obtained in this way).  In particular, this is true of your $C'(X)$, so it gives a compactification of $X$ (as long as $X$ is locally compact).
Comment: Every function with compact support extends continuously to every compactification, by just making it $0$ at all the new points.  So it is necessary for $A$ to contain all functions with compact support.  Conversely, functions with compact support generate the topology on $X$ (that is, the topology on $X$ is the coarsest topology making them all continuous).  So, if they all extend continuously to $\mathbf{X}$, then the image of $X$ in $\mathbf{X}$ must have the same topology, so it's an embedding.
Comment: Let $\beta X$ denote the Stone-Cech compactification of $X$ and let $i:X\to\beta X$ be the canonical inclusion map.  The inclusion homorphism $A\to C_b(X)$ can be thought of as a homomorphism $C(\mathbf{X})\to C(\beta X)$, which is given by composition with some continuous map $f:\beta X\to\mathbf{X}$ (which is surjective because the inclusion homomorphism is injective).  So, composing $f$ and $i$, we have a map $fi:X\to\mathbf{X}$, and the inclusion homomorphism $C(\mathbf{X})\cong A\to C_b(X)$ is given by composition with $fi$.
Comment: This means that the elements of $A$ are exactly the continuous functions on $X$ which can be obtained by composing a continuous function on $\mathbf{X}$ with $fi$.  In other words, if you think of $fi$ as an inclusion of $X$ into $\mathbf{X}$, they are exactly the continuous functions on $X$ which extend continuously to $\mathbf{X}$.
Comment: That follows from the fact that $X$ is locally compact Hausdorff, and involves Urysohn's lemma.
Comment: Eric, I am sorry for using a non-standard notation. Also, this is a second question of mine that you answered with a crystal clear and thorough explanation. Thank you.
Comment: If you don't mind, I would like to ask you one more question: what happens if we restrict what I called $C(X)$ to some smaller sub-algebra? For example, what if  $C'(X)$ is a collection of bounded and continuous functions from a metric space $X$ to $\mathbb{C}$ such that each for all $\epsilon>0,$ and all $r>0$ there exists a compact set $K$ in $X \times X$ such that outside of that compact set $K$ we have
 $$d(x,y)< \epsilon \implies d(f(x),f(y))<R.$$
$C(X)$ is still a unital and abelian $C^*$-algebra (under the usual operations). Is $\mathbf{X}$ still a compactification of $X?$
Comment: Great! "A corresponds to a compactification of X iff A contains all functions with compact support" - why is that? I'm sorry for so many questions!
Comment: I am not exactly sure that I understand your last comment. If $A$ is a $C^*$-subalgebra of $C(X)$ (I am using your notation/standard notation), then there exists $\mathbf{X}$ such that $A$ is isomorphic to $C(\mathbf{X}).$ However, $\mathbf{X}$ is no longer the Stone-Cech compactification of $X.$ Thus, I am not completely sure why continuous extensions of functions with compact support matter. In other words, I don't know the following: 1)  If $\mathbf{X}$ happens to be a compactification of $X,$ then why is it necessary that $A$ contains all the functions with compact support?
Comment: Maybe my confusion comes from not knowing how the isomorphism really looks like. If $f$ belongs to $A$, then does $f$ correspond to its extension in $C_b(\mathbf{X})$? Or is it correspondence only true in case of $C_b(X)?$
Comment: Thank you for all your time and answers! One more thing seems unclear to me, though. "Functions with compact support generate the topology on X" - I understand that functions with constant support generate some topology on $X.$ However, how do I know that that topology is not "coarser" than the original topology on $X?$


use inverse Wishart for variance in MCMC

Question: When you have a posterior that looks like as one step in Gibber Sampler
$P(\xi | \Sigma_\xi, \theta) ∝ exp\{-1/2 \xi\Sigma_\xi^{-1}\xi\}P(data | \xi, \theta)$
Do you always assume inverse Wishart for $\Sigma_\xi$?
$\Sigma_\xi^{-1} $ ~ $Wishart(\nu_n, \Delta_n)$
where,
$\nu_n = \nu_o + N$, and
$\Delta_n^{-1} = \Sigma\xi\xi^{-1} + \Delta_o^{-1}$
When I check the original derivation, it was for the case when the likelihood was in exponential family. Would this be true if $P(data | \xi, \theta)$ was an arbitrary function, not an exponential family?
I saw one author doing exactly this for a very complex likelihood function.
Answer: There is no "default" priors for anything. Wishart distribution is commonly used for variance because it is a conjugate prior for multivariate normal distribution, but that does not make it anyhow "better" then any other choice for prior distribution.
Comment: I agree with your argument that it is not better than any other. But is it common practice to just default to it since there is nothing else better?
Comment: @biomass better in what sense?
Comment: in the sense you meant. I am just wondering whether what this author did is common practice.
Comment: @bhomass but I didn't say it's better.
Comment: I didn't say you said it's better. You quoted "better", I am only paraphrasing. To be exact, you said it's not better, I am using it to also mean it's not better, only a common practice.
Comment: @bhomass it used commonly because it's conjugate and it makes computation easier. There are often cases where you would choose non-conjugate prior instead of conjugate prior (e.g. people often use uniform priors for different things).
Comment: I don't have problem with the notion that you are free to choose just about any prior. But in this case, the wishart parameters are derived on the basis that the likelihood function is a exponential family, when it really isn't. This is the first time I encountered such looseness in statistical derivations. I have added the posterior parameter formula in the original post.


Simple Derivation of Functional Equation Question (L'Hospital's Rule)

Question: First, the question is:

$f$ is a differentiable function and $f : R \rightarrow R$
$xf(x)-yf(y)=(x-y)f(x+y)$
$f'(2x)=?$

My approach for problem is using L'Hospital's rule:
$$ \frac{xf(x)-yf(y)}{x-y}=f(x+y) $$
Assuming $y=x$
$$ \lim_{y\to{x}} \frac{xf(x)-yf(y)}{x-y} = f(2x)$$
Taking the limit using l'hospital's rule:
$$ (yf'(y)+f(y))|_{y=x} = f(2x) $$
Taking the derivative of both sides:
$$ f'(x)+xf''(x)+f'(x)=2f'(2x) $$
But the answer is $f'(2x)=f'(x)$
How is this possible?
Comment: Could it be that $f$ must have the form $f(x)=ax+b$? (All such functions satisfy the functional equation.)
Comment: I'm not sure, but if $f$ is as I wrote above, then $f''(x)=0$, and thus $2f'(x)=2f'(2x)$, so the statement follows. I have to sleep, but I'll have a look at this problem tomorrow again...
Comment: @mickep thanks, but what's wrong with my solution? Is there a way not using functional equation techniques ie. finding the function? Maybe a calculus route?
Answer: In equation
$$
xf(x)+yf(y)=(x-y)f(x+y),
$$
differentiate with respect to $x$ to get
$$
xf'(x)+f(x)=(x-y)f'(x+y)+f(x+y).
$$
Now, set $y$ to $x$ and to $-x$ to deduce that
$$
xf'(x)+f(x)=f(2x)=2xf'(0)+f(0),
$$
which proves that $f(x)=xf'(0)+f(0).$
Comment: This is the solution. Thanks.


Use of aubio.note() in python?

Question: I'm trying to use aubio to find the notes in a recording. Whenever I call aubio.notes with aubio.notes(filename), the python shell crashes (windows dialogue: pythonw.exe has stopped working). The only "documentation" I found are these pages for the command line commands. I figured out the first argument is a string (presumably file name?). Based on the uses of aubio.pitch and aubio.tempo and aubio.source, note is a class, and methods are used on an instance of this class. Does anyone know how to use this?
It does work when called as n = aubio.note(), but I've no clue where to go from there
Answer: See https://github.com/aubio/aubio/issues/60
See also this demo
Please open a new bug report whenever you see a such crash.

Use of aubio.note() in python?


Implicit Function Theorem Help in a Macroeconomic Model

Question: I'm having trouble with this problem:
"A macroeconomic model contains the following equations characterizing its steady state:
$1=B(f'(K)+1-Y)$
$C+YK=f(K)$
$B$ and $Y$ are parameters between $0$ and $1$.
The question is, "do these equations implicitly define $C$ and $K$ as $C^1$ functions of $Y$ in an open ball around $(C^*,K^*)$? If so find the derivatives of these implicit functions with respect to $Y$.
Now I have no problem doing implicit function theorem problems in my textbook. But I don't know where to start with this one. Help! Can I just take the derivative of $f(k)$ and plug it into the first function?
EDIT: Here is what I know about $f$:
$$f:\mathbb{R}^+ \to \mathbb{R}^+ , f \in C^2 \text{ and } f' > 0, f'' < 0.$$
Answer: One (standard) method is to differentiate the system. Since $B$ is held constant, we can set ${\rm d}B=0$. This should yield
$$0=f''(K){\rm d}K-{\rm d}Y$$
$${\rm d}C+K{\rm d}Y+Y{\rm d}K=f'(K){\rm d}K$$
This is a linear system of equations in the differentials,
$$\pmatrix{0 & f''(K) \cr 1 & Y-f'(K)}\pmatrix{{\rm d}C \cr {\rm d}K}=\pmatrix{{\rm d}Y \cr -K{\rm d}Y}$$
The condition in the implicit function theorem is the same that the coefficient matrix in this system is invertible. This is OK since $f''<0$, and then you can solve for ${\rm d}C$ and ${\rm d}K$ by linear algebra.
Comment: Thanks! I was having trouble getting off the ground and this helps a lot.
Comment: just one question. Since why is a constant, why isn't dY=0 too?
Comment: The usual way to think about this is that for each fixed value of $Y$, you can solve your system of equations and find $C$ and $K$. Now what you are supposed to describe in your problem is how your solution of $C$ and $K$ changes when you start with a slightly different value of $Y$.
Comment: oh ok so Y varies because we are deriving C and K with respect to Y. Got it
Comment: Having trouble solving dC and dK w/ linear algebra. Is using the Jacobian a better method?
Comment: The two methods give you the same equations to solve, so in terms of computational difficulty they are fully equivalent. If you prefer, you can solve the first equation for ${\rm d}K$ and substitute into the second. You may also "divide through" by ${\rm d}Y$ to get equations between derivatives instead of differentials.
Answer: First of all, I think that your confusion might be caused (at least partially) by denominating $Y$ as a parameter. In fact, to use the Implicit Function Theorem, it should be ssen as a variable (it's implied in the problem). :D
If that's not the case, here's my answer, as fully detailed as I could do it (so you can see where you are confused):
To use the Implicit Function Theorem, the first thing you need is to have functions that define a relation between all your variables. These functions must be constant (and not other variable), and it's commonly regrouped so the functions are xero-valued (not that it really mattters).
Let's reorder the equations to be expressions equal to 0, and then name the equations as functions of $(Y,C,K)$:
$$\begin{array}{rcl}
F(Y,C,K)&=&B(f'(K)+1-Y)-1=0\\
G(Y,C,K)&=&C+YK-f(K)=0
\end{array}$$
Now you have two functions, $F$ and $G$, that define a relation between $C$, $K$ and $Y$. With 2 functions and 3 variables, you could say that two of those variables can be expressed as a $C^1$ function of the remaining one in an open ball, provided certain conditions are met.
The condition for $C$ and $K$ to be functions of $Y$ in an open ball around $(C*,K*)$ is that the Jacobian of the functions with respect to the variables is invertible, namely
$$\det\biggl(J_{(C,K)}^{(F,G)}\biggr)\neq 0$$
where
$$ J_{(C,K)}^{(F,G)}=\frac{\partial (F,G)}{\partial (C,K)}=\Biggl[\begin{array}{cc}\frac{\partial F}{\partial C}&\frac{\partial F}{\partial K}\\\frac{\partial G}{\partial C}&\frac{\partial G}{\partial K}\end{array}\Biggr]$$
is the Jacobian of the functions $F$ and $G$ (seen as a vector function) with respect to the variables $C$ and $K$.
Let's calculate the Jacobian:
$$\begin{array}{rcl}
\frac{\partial F}{\partial C} &=&0\\
\frac{\partial F}{\partial K} &=&Bf''(K)\\
\frac{\partial G}{\partial C} &=&1\\
\frac{\partial G}{\partial K} &=&Y-f'(K)\\
\end{array}$$
Then
$$\begin{array}{rcl}
\det\biggl(J_{(C,K)}^{(F,G)}\biggr)&=&\frac{\partial F}{\partial C}\cdot \frac{\partial G}{\partial K}-\frac{\partial F}{\partial K}\cdot\frac{\partial G}{\partial C}\\
&=&0-1\cdot(Bf''(K))\\
&=&-Bf''(K)
\end{array}$$
We know that $f''<0$ everywhere in its domain, so the only way that the expression is zero is that $B=0$. But $B\neq0$, because that would imply $1=0$ in the expression $F(Y,C,K)$.
So, the determinant of the Jacobian is NOT zero, and the variables $C$ and $K$ can be expressed as $C^1$ functions of $Y$ in an open ball.
Now, we need to find the expressions of the partial derivatives $C_Y$ and $K_Y$. This can be done by calculating
$$ C_Y=-\frac{J_{(Y,K)}^{(F,G)}}{J_{(C,K)}^{(F,G)}}$$
$$ K_Y=-\frac{J_{(C,Y)}^{(F,G)}}{J_{(C,K)}^{(F,G)}}$$
Note that the derivatives are calculated using the Jacobian of the functions $(F,G)$ with respect to all variables $C$, $K$ and $Y$. The numerator and denominator switch their variables: in the demoninator, you use the "dependent" variables, and in the numerator you switch the wanted dependent variable with the requested independent variable.
We need to find
$$\begin{array}{rcl}
\Rightarrow J_{(Y,K)}^{(F,G)}&=&F_YG_K-F_KG_Y\\
&=&(-B)\cdot(Y-f'(K))-(Bf''(K))\cdot(K)\\
&=&BY+Bf'(K)-Bf''(K)
&&\\
&&\\
\Rightarrow J_{(C,Y)}^{(F,G)}&=&F_CG_Y-F_YG_C\\
&=&(0)\cdot(K)-(-B)\cdot(1)\\
&=&B
\end{array}$$
Finally,
$$\begin{array}{rcl}
C_Y&=&-\frac{J_{(Y,K)}^{(F,G)}}{J_{(C,K)}^{(F,G)}}\\
&=&-\frac{BY+Bf'(K)-Bf''(K)}{-Bf''(K)}\\
&=&\frac{Y+f'(K)-f''(K)}{f''(K)}\\
&&\\
&&\\
K_Y&=&-\frac{J_{(C,Y)}^{(F,G)}}{J_{(C,K)}^{(F,G)}}\\
&=&-\frac{B}{-Bf''(K)}\\
&=&-\frac{1}{f''(K)}
\end{array}$$


SSIS Problem reading from flat file

Question: My SSIS program reads as input from a .csv file.
The file has about 60,000 rows. And my SSIS package fails during the read, saying cannot convert a certain column due to potential loss of data.
Now, I am certain that the majority of the rows are correct. As I have tried pasting random subset of the file and the SSIS reads fine.
But I can't figure out a way to determine exactly on what line did my package fail.
I have spent 2 months on this problem, any advice?
Answer: You could find out the first culprit with 16 iterations. Here is a brain + brawn method:
First: Back everything up. Make copies of backups in safe places. Sorry to state the obvious, but I've recently been bitten, and I know better.
The file with 60K records - let's call this your base file.

Split base file into two files (FileA, FileB)
Use one of them as input.
Run SSIS - if SSIS fails, use FileA as your base file - else use FileB as base file
Go to step 1

You will have the offending record at the 16 iteration. (60k, 30k, 15k, 7500, 3750, 1875, 937, 468, 234, 117, 58, 29, 14, 7, 3, 1)
Turn logging on for everything and rerun the SSIS package. You should have the offending record in the base file and the exact data point in the log.
Answer: First, simplify the problem. Create a data flow task that only uses this flat file source, and some dummy destination. Watch that fail.
Turn on all logging and page through the logs. Turn off logging areas you find are obviously worthless, and run it again.
Also, you should configure the error output of the source and/or destination: whichever one is giving you the error. Send the erroneous row to a separate destination that you can look at after the run.
Answer: Most of the time when I have run across this it was the result of either data that was longer than expected (i.e. trying to fit a 60 character string into a varchar(50) field), or it was a number where precision might be lost (i.e. fitting a 26.5 into an integer field or a 26.55 into a number field that only allows for one decimal place).
Answer: Set DefaultBufferMaxRows = 1
This will read and process each line one-by-one and will fail on the row that it's having conversions issues with.
There's no need to do this manually by splitting out the file.

SSIS Problem reading from flat file


Optimal SVN server configuration for personal use

Question: I have a desktop machine that is currently set up as an SVN server.  Currently I have it set up to be accessible via http (using Apache), but only for the local network, as I wasn't confident in my handling of the security issues involved in opening it up.
My two options for handling that, as I see it, are SVN over SSH and setting up a VPN while maintaining the "local network only" configuration for SVN.  Does anyone have a strong idea of the pros and cons of these two setups?  In particular, I've never tried to set up a personal VPN before, so I don't know how much pain and expense is usually involved with OpenVPN and such.
[Aside: I don't know if this is actually more appropriate to SuperUser because it's a server for personal use - if it is, can someone please move it?  Thanks!]
Answer: Why not use apache mod_dav_svn (w/wout mod_ssl) for providing remote access to your svn with apache basic auth at least.
Comment: Because that adds an extra Apache module, meaning more memory usage and more to maintain.  For a single user, svn+ssh://my-username@my-server/my-home/my-repo is the most simple way of accessing an svn server, no need to bother with any new software.
Answer: If it's just you accessing it, svn+ssh is ABSOLUTELY the way to go. SSH is best-of-breed security, you get privacy and authentication for free, and if you run the server via SSH, you've already got it set up.
The only downside to doing it over SSH is that if you have multiple users accessing it, each has to have their own accounts and you have to manage permissions carefully or one person won't be able to access the repo after another person commits to it.
For that reason, I do not recommend svn+ssh for situations where multiple people will need commit access. In situations where just one person needs to commit, it's perfect.
Answer: You don't need to run SVN+SSH to secure your subversion repo, run it over https instead and you'll get enough encryption to prevent snooping. Access control is the next thing you add, so you always have to log on - and only give it a single user.
As long as the repo file are not accessible, you should be perfectly secure. If you really want to be paranoid, add a firewall or apache directive restricting access to your client IP only.


Comparing coefficients across segmented regression models given aggregated, heterogenous data

Question: My goal is to compare $\beta_1$ across $7$ models:
\begin{align*}
Y^1_t &= \beta_0 + \beta_1 X^1_t + \epsilon_t \\
Y^2_t &= \beta_0 + \beta_1 X^1_t + \epsilon_t \\
&\vdots \\
Y^7_t &= \beta_0 + \beta_1 X^7_t + \epsilon_t,
\end{align*}
where the realizations of $Y^L_t$ is time series data and $X^L_t$ is a binary variable for $L=1, 2, \dotsc, 7$, interpreted as a treatment.
I want to compare $\beta_1$ across models to determine if the treatment had a heterogenous effect on $Y^L_t$. To be concrete, the response variables are different types of crime.
The problem is the means and variances of $Y^1, Y^2, \dotsc, Y^7$ vary greatly. For the sake of argument, suppose $Avg(Y^1) = 100, Avg(Y^2) = 4$, and $\hat{\beta^1} = -25, \hat{\beta^2} = -1$. In terms of averages, the effect is the "same", but the coefficients are quite different. So a direct comparison becomes difficult if not untenable.
I have tried to identify some approaches to compare the treatment effects:

Percent change

Using the pre-treatment average of $Y^L$, compute percent change $PC = \frac{\beta_1}{Avg(Y^L)} \cdot 100$, and compare those.

Standardize $Y^L$

Using overall Avg / Std, or the pre-treatment Avg / Std, estimate the coefficients and make direct comparisons (perhaps using a hypothesis test of their difference).

$Z$-test for difference of coefficients (or some other test)

Seemingly unrelated regression

Ideally, I want to determine if the difference in effects is statistically significant or not.
Answer: If willing to reparametrize the model, you might consider Poisson regression. So for Poisson regression we have:
$$\mathbb{E}[Y_t^1] = \exp(\beta_{01} + \beta_{11}X_t^1)$$
$$\mathbb{E}[Y_t^2] = \exp(\beta_{02} + \beta_{12}X_t^2)$$
etc. for your stacked equations. Note that $\exp(\beta_{01} + \beta_{11}X_t^1) = \exp(\beta_{01}) \cdot \exp(\beta_{11}X_t^1)$, and so in this formulation you get your proportional interpretation of the effects for $\beta_{11}$, $\beta_{12}$. (But I would still test for equality on the linear scale.)
For this simple of a model with all binary inputs on the right hand side, each of them will estimate the same expected value on the left hand side, it just changes the parameters and the nature of how you generalize across equations.
Stata has a program to do seemingly unrelated regression for glm's, see the suest command. Another way to do it though is to stack the equations, fit a single model, and do a likelihood ratio test for the restricted vs the model allowing the treatment effects to vary across the equations. Example below in R:
# Simulating two crime types
set.seed(10)
n <- 1000
t <- rep(0:1,n/2)
# crime type 1, ~10
l1 <- exp(2 + 0.5*t)
c1 <- rpois(n,l1)
# crime type 2, ~130
l2 <- exp(4.6 + 0.5*t)
c2 <- rpois(n,l2)
# Prepping data
ctype <- rep(c("c1","c2"),each=n)
data <- data.frame(crime=c(c1,c2),treat=c(t,t),c=ctype)

# Linear regression
linmod <- lm(crime ~ c + treat:c - 1, data)
summary(linmod)

# Poisson regression
pmod <- glm(crime ~ c + treat:c - 1, family='poisson', data)
summary(pmod)

# Showing linmod and pmod give the same marginal predictions!
all.equal(predict(linmod),predict(pmod,type='response'))

# Restricted model with no treatment het
pmod_restrict <- glm(crime ~ c + treat - 1, family='poisson', data)
anova(pmod_restrict,pmod,test='LRT') #Fail to reject null

Comment: Here you go @Andrew, wrote a blog post to show how to do the [Wald tests in python](https://andrewpwheeler.com/2021/06/18/wald-tests-via-statsmodels-python/) via the stacked model approach.
Comment: Does the LRT apply to the overall difference between the restricted / unrestricted models, or does it apply to the specific coefficient? Say I have 3 regressors (the same for both models) and I restrict x1, does this test make a claim about x1?
Comment: LRT is for the overall model - so all differences in the model at once. If you want to test individual coefficient contrasts one-by-one, doing the approach via SUR and then Wald tests makes sense. (You can do the Wald tests via the stacked model as well.)
Comment: I'm assuming the Wald Test applies to the coefficient that is restricted? Is there a way to implement a SUR model with Wald tests in Python? I have attempted Statsmodels and linearmodels, but the former has not released a stable version of SUR, and the latter does not have Wald for SUR, to my knowledge.
Comment: Yeah not sure how Stata implements SUR for GLM's, so if going with python would suggest the stacked model approach. For the Wald tests you should adjust the variance/covariance matrix to account for clustering (and possible overdispersion with Poisson models). See [here](https://stackoverflow.com/a/28930697/604456) for example Wald test in python. *I believe* the R matrix should sum to 0 in this example, so if doing three coefficients set two to -1/2 and the other coefficient to 1.


Using FFmpeg, How to convert an frame image sequence to video(video codec : dvvideo)?

Question: I'm working on an image processing project.
The project proceeds as follows:
Input Video -> frame Sequence -> (Processing) -> Output Video
I want to create the output video with the same specifications as the input video.
The specs of the input video we use are:
- Size : 1280x1080, Codec : dvvideo, pixel_fmt : yuv422

But I am struggling with the following error:
Duration: 00:00:03.20, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: png, rgb24(pc), 1280x1080 [SAR 3:2 DAR 16:9], 29.97 fps, 29.97 tbr, 29.97 tbn, 29.97 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> dvvideo (native))
Press [q] to stop, [?] for help
[dvvideo @ 0x55b965b745c0] DVCPRO HD encoding is not supported.
[dvvideo @ 0x55b965451780] ff_frame_thread_encoder_init failed
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

The code is below. (subprocess in python)
ffmpeg -y -f image2 -r "{fps}" -i {input_img_root} -s 1280:1080 -b:v {bit_rate}  -vcodec dvvideo -pix_fmt yuv422p "{output}"

(fps and bitrate are taken from the input video. and input_img_root and output are paths.)
I've been trying for several days, but it doesn't work.
If you know any way to make a new video while maintaining the video specifications (Especially dvvideo codec)(not using ffmpeg), it would be nice to share it.
Comment: You're hiding too much info behind python variables (as to what you actually commanded ffmpeg to do) but one error is `-s 1280:1080` which needs to read `-s 1280x1080`
Answer: I solved this problem.
This problem was due to the version of FFmpeg.
After upgrading the version of FFmpeg to 4.4.2(latest version), the error disappeared.


Can a disabled module pose a security risk?

Question: Is it possible for a module that is installed but not enabled to pose a security risk?
Corollary question: Is it a recommended practice to update disabled modules when security updates become available?
Answer: I say yes since the vulnerable code is still sitting right there in the web directory which can (will) be scanned by potential attackers. Monitoring disabled modules is certainly part of our update routine.
I didn't find a whole lot of discussion on this in regards to Drupal in particular, likely because it isn't really a Drupal-related issue (see note below). Greg Knaddison (@greggles on Drupal.org) noted on a thread a few years back that:

Sometimes the vulnerabilities exist regardless of whether or not the
  module is enabled, so I think it's better to warn on all modules.

Source: Issue #942960: Don't warn about security updates for disabled modules
As always, best to err on the side of caution.
Note: I think this is more a server admin related question, but I hesitate to flag it as not relevant since IMHO people who aren't server admins shouldn't really be expected to know that.


Probability Distribution on unit circle

Question: I currently stuck at following task:
So I have a probability distribution:
$$
f^{X,Y}(x,y) = \frac{1}{\pi}\; \forall(x,y)\in A
$$
where $$
A = \{ (x,y) : x^2+y^2 \leq 1 \}
$$
-I now want to calculate the Variance(X), Variance(Y) and the Covariance(X,Y). I know that Cov(X,Y) = 0, but how do I show this ?
-Also I want to show that X and Y are not indipendent. I mean intuitively it is pretty clear that X and Y are dependent of each other, but how do i show it mathematically? So obviously i need to show that
$$
\mathbb{P}(X\cap Y) =\mathbb{P}(X)\mathbb{P}(Y)
$$
but how ?
Answer: Hints: covariance can easily be calculated using polar coordinates: $cov(X,Y)=\int_0^{1} \int_0^{2\pi} (r\cos \, \theta)(r\sin \, \theta) rdrd\theta$. To show that $X$ and $Y$ are not independent consider $\{Y>\frac 1 2\}$ and $\{|X| \leq\frac {\sqrt 3} 2\}$. Note that $X^{2}+Y^{2} \leq 1$ so $Y>\frac 1 2$ implies $|X| \leq\frac {\sqrt 3} 2$. If $X$ and $Y$ were independent we would have $P\{Y>\frac 1 2\}P\{|X| \leq\frac {\sqrt 3} 2\}=P\{ Y>\frac 1 2,|X| \leq\frac {\sqrt 3} 2\}$. But the right side is simply $P\{Y>\frac 1 2\}$. So the equation becomes $P\{|X| \leq\frac {\sqrt 3} 2\}=1$ (because $P\{Y>\frac 1 2\} \neq 0$). Can you check that this is a contardiction? (Use polar coordinates for all probabilities).
Comment: But in order to calculate Var(x) and Var(y) i first have to calculate the marginal distributions right ? So that I have:
$$
\mathbb{E}(X) = \int x f^X(x) dx
$$
and then I substitute $ x = r \cdot cos(\theta)$ right ?
Comment: $Var(X)=\int_A x^{2} \frac 1 {\pi} dxdy$ which can be calculated easily using polar coordinates. (Note that $EX=EY=0$ by symmetry).
Comment: @RedCrayon The formula $Eg(X,Y) =\int \int  g(x,y) f_{X,Y}dxdy$ can be used when $g(x,y)$ depends only on one variable (like $g(x,y)=x^{2})$. So you can find $var (X)$ without calculating the density of $X$.

Probability Distribution on unit circle


How to edit user profile in Django

Question: I'm trying to create an "Edit Profile" form in the fronted. What happens is that my form(i'm not 100% sure) tries to create a user instead of finding the current user and update his profile. So I think that's the issue. Checked many questions here but none was clear enough. The fields I'm trying to edit are email(to be checked if email already exisit), Name, user type, and password.
below is the code for forms.py
class ProfileForm(forms.ModelForm):
    password1 = forms.CharField(label='Password', widget=forms.PasswordInput)
    password2 = forms.CharField(label='Confirm Password', widget=forms.PasswordInput)

    class Meta:
        model = User
        fields = ('full_name', 'active', 'admin','email','user_type')

    def clean_password2(self):
        # Check that the two password entries match
        password1 = self.cleaned_data.get("password1")
        password2 = self.cleaned_data.get("password2")
        if password1 and password2 and password1 != password2:
            raise forms.ValidationError("Passwords don't match")
        return password2

    def save(self, commit=True):
        # Save the provided password in hashed format
        user = super(ProfileForm, self).save(commit=False)
        user.set_password(self.cleaned_data["password1"])
        # user.active = False # send confirmation email
        if commit:
            user.save()
        return user

code for views.py
def profile(request):
    user = User.objects.filter(email = request.user)
    # form = ProfileForm(user.values().first())
    if request.method == 'POST' :
        form = ProfileForm(request.POST)
        if form.is_valid():
            form.save(commit=True)
    form = ProfileForm(user.values().first())
    # print(user)
    context = {
    'object_list': user,
    'form': form
    }
    return render(request, 'userpanel/profile.html',context)

Thanks in advance
Comment: Do you want edit profile of logged user?
Comment: @MohammadAli yes
Answer: You didn't pass instance argument to the form. Without instance it will try to create new object. It should be like this:
form = ProfileForm(request.POST, instance=user)

Also you don't need to query user with this: user = User.objects.filter(email = request.user). Just add login_requred decorator to ensure user is authenticated and use request.user:
from django.contrib.auth.decorators import login_required

@login_required
def profile(request):
    if request.method == 'POST' :
       form = ProfileForm(request.POST, instance=request.user)

Comment: when I was using the filter then it was giving the error 'QuerySet' object has no attribute '_meta'. now after user the request.user that error too gone. thanks for the update.

How to edit user profile in Django


Como pegar a assinatura de métodos alterados em um commit

Question: Então pessoal, eu preciso da assinatura de todos os métodos que foram alterados em um commit, seja atualizado, removido ou adicionado.
Exemplo:
Neste commit
Os métodos alterados foram:
 - br.ufrn.ase.Classe1.metodoB(int b)U
 - br.ufrn.ase.Classe1.getV()D
 - br.ufrn.ase.Classe1.metodoadicionado()A
 - br.ufrn.ase.Classe2.metodoQualquer(int i)A
 - br.ufrn.ase.Classe2.outro(int j)A

Alguém tem ideia de como fazer isso ou sabe de alguma lib que já faça? O código de acessar os commits e pegar o blob eu já tenho.
Estou desenvolvendo em Java, mas a lib pode ser em qualquer linguagem.
Obrigado desde já.
Answer: Consegui resolver esse problema usando a seguinte combinação:

Com um git blame entre o commit desejado e o seu parent eu consigo extrair quais linhas foram modificadas e pego os respectivos números das linhas;
Implementei um JDT Parser (usando AST.JLS8) que faz o parse dos arquivos modificados armazenando a linha inicial e final de cada método declarado;
Daí é só passar o número da linha modificada obtida no passo 1 e procurar dentro do range dos métodos obtidos no passo 2.

Não consegui um bom resultado com esse exemplo do Pedro, alguns retornaram o pacote da classe no lugar da assinatura do método.
Answer: GIT já identifica, quando possível, o método que foi modificado.
Basta lançar o comando:
git show $COMMIT

Para filtrar melhor o resultado pode ser por linha de comando (via linux)
git show $COMMIT | grep @@ | sed 's/@@.*@@//g'

Este comando procura pelos arrobas, indicando a mudança, depois remove o começo da linha que contem o range de linhas modificados.
Em windows, voce pode fazer a primeira parte com FINDSTR.
Estou usando git v2.4.0.


Sudo resetting environment variables

Question: Whenever I use the sudo command, none of my environment variables appear to be preserved.
I've tried modifying the sudoers file, by removing the env_reset specification.
<code>#Defaults   env_reset
</code>
I've also tried adding the env variable that I want using env_keep.
<code>Defaults   env_keep += "DYLD_LIBRARY_PATH"
</code>
I'm using Mac Snow Leopard.
Is there something that I am missing?
Answer: DYLD_LIBRARY_PATH seems to be excluded from the
    Defaults   env_keep += "Defaults   env_keep += "DYLD_LIBRARY_PATH"
"
If you try any other variable it works for example
    Defaults   env_keep += "VAR"
You can probably get over this by providing DYLD_LIBRARY_PATH on the commandline
for example (where cmd is the command you want to pass DYLD_LIBRARY_PATH)
<code># sudo DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH cmd
</code>
Hope this helps

Sudo resetting environment variables


Export/Import OWASP ZAP Passive Scan Rules

Question: Is there any way to create a scan policy for passive scans? I know you can create and modify scan policies for the active/attack scanning, but i'm wondering if you can do the same for the passive scan rules or if you have to individually modify them on every machine?
Comment: You can enable/disable specific rules via the API, or through scan policy. Choose what fit better your scenario.
Comment: Thanks, I was able to figure that part out, but I was more so wondering if there is a way to create a passive scan policy the same way you can create an active scan policy?
Comment: From my understanding yes - look on this [policy](https://github.com/zaproxy/zaproxy/blob/develop/docker/policies/API-Minimal.policy) for example.
Answer: There's an existing ticket open to unify Active/Passive Scan handling in a singular policy type interface: https://github.com/zaproxy/zaproxy/issues/3870.
If you're really interested in that you could support it on BountySource (https://www.bountysource.com/issues/49047644-improved-active-passive-rules-management) and see if that draws some attention/action.
Another option you could go with is to create a quick script that uses ZAP's web API to apply a Passive Scan rule "policy". Relevant endpoints include: pscan/view/scanners/, pscan/action/disableAllScanners/, pscan/action/enableScanners/. Here's a python example:
from zapv2 import ZAPv2 as zap
import time

apikey = "apikey12345" #Your apikey
z = zap(apikey=apikey, proxies={"http": "http://127.0.0.1:8080", "https": "http://127.0.0.1:8080"})
time.sleep(2) #Might need to be longer depending on your machine and if ZAP is already running or not

print "Disabling all passive scan rules.."

z.pscan.disable_all_scanners()

scanners = z.pscan.scanners

for scanner in scanners:
    print scanner.get("id") + " : " + scanner.get("enabled") + " : " + scanner.get("name")

to_enable = "10020,10021,10062" #Customize as you see fit
print "\nEnabling specific passive scan rules..[" + to_enable +"]"

z.pscan.enable_scanners(to_enable)

print "\nListing enabled passive scan rules.."

scanners2 = z.pscan.scanners

for scanner in scanners2:
    if (scanner.get("enabled") == "true"):
        print scanner.get("id") + " : " + scanner.get("enabled") + " : " + scanner.get("name")

Finally you could configure ZAP on one system, then copy that config.xml to other systems as needed.


If $(F:E)<\infty$, is it always true that $\operatorname{Aut}(F/E)\leq(F:E)?$

Question: If I understand correctly, Arturo Magidin says in this comment that the following is true.

If $E\subset F$ is a finite field extension, then $|\operatorname{Aut}(F/E)|\leq (F:E).$

I understand why this is true when the extension is simple, but the general case eludes me. (I also understand that my particular extension in that question is simple, so this is irrelevant to the problem, but I would like to understand this.)
Let $E\subset F$ be a finite extension. Any finite extension is finitely generated, so let $$F=E(a_1,\ldots,a_n).$$ Let $$E=F_0\subset F_1\subset\ldots\subset F_n=F,$$ with $F_i=F_{i-1}(a_i)$ for $i=1,\ldots,n.$ Let $f_i$ be the minimal polynomial of $a_i$ over $F_{i-1}$, and let $g_i$ be the minimal polynomial of $a_i$ over $E$. Then $$(F:E)=(F_n:F_{n-1})\cdots (F_1:F_0)=\deg(f_n)\cdots\deg(f_1).$$
I think I can't use $f_i$ to bound the order of $\operatorname{Aut}(F/E).$ Surely, $$|\operatorname{Aut}(F/E)|\leq \deg (g_n)\cdots\deg (g_1),$$ because the generators must be mapped to roots of their minimal polynomials over $E$ (taking non-minimal polnomials here would only make things worse), but this doesn't work because the inequality we have is $$\deg (f_i)\leq\deg (g_i),$$ not the other way around.
Comment: You should use Dedekind's theorem: see [here](http://math.stackexchange.com/questions/141548).
Comment: In fact, you CAN say $|\mathrm{Aut}(F/E)| \leq \mathrm{deg}(f_n) \cdots \mathrm{deg}(f_1)$. Prove this by induction. If you know this is true for $n=k$, take any automorphism $\sigma$ of $F_k/E$. If $\tilde{\sigma}$ is an extension of $\sigma$ to $F_{k+1}$, then under $\tilde{\sigma}$ a root of $f_{k+1}$ must go to a root of $\sigma (f_{k+1})$ (this meaning $\sigma$ applied to the coefficients of $f_{k+1}$). There are thus at most $\mathrm{deg}(f_{k+1})$ such extensions, and so at most $\mathrm{deg}(f_{k+1}) \mathrm{deg}(f_k)\cdots \mathrm{deg}(f_1)$ automorpshims of $F_{k+1}/E$.
Comment: @BarrySmith Thanks. Could you clarify one thing for me? You're counting the extensions of automorphims of $F_k/E.$ How do we know it's the same as counting the automorphisms of $F_{k+1}/E?$ Can't there be an automorphism $\rho$ of $F_{k+1}/E$ such that $\rho|_{F_k}$ is not an automorphism of $F_k/E?$ I mean, can't this restriction have an image that's different from $F_k?$ Certainly, this image has to share the copy of $E$ with $F_k,$ but is this enough?
Comment: You're right --- I meant to say to take $\sigma$ to be any embedding of $F_k$ into $E$ (which is what must be induced by an automorphism of $E$).  If $\tilde{\sigma}$ is an extension, then under $\tilde{\sigma}$, a root of $f_{k+1}$ must still go to a root of $\sigma (f_{k+1})$.  This is essentially the idea Serge provided --- that counting embeddings is cleaner than counting automorphisms --- but I wanted to point out that you can apply this idea in the context of the work you gave above, and also to indicate that there is no need to involve an algebraically closed field.
Comment: @BarrySmith I think I understand what you are saying, but I'd like to make two points perfectly clear. Did you mean "embedding of $F_k$ into $\overline{E}?$ And when you said that $\sigma$ is what must be induced by an automorphism of $E$, what did you mean? Isn't $\sigma|_E=\operatorname{id}_E?$
Answer: Instead of counting automorphisms of $F$ over $E$, count embeddings $\sigma: F\hookrightarrow \bar E$ that act identically on $E$. If we denote the number of such sigmas by $(E:F)$, then  one has $(K:F)=(K:F)(F:E)$ if $K\supset F\supset E$ is a tower of finite extensions. One checks immediately that $(K:F)\le [K:F]$ if $K=E(\alpha)$ is a simple extension, which implies the inequality $(F:E)\le[F:E]$ for arbitrary finite extensions.
Obviously, order of $\mathrm{Aut}(F/E)$ is less or equal to $(F:E)$.
Cf Lang's "Algebra" (3 ed.), Ch. 5, Secs. 2-4.
Comment: Thanks for the answer. Did you mean "Obviously, order of $\operatorname{Aut}(F/E)$ is less or equal to $(F:E)$," with $(F:E)$ instead of $[F:E]$ at the end?
Comment: Yes, this is what I meant. Sorry for the typo.
Comment: No problem. :) I'll edit that.


Problemas com o módulo gm (GraphicsMagick) e node.js

Question: Estou fazendo um projeto teste para ver as funcionalidades do módulo gm, pois preciso manipular imagens no servidor (redimensionar, colocar watermark, gerar thumb, entre outros).
Pois bem, instalei o GraphicsMagick via Homebrew usando o comando
brew install GraphicsMagick

E a instalação rolou ok.
Instalei o módulo no projeto via npm usando
npm install gm --save

Estou usando express, e no meu route usei o seguinte código
exports.tamanho = function(req, res){

gm('./images/1.jpg').size(function(err, value){
  if(err) res.json(500, {message: err});
  res.json(200, {messagem: value});
});}

Fazendo o devido require no topo usando var gm = require('gm');
Este código está retornando para mim um erro que não consegui identificar
{message:{"code": null,signal": "SIGTRAP"}}
Alguém que já teve algum problema parecido sabe o que pode ser, se consegue resolver ou se tem algum outro módulo bom de usar para as tarefas que preciso?
Comment: O que faz esse método `size`? Não achei na [documentação do gm](http://aheckmann.github.io/gm/docs.html).
Comment: http://aheckmann.github.io/gm/docs.html#getters
Coloquei ele apenas para testar...mas tenho um de recortar que dá o mesmo erro
Answer: Resolvi removendo o ImageMagick que estava instalado e o GraphicsMagick.
Reinstalei apenas o GraphicsMagick e funcionou.


Quantile of kernel density estimator

Question: I consider the kernel density estimation, and I want to find the quantile the quantile at some level (0.95) of the KDE.
z<-rnorm(25)
v<-density(z)
plot(v$x,v$y,type="l")

qt<-quantile(z,0.997)
abline(v=qt,col="red")

plot(v$x,cumsum(v$y),type="l")
abline(h=0.997,col="blue",type="p")

The first plot is the plot of the KDE and the line of the quantile of the data.
How to get the distribution of the KDE?
How to get the quantile with respect to the KDE? Is only for comparison.
Comment: This is a different question. Why do you want to obtain the quantiles from KDE? Why not take the empirical quantiles? If you really want to take the extreme quantiles as on the example, then the KDE approximation of the distribution won't be accurate for estimating the tails of the distribution, so the whole procedure doesn't seem to be reasonable.
Comment: The answer to this question was given in [answer to your previous question](https://stats.stackexchange.com/questions/341012/kernel-density-estimation-as-a-density/341018#341018). Densities do not sum to 1, they *integrate* to 1
Comment: I agree with @Tim. Quantile estimation is in my view best approached directly. With kernel density estimation you can't escape the need for choice of kernel shape and width -- even if the choice is made by program defaults. Such variations are especially important near $P = 0$ or $P = 1$. The method of Harrell and Davis is especially worth mentioning (reference at https://www.jstor.org/stable/2335999). There are implementations in R, Stata and presumably all other good statistical software. If there isn't one in your software, it's not good....
Comment: Yes, but How I construct the distribution starting from the density?
Comment: Is only for comparison
Answer: The cdf corresponding to the kernel density estimate $\hat f(x)$ is given by
$$
\hat F(x)=\int_{-\infty}^x \hat f(u)du = \frac1n\sum_{i=1}^n \phi(\frac{x-x_i}h)
$$
where $h$ is the bandwidth and $\phi$ is the cdf of the kernel used (e.g. the cdf of the standard normal).  So equating this to $\alpha$ and solving for $x$ (numerically using some root finding algorithm) gives you the corresponding estimate of the $\alpha$-quantile (red line below), slightly larger than the empirical quantile (blue line).
R implementation:
set.seed(1)
z<-rnorm(25)
v<-density(z)
plot(v,main="")

g <- function(x, z, bw, p) sum(pnorm(x-z, sd=bw))/length(z) - p
abline(v=uniroot(g,range(v$x)+c(-1,1),z=z,bw=v$bw,p=.95)$root,col="red") # quantile based on kernel density estimate
abline(v=quantile(z,.95),col="blue") # empirical quantile
points(z,rep(0,25))

Answer: In my programs, I am using a split approach. Within the distribution range the emperical quantiles are accurate enough usually, so I am sorting the data and calculate the quantiles from this, best with linear interpolation. I think Excel or other programs are doing so too. For extreme percentile e.g. beyond (n-1)/n and using KDE often only the last few kernels matter, so only these have pdf>0, and only these can make cdf<1. If only the last kernel matters I need only the CDF of that last kernel and the cdf (close to 1.0) will become 1-CDFkernel/n. And this is easy to invert for all practical kernels. And I do not need to deal with root finding techniques.

Quantile of kernel density estimator


Using FreeScale Code Warrior to link GCC object files with __eabi calls in main

Question: I'm linking with FreeScale Code Warrior 5.9 for MPC5554, which is a PPC chip with an e500mc core.  I'm compiling with GCC 4.6.4 compiled for powerpc-none-eabi.
I am compiling some C code for my project, including the file with main, using a GCC cross compiler.  I need to link using Code Warrior, though.  GCC adds a call to a function called __eabi at the start of main, which is supposed to initialize a couple of registers.  It cannot be removed, and is unnecessary for what I'm doing.  GCC expects it to be provided by the crt.
These are my compile flags for GCC:
-O2 -std=c99 -w -mno-eabi -mregnames -mcpu=e500mc -misel=yes -mfloat-gprs=single -mno-string -msdata=none
The documentation for GCC PowerPC says there is a -no-eabi argument, which should remove this, but then if you dig into it there are a number of bugs reports pointing out that the documentation is wrong and the flag does not actually remove the call to __eabi.
I tried simply adding an __eabi function to the crt from CodeWarrior.  In __ppc_eabi_init.c I added the function:
asm extern void __eabi(void)
{
    addis   r13,r0,_SDA_BASE_@h
    ori     r13,r13,_SDA_BASE_@l
    addis   r2,r0,_SDA2_BASE_@h
    ori     r2,r2,_SDA2_BASE_@l
    blr
}

However, when I try to link, I still get the error that __eabi from main in main.o is undefined.  It seems the Code Warrior linker can't find my function.  I verified that __ppc_eabi_init.c is first in the link order.
How can I get this to link?  Is there a way to remove the call to __eabi?  Why is Code Warrior unable to find the function when I provide it?
Comment: the codewarrior assembly language has the directive `.global` to make a label visible to other files.  So I would expect, at a minimum that the first line would be: `asm global void __eabi(void)`
Comment: @user3629249 you would think that, but it expects the `extern` keyword in this context.
Answer: It turned out that I had the right idea, but that I was accidentally putting the __eabi function definition inside an #ifdef block that was being excluded from assembly.  When I moved the definition outside this block it linked properly.
I also modified the provided linker scripts so they would correctly locate the .rodata.cst* sections that GCC uses for read-only data.  I am not sure this was important, but I was getting a warning from the linker, so I fixed it.
In MPC5554.lcf, I changed
.rodata (CONST) : {
    *(.rdata)
    *(.rodata)
}

to read
.rodata (CONST) : {
    *(.rdata)
    *(.rodata)
    *(.rodata.cst*)
}


Access-Control-Allow-Origin header missing from a response

Question: I'm trying to debug an issue.
On one of my sites, the font files are not loading correctly. Someone has reported this error in the inspector log:
Font from origin 'http://d1h0r2f9g9fk4d.cloudfront.net' has been
blocked from loading by Cross-Origin Resource Sharing policy: No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://bit.ly/1Z4W4JZ' is therefore
not allowed access.

The person who is saying he can't see the font files has also attached this screenshot of the website:

You can see that the fonts (which should be carets are not showing).
In this case, I believe it's font-awesome that isn't loading for him.

Cloudfront is the CDN
nGinx is the origin (and is correctly sending the Access-Control-Allow-Origin header)

I have performed a curl -v -I and you can see this as the response:
My working response
curl -v -I http://d1h0r2f9g9fk4d.cloudfront.net/static/release/fonts/fontawesome-webfont.ttf?v=4.3.0
* Hostname was NOT found in DNS cache
*   Trying 188.8.131.52...
* Connected to d1h0r2f9g9fk4d.cloudfront.net (18.104.22.168) port 80 (#0)
> HEAD /static/release/fonts/fontawesome-webfont.ttf?v=4.3.0 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: d1h0r2f9g9fk4d.cloudfront.net
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Connection: keep-alive
Connection: keep-alive
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
Server: nginx/1.4.6 (Ubuntu)
< Date: Wed, 06 Jan 2016 09:33:59 GMT
Date: Wed, 06 Jan 2016 09:33:59 GMT
< Last-Modified: Tue, 09 Jun 2015 10:46:31 GMT
Last-Modified: Tue, 09 Jun 2015 10:46:31 GMT
< ETag: "5576c407-1dcec"
ETag: "5576c407-1dcec"
< Expires: Thu, 04 Feb 2016 18:03:03 GMT
Expires: Thu, 04 Feb 2016 18:03:03 GMT
< Cache-Control: max-age=2592000
Cache-Control: max-age=2592000
< X-Varnish: 2146103981 2146009331
X-Varnish: 2146103981 2146009331
< Age: 55857
Age: 55857
< Via: 1.1 varnish, 1.1 f836ea1710367746c54dbe5fbb422013.cloudfront.net (CloudFront)
Via: 1.1 varnish, 1.1 f836ea1710367746c54dbe5fbb422013.cloudfront.net (CloudFront)
< X-Hashed-On: /static/release/fonts/fontawesome-webfont.ttf?v=4.3.0*cdn.rentivo.com
X-Hashed-On: /static/release/fonts/fontawesome-webfont.ttf?v=4.3.0*cdn.rentivo.com
< X-Discovery: not-set
X-Discovery: not-set
< X-Cache-Lookup: lookup
X-Cache-Lookup: lookup
< X-Cachable: 1
X-Cachable: 1
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< X-Cache: Miss from cloudfront
X-Cache: Miss from cloudfront
< X-Amz-Cf-Id: UlVhI7Nix19cnSqakrZ3dqVta9ROM8thQ9c0rixacW-dZpC9wCCe4Q==
X-Amz-Cf-Id: UlVhI7Nix19cnSqakrZ3dqVta9ROM8thQ9c0rixacW-dZpC9wCCe4Q==

<
* Connection #0 to host d1h0r2f9g9fk4d.cloudfront.net left intact

You can clearly see that  Access-Control-Allow-Origin: * is present in the font headers.
I have tried invalidating the Cloudfront distribution, thinking, perhaps the edge server has an old version which did not have these headers, but the person is still saying that he's been unable to view the headers.
I asked him to perform a curl log for me and this was his response.
His failed response
curl -v -I http://d1h0r2f9g9fk4d.cloudfront.net/static/release/fonts/fontawesome-webfont.ttf?v=4.3.0
*   Trying 188.8.131.52...
* Connected to d1h0r2f9g9fk4d.cloudfront.net (126.96.36.199) port 80 (#0)
> HEAD /static/release/fonts/fontawesome-webfont.ttf?v=4.3.0 HTTP/1.1
> Host: d1h0r2f9g9fk4d.cloudfront.net
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Encoding: gzip
Content-Encoding: gzip
< Content-Length: 71646
Content-Length: 71646
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< ETag: "5576c407-1dcec"
ETag: "5576c407-1dcec"
< Server: nginx/1.4.6 (Ubuntu)
Server: nginx/1.4.6 (Ubuntu)
< Expires: Wed, 03 Feb 2016 10:08:00 GMT
Expires: Wed, 03 Feb 2016 10:08:00 GMT
< Last-Modified: Tue, 09 Jun 2015 10:46:31 GMT
Last-Modified: Tue, 09 Jun 2015 10:46:31 GMT
< Connection: keep-alive
Connection: keep-alive
< Date: Wed, 06 Jan 2016 01:04:10 GMT
Date: Wed, 06 Jan 2016 01:04:10 GMT

<
* Connection #0 to host d1h0r2f9g9fk4d.cloudfront.net left intact

I CANNOT understand what's happening. Could his ISP be performing some ridiculous snooping/optimization on him? If you notice, the header responses don't even match up. You would at the least expect to see
< X-Cache: Miss from cloudfront
X-Cache: Miss from cloudfront
< X-Amz-Cf-Id: UlVhI7Nix19cnSqakrZ3dqVta9ROM8thQ9c0rixacW-dZpC9wCCe4Q==
X-Amz-Cf-Id: UlVhI7Nix19cnSqakrZ3dqVta9ROM8thQ9c0rixacW-dZpC9wCCe4Q==

These are all missing.
Does anyone have any insight?
Comment: Something definitely does not add up. A cached response from CF should also have an `Age:` header.  What if they request from CF over https? `*.cloudfront.net` (as you probably know) has a wildcard cert.  HTTPS should stop naive ISP snooping/tweaking, which is what I am inclined to suspect, here.  My next step would be enabling CloudFront logs, if not already done, so you can confirm whether each request actually arrives at CF from the user.
Comment: Although I am not inclined to suspect a bug in CloudFront, the presence of `Content-Encoding: gzip` in one response but not the other prompts me to ask whether you have enabled the new [on-the-fly gzip compression feature](https://aws.amazon.com/blogs/aws/new-gzip-compression-support-for-amazon-cloudfront/) in CloudFront.  While unlikely, this needs to be checked.  It's also possible the the far-end ISP is doing gzipping and/or caching, and their implementation is broken.
Answer: Late response but stumbled across this thread while struggling with the same problem.
Try adding an Origin header to your curl request; this fixed it for me - something like:
curl -v -I "http://d1h0r2f9g9fk4d.cloudfront.net/static/release/fonts/fontawesome-webfont.ttf?v=4.3.0" -H "Origin: https://example.com"


Why interrupt is stopping program instead of jumping back?

Question: i'm working on arduino and i have made two modes in loop(),mode1 in while loop and mode second in while loop they have conditions to be met, i switch between them using button that interrupts routine that is currently executed,changes flags to force program to change to second mode and for some reason instead of breaking out of current while loop (since conditoins are not met anymore )to change to the other it just doesnt respond
or changes from automatic->stops responding
#define modeSwitch 2
boolean manual;
boolean automatic;
String sOld;
boolean flagPrint = false;
volatile int modeSwitchValue;
String receivedData = "";
String invitation = "Welcome to Morse Code Encoder!\nPlease choose a mode: Write 'automatic' for an automatic mode or 'manual' or manual code.\n";

void setup() {
  pinMode(modeSwitch, INPUT);
  attachInterrupt(0, changeMode, RISING);
  manual = false;
  automatic = false;
  volatile int modeSwitchValue = 5;
  Serial.begin(9600);
  Serial.println(invitation);
}

void loop() {
  if (Serial.available() > 0) {
    receivedData = Serial.readStringUntil('\n');
    if (receivedData == "manual" ) {
      manual = true;
      automatic = false;
      modeSwitchValue = 0;
    }
    if ( receivedData == "automatic") {
      automatic = true;
      manual = false;
      modeSwitchValue = 1;
    }

    while ((manual == true) && (modeSwitchValue == 0) ) {
      String s = "Manual mode:";
      while (checkPrint(s) == true) {
        Serial.println(s);
      }

      automatic = false;
    }

    while ((automatic == true) && (modeSwitchValue == 1)) {
      String s = "Automatic mode:";
      while (checkPrint(s) == true) {
        Serial.println(s);
      }
      manual = false;
    }

  }

}

void changeMode() {

  if (  modeSwitchValue == 0) {
    modeSwitchValue = 1;
    automatic = true;
    manual = false;

  }
  if ( modeSwitchValue == 1) {
    modeSwitchValue = 0;
    manual = true;
    automatic = false;

  }
}

I deleted checkPrint() function cause it only ensures msgs are printed once so its not essential to show here i think
I thought that while executing automatic code, i press button and it interrupts,changing mode variable then goes back to same place in program where the interrupt occured. then software notices that while loop condition of automatic mode is not fulfilled anymore so it breaks out to loop() and finds while loop of manual mode to be executed since the conditions are met. am i thinking wrong? or have i missed something?
Comment: What do you expect `buttonState == LOW;` and `buttonState == HIGH;` to do?
Comment: to remember in which mode we currently are and to which it would be changed next
Answer: If you really want to attach an ISR to the RISING edge of a button signal, you should ignore multiple triggers. Minimum 2 msec, but why not ignore everything that's unrealistic (e.g. 100 msec)
As you just toggle two states, I'd propose a boolean variable istate.
volatile bool istate; // the state maintained in the ISR changeMode

void changeMode() {
  static unsigned long lastime;
  if (millis() - lastime > 100) {
     lastime = millis();
     istate = ! istate;
  }
}

All other redundant stuff (automatic, manual) can be built from that volatile ISR variable in the main code, if necessary.
In my unterstanding, the name buttonState is just wrong. And the variable unnecessary.

Do you know the data type enum?
enum {MANUAL, AUTO} mode;
 if (istate) mode = AUTO;
 else        mode = MANUAL;

might be nice for your purpose.
Comment: I would not use 'auto' since it is a keyword in c++.
Comment: @tddguru: agreed. Changed sample above.
Answer: You are mixing up assignment "=" and comparison "==" in your changeMode() function.  Here is a corrected version of the function.
void changeMode() {
  if ( buttonState == HIGH) {
    modeSwitchValue = 1;
    automatic = true;
    manual = false;
    buttonState = LOW;
  }

  if (buttonState == LOW) {
    modeSwitchValue = 0;
    manual = true;
    automatic = false;
    buttonState = HIGH;
  }
}

You should also realize that mechanical buttons can have contact bounce.  Since you are using an interrupt to detect changes in the button state you might get erratic behavior.  Usually, it is better to poll the button state and apply a de-bounce time delay on the order of 20ms.
You also have not shared how the button is wired.  Since you are detecting a rising edge, I assume that the button is wired to pull the pin high when pressed. If you are using a single-pole single-though button, you will need a pull down resistor to ground, otherwise the interrupt pin will float.
You can avoid the required pull-up or pull down resistor, by connecting a simple button between the input pin and ground.  Then change your initialization code to
pinMode(pin, INPUT_PULLUP)

You will then want to trigger interrupt on the falling edge.
I hope this helps.
Comment: oops what a fail -.-" i cant believe i didnt notice that earlier, thank you, i'll check the debounce problem also
Comment: I changed that however that button doesnt seem to respond at all to being pushed
Comment: Due to bouncing, interrupt and buttonpresses are a very bad combination.


Optimization of query/calling function

Question: I am getting users from database and returning them as array. Query returns 22000 users, which are correct, now i am trying to save these users info CSV file, but before this i have to check extra features of each user e.g Applied Opportunities, so checking each single user , getting each single ID from array and passing it to the other function takes lot and lot of time, and i have 9 other functions that i need to check on. What is the best and optimized way to do this. Please see the code below
<code>$sql = "SELECT id, first_name,last_name,position_id,user_company,role,last_login_date,
sf_id,user_email,login_count,user_registered FROM er_users";
$query = $this->db->query($sql);
$query = $query->result_array();

foreach ($query as $key => $q) {
   $a_opps = vt_applied_opps((object)$q);
   $s_opps = vt_suggested_opps((object)$q);
   $i_requests_e = vt_interview_request((object)$q);
   $i_requests_c = vt_company_interview_request((object)$q);

   $i_to_apply = get_invitations_count($q["id"]);
   $councils = vt_ac_count((object)$q);
   $cbe = vt_cbe_credits((object)$q);
   $forum = vt_forums((object)$q);
   $bio = vt_is_bio((object)$q);
   $resume = vt_is_resume((object)$q);

   $q["a_opps"] = $a_opps;
   $q["s_opps"] = $s_opps;
   $q["i_requests_e"] = $i_requests_e;
   $q["i_requests_c"] = $i_requests_c;

   $q["i_to_apply"] = $i_to_apply;
   $q["councils"] = $councils;
   $q["cbe"] = $cbe;
   $q["forum"] = $forum;
   $q["bio"] = $bio;
   $q["resume"] = $resume;
   $query[$key] = $q;
}
</code>
These are the functions, and their results is being stored in variable and then those variables are being used in an array. Please help me to optimize this method. As there are 22K records, so foreach loop is running 22K times and inner functions are 22K times as well, please help me to optimize this way.
Comment: Do the functions inside the foreach contain queries?
Comment: The best way - get all additional data with one sql query. For example instead calling vt_applied_opps((object)$q) for each user, make single sql query that return data for all users.
Comment: Can these functions be moved to the database? Perhaps a stored procedure.
Comment: Is the information from these other functions stored within the database? If so then why not use subqueries to gather all of the information in a single pass? https://dev.mysql.com/doc/refman/5.7/en/subqueries.html
Answer:
$sql = "SELECT id,
  first_name,last_name,position_id,user_company,role,last_login_date,
  sf_id,user_email,login_count,user_registered FROM er_users";
  If you only need id then remove remaining table field from SELECT clause.

Second,

$a_opps = vt_applied_opps((object)$q);    $s_opps =
  vt_suggested_opps((object)$q);    $i_requests_e =
  vt_interview_request((object)$q);    $i_requests_c =
  vt_company_interview_request((object)$q);
$i_to_apply = get_invitations_count($q["id"]);    $councils =
  vt_ac_count((object)$q);    $cbe = vt_cbe_credits((object)$q);
  $forum = vt_forums((object)$q);    $bio = vt_is_bio((object)$q);
  $resume = vt_is_resume((object)$q);

If these all function getting the appropriate results from the database only , then you can JOIN those tables with the above SQL, and ignore the function call.
These really might help.
Also check the Table.field indexing, if you are using the fields in WHERE clause, and those fields does not have indexing, then please add indexing for those fields make fast SELECT SQL.
Comment: i cannot use joins as because the records in database are above 100K so they will become more slow ....
Comment: JOIN will become slow, but if you will use indexing, it will become fast.

Optimization of query/calling function


Can we keep only X number of files rotated by Y MB with Apache rotatelogs

Question: Can we rotate Apache logs keeping n number of backups rotated by size.
We can rotate apache logs with size by doing following:
ErrorLog "|bin/rotatelogs.exe  logs/error_log-%Y-%m-%d.log 20M"
How I can keep only 5 backups always? What I mean is files should be always when it reaches to 20 MB and it should always keep 5 backups.
I am using Apache 2.4.20 on windows.
Answer:
rotatelogs - Piped logging program to rotate Apache logs
Synopsis
rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -D ] [ -t ]
  [ -v ] [ -e ] [ -c ] [
  -n number-of-files ] logfile rotationtime|filesize(B|K|M|G) [ offset ]
Options

-n number-of-files
Use a circular list of filenames without timestamps. With -n 3, the
  series of log files opened would be logfile, logfile.1,
  logfile.2, then overwriting logfile. Available in 2.4.5 and later.

With -n you'd remove the date from the logfile name:
ErrorLog "|bin/rotatelogs.exe -n 5 logs/error_log.log 20M"


Calculate $A^5 - 27A^3 + 65A^2$, where $A$ is the matrix defined below.

Question: If $A=\begin{bmatrix} 0 & 0 & 1 \\
3 & 1 & 0 \\
-2&1&4\end{bmatrix}$, find $A^5 - 27A^3 + 65A^2$
$$A=\begin{bmatrix} 0 & 0 & 1 \\
3 & 1 & 0 \\
-2&1&4\end{bmatrix}$$
Let $\lambda$ be its eigenvalue, then
$$(A-\lambda I) = \begin{bmatrix} 0-\lambda & 0 & 1 \\
3 & 1-\lambda & 0 \\
-2&1&4-\lambda\end{bmatrix}$$
$$|A-\lambda I| = -(\lambda)^3 + 5(\lambda)^2 - 6(\lambda) +5$$
Using Cayley-Hamilton theorem
$$A^3-5^2+6A-5=0$$
How do I use this find $A^5 - 27A^3 + 65A^2$?
Comment: By Cayley-Hamilton, it is actually $A^3-5A^2+6A-5I=O$.
Answer: To add to the above answer, using  $A^3 = 5A^2 - 6A + 5$ to simplify further $A^2(5A^2 -33A + 70) = A(5(5A^2 - 6A + 5) - 33A^2 + 70A) = A(-8A^2 + 40A + 25) = -8(5A^2 -33A + 70) + 40A^2 + 25A = 289A - 560$
Comment: small typo at the end: you must plug in $A^3$.
Answer: I don't see a way to find the answer directly but you can certainly simplify your calculations; since $A$ commutes with itself we may factor the polynomial however we want. Write
$$
A^5-27A^3+65A^2=A^2(A^3-27A+65)=A^2\left[(A^3-5A^2+6A-5)+(5A^2-33A+70)\right].
$$
Now by Cayley Hamilton you only need to compute $A^2(5A^2-33A+70)$, which cuts down on the powers of $A$ that you need to calculate directly and reduces the problem to a bunch of addition and two matrix multiplications.
Answer: Doing long division:
$$A^5-27A^3+65A^2=(A^2+5A-8)(\underbrace{A^3-5A^2+6A-5}_{=0})+73A-40.$$
Hence it is:
$$A^5-27A^3+65A^2=73A-40I=\begin{pmatrix}-40&0&73\\ 219&33&0\\-146&73&252\end{pmatrix}.$$
Answer: Let
$p(x) =x^5 -27x^3 + 65x^2; \tag 1$
one can always compute $p(A)$ directly, but perhaps the following requires less arithmetical work:
Let
$\chi(x) = x^3 - 5x^2 +6x - 5 \tag 2$
be the characteristic polynomial of $A$.
By the division algorithm for polynomials, there exist unique
$q(x), r(x) \in \Bbb Q[x] \tag 3$
such that
$p(x) = q(x)\chi(x) + r(x), \tag 4$
with either $r(x) = 0$ or $\deg r(x) < 3 = \deg \chi(x)$; thus, since
$\chi(A) = 0, \tag 5$
we have from (4)
$r(A) = p(A) - \chi(A)q(A) = p(A) - 0 \cdot q(A) = p(A); \tag 6$
thus, we compute $r(x)$ by synthetic (long) division of polynomials; it's not too much work; we find
$q(x) = x^2 + 5x - 8, \tag 7$
and
$r(x) = 73x - 40; \tag 8$
the reader may easily check that
$p(x) = x^5 - 27x^3 + 65x^2$
$= (x^2 + 5x - 8)(x^3 - 5x^2 +6x - 5) + 73 x - 40 = q(x)\chi(x) + r(x); \tag 9$
thus
$p(A) = r(A) = 73A - 40I = 73 \begin{bmatrix} 0 & 0 & 1 \\ 3 & 1 & 0 \\ -2 & 1 & 4 \end{bmatrix} - 40I = \begin{bmatrix} -40 & 0 & 73 \\ 219 & 33 & 0 \\ -146 & 73 & 252 \end{bmatrix}. \tag{10}$


Different blocks on the same page, dependent on source link?

Question: Is it possible to have a page that shows (e.g.) one of three different blocks, depending on the originating page link? I'm thinking the only way to achieve this would be to use # or something similar at the end of the URL.
Example:
<code>link one: mysite.com/mypage#one
link two: mysite.com/mypage#two
link three: mysite.com/mypage#three
</code>
And then create the three blocks, with the visibility settings set appropriately.
Is this a robust solution that would work with URL aliases etc.?
Cheers,
James
Answer: If you do it programmatically you can always parse url and display different content for different anchor or other param. You can always use one block for display content according to conditions.
Take a look to Context or Rules modules, maybe you find easier solution


Mysql LIKE clause and separate words in a field

Question: I currently use a mysql statement like the one below to search post titles.
select * from table where title like %search_term%
But problem is, if the title were like: Acme launches 5 pound burger and a user searched for Acme, it'll return a result. But if a user searched for Acme burger or Acme 5 pound, it'll return nothing.
Is there a way to get it to return results when a users searches for more than one word? Is LIKE the correct thing to use here or is there something else that can be used?
Comment: How about matching against a full text index?
Answer: To search for a string against a text collection use MATCH() and AGAINST()
SELECT * FROM table WHERE MATCH(title) AGAINST('+Acme burger*')

or why not RLIKE
SELECT * FROM table WHERE TITLE RLIKE 'Acme|burger'

or LIKE searching an array, to have a compilation of $keys
$keys=array('Acme','burger','pound');
$mysql = array('0');
foreach($keys as $key){
    $mysql[] = 'title LIKE %'.$key.'%'
}
SELECT * FROM table WHERE '.implode(" OR ", $mysql)

Comment: you will need a full text index though.
Comment: @nl-x What do you mean by full text index? The column type should be text? Right now its varchar.
Comment: I tried your MATCH solution. But that'll return everything with the word acme even when 'acme burger' is searched for. But that match is a good one.
Answer: You could use a REGEXP to match any of the words in your search string:
select *
from tbl
where
  title REGEXP CONCAT('[[:<:]](', REPLACE('Acme burger', ' ', '|'), ')[[:>:]]')

Please notice that this will not be very efficient. See fiddle here.
If you need to match every word in your string, you could use a query like this:
select *
from tbl
where
  title REGEXP CONCAT('[[:<:]]', REPLACE('Acme burger', ' ', '[[:>:]].*[[:<:]]'), '[[:>:]]')

Fiddle here. But words have to be in the correct order (es. 'Acme burger' will match, 'burger Acme' won't). There's a REGEXP to match every word in any order, but it is not supported by MySql, unless you install an UDF that supports Perl regexp.
Comment: This too returns every title with the word acme in it when you search for acme burgers.
Comment: @Norman updated my answer. But if you need to match every word, in any order, there's a regexp but it is not natively supported by MySql...so another solution is needed
Answer: What you need to do is construct a SQL such that, for example:
select * from table where title like "%Acme%" and title like "%burger%"

In short: split the string and create one like for each part.
It might also work with replacing spaces with %, but I'm not sure about that.
Answer: The best thing is thing use perform union operation by splitting your search string based on whitespaces,
FOR Acme 5 pound,
SELECT * FROM TABLE WHERE TITLE LIKE '%ACME 5 POUND%'
UNION
SELECT * FROM TABLE WHERE TITLE LIKE '%ACME%'
UNION
SELECT * FROM TABLE WHERE TITLE LIKE '%5%'
UNION
SELECT * FROM TABLE WHERE TITLE LIKE '%POUND%'

Find out a way to give the first query a priority. Or pass the above one as four separate queries with some priority. I think you are using front end tp pass query to data bases, so it should be easy for you.
Comment: Why not use `OR` instead of `UNION`?
Comment: yeah you can do that. But I don't know which one will be efficient.
Answer: <?php
$search_term = 'test1 test2 test3';
$keywords = explode(" ", preg_replace("/\s+/", " ", $search_term));
foreach($keywords as $keyword){
 $wherelike[] = "title LIKE '%$keyword%' ";
}
$where = implode(" and ", $wherelike);

$query = "select * from table where $where";

echo $query;
//select * from table where title LIKE '%test1%' and title LIKE '%test2%' and title LIKE '%test3%'

Comment: Is there a way to do this in MySQL?