iOS data sharing to Google Apps for social features: Workable?

Question: I'm putting on my thinking cap for this one, looking for a high level overview of possibility.  I'm the author of an iPhone app that tracks user generated statistics.  Data is essentially stored in a table on the device, with each entry having several fields/columns.  Users can then sort that data, view graphs, and do other nerdy number crunching stuff.  I want to take it farther and incorporate a sharing platform online.  Game Center, Open Feint and other third party platforms are too narrow in scope.  I'm interested in writing a web app, that users can visit to do three things:

Post New Table entries (automated by device)
View own entries and share via web (read only)
View other members table entries (read only)

Is Google App Engine a viable solution for this?  My iOS app will require a POST URL, and the web app will need to save that authenticated user's data, and possibly return an "identifier" value to be referenced in the future in case the user needs to modify the item on the web.  It will also require a GET URL to retrieve the authenticated user's statistics one by one or in total.
Next, the user will be able to visit the website and type in a "username" (probably email address) and see read-only statistics that have been submitted.
Thank you in advance for your input.
Answer: Sure, you could do all that with App Engine. If you wanted to, you could even make that local table of data sync with your app on GAE.
Really, there's not much on GAE that you can't do, although sometimes the database layer take some time to get used to.
My personal preference would be to set up a Django instance, but you could use any Python or Java-based app, depending on what you're comfortable with.
In short, yeah, you won't have any trouble doing basic REST work with GAE, and you could probably push it harder to do some more number crunching on those stats.


Canonical bundle of $S^1$ is trivial

Question: I have to show that the canonical bundle of $S^1$ is trivial. The canonical bundle of $S^1=T^*S^1$ namely is equal to the cotangent bundle. But how I can show that this cotangent bundle is trivial? I have to show that it is isomorphic at a product like $(S^1)^*\times \mathbb{R}$...but i think isn't correct...please help!!
Comment: It's enough to show that $TS^1$ is trivial. Can you draw a nowhere-vanishing vector field on the circle?
Comment: @AsalBeagDubh in my definition a vector bundle is trivial if it is isomorphic as vecor bundle to a product, in this case, $S^1 \times \mathbb{R}$. I know that the vector field $v=x \frac{\partial}{\partial y}-y \frac{\partial}{\partial }$ is a nowhere-vanishing vector field on the circle...
Comment: Right! Now it's a straightforward lemma that a vector bundle of rank $r$ is trivial iff it has $r$ global sections which are linearly independent at each point. Apply that in your case...
Comment: By the way, I don't know what the notation $(S^1)^*$ means in your question. It seems like a typo.
Comment: @AsalBeagDubh sorry but I don't see the connection. You may say that a vector bundle is a section of a tangent bundle? And if it is the case how can i show that these section are linearly independent?
Comment: Exactly: I say that a vector field is a section of the tangent bundle. (If you don't like that you can write a nowhere-zero section of the cotangent bundle instead.) What does it mean for a vector in a 1-dimensional vector space to be linearly independent?
Comment: @AsalBeagDubh that it is not the zero vector...
Comment: @AsalBeagDubh Ok. I have understood what you say. Only one more things...what is your definition of global section? where I can find a proof of the proposition that you have written before?
Comment: See Milnor--Stasheff, *Characteristic Classes*.
Answer: It will suffice to show that $TS^1$ is trivial, since the dual of a trivial bundle is trivial. That $TS^1$ is trivial follows from the general fact that a Lie group $G$ (such as $S^1$) has a trivial tangent bundle. Let $\mathfrak{g}$ be the Lie algebra of $G$. An explicit trivialization of $TG$ is $$\varphi:G\times\mathfrak{g}\rightarrow TG$$ $$(g,\xi)\mapsto (g,d_eL_g(\xi)),$$ where $d_eL_g:\mathfrak{g}\rightarrow T_gG$ is the differential of left multiplication by $g$ at the identity $e$.

Canonical bundle of $S^1$ is trivial


Reference request for texts that give examples of categories and some of their properties

Question: I am looking for texts that give examples of categories and some of their distinguishing properties. There could be a section on the category of sets and its properties, the category of groups and its properties, the category of finite sets and its properties, the category of topological spaces and its properties, etc. I would love to know if there is such a text out there.
Answer: One could make the argument that all category texts by necessity have lots of examples and discuss their properties, but I'm interpreting this question specifically as a request for texts that do a particularly good job of making it easy to see a large body of examples and their properties.
My recollection of a really large indexing of categories is the one in the back of

Herrlich, Horst, and George E. Strecker. Abstract and concrete categories: the joy of cats. Wiley, 1990.

There must be at least 100 categories named, and they all have a row full of references to places earlier in the text, no doubt describing various aspects of them.
Comment: This sort of thing would be a good idea for a database-type site of examples.  I have such a site [for rings](https://ringtheory.herokuapp.com/). It would be easy to [adapt the code](https://github.com/rschwiebert/RingApp) to do a site for categories, and I would be happy to help someone to do so, if they're willing to find somewhere to host such a site.


CALayer transform doesn't apply if NSView isn't created yet at the moment

Question: Environment: OS X 10.11 SDK, XCode 7.2, Cocoa;
Context: Some complex transformation should be applied in the moment of views creation;
When I'm trying to add NSView programmatically transform property is simply ignored if it's placed before addSubview call:
- (void)applicationDidFinishLaunching:(NSNotification *)aNotification {
   NSView *view = [[NSView alloc] initWithFrame:NSMakeRect(100, 100, 100, 100)];
    CALayer* newLayer = [CALayer layer];

    newLayer.transform = CATransform3DMakeRotation(angle, 0.1f, 0.1f, 0.1f);

    [newLayer setBackgroundColor:[[NSColor greenColor] CGColor]];

    [view setLayer:newLayer];
    [view setWantsLayer:YES];
    [_window.contentView addSubview:view];
}

- (IBAction)click:(id)sender {
    angle += 0.1f;
    [[_window.contentView subviews] objectAtIndex:2].layer.transform = CATransform3DMakeRotation(angle, 0.1f, 0.1f, 0.1f);
    CATransform3D t = [[_window.contentView subviews] objectAtIndex:2].layer.transform;
    [[_window.contentView subviews] objectAtIndex:2].layer.edgeAntialiasingMask = !CATransform3DIsIdentity(t);
}

If I click on the button, the view would rotate as it should. Also, applying transformation after adding the view to the window works too:
- (void)applicationDidFinishLaunching:(NSNotification *)aNotification {
    NSView *view = [[NSView alloc] initWithFrame:NSMakeRect(100, 100, 100, 100)];
    CALayer* newLayer = [CALayer layer];

    [newLayer setBackgroundColor:[[NSColor greenColor] CGColor]];

    [view setLayer:newLayer];
    [view setWantsLayer:YES];

    [_window.contentView addSubview:view];
    view.layer.transform = CATransform3DMakeRotation(angle, 0.1f, 0.1f, 0.1f);

}

Do I miss some requirements or just misunderstood how CALayer transform works?
Comment: Is there a question in here somewhere, or are you attempting to file a bug report by posting on SO?
Comment: @bhaller Added the question. I don't think that's a bug, just a lack of knowledge.
Comment: If OS X layers work the way iOS layers work, this is totally expected behavior.
Comment: @matt, maybe I was unclear. Just created project with the same principe: ``` UIView *view = [[UIView alloc] initWithFrame:CGRectMake(100, 100, 100, 100)];
    view.layer.transform = CATransform3DMakeRotation(0.5f, 0.1f, 0.1f, 0.1f);
    [view.layer setBackgroundColor:[[UIColor greenColor] CGColor]];
    [self.window.rootViewController.view addSubview:view];
``` And it works perfectly (transformation has applied even before the view added to UIWindow.
Comment: Hmm, yes, I see what you mean. But a view "has" a layer in a very different sense in Mac OS X. iOS views are born with a layer (they can't exist without one), but not every Mac OS X view has one; you have to ask for it (as you did). It's more of a second-class citizen.
Answer: A solution I've come up with is not an attractive one but solve the problem. I just postpone the transformation to the moment of the view initial rendering.
@interface DebuggableView: NSView
    @property (nonatomic, assign) CATransform3D transform;
    @property (nonatomic, assign) bool shouldBeTransformed;
@end;

@implementation DebuggableView

- (instancetype) initWithFrame:(NSRect)frameRect
{
    self = [super initWithFrame:frameRect];
    if (self)
    {
        self.wantsLayer = YES;
    }
    return self;
}

- (void)layout
{
    [super layout];
    if (self.shouldBeTransformed) {
        self.layer.transform = self.transform;
        self.shouldBeTransformed = NO;
    }
}

@end


Why does my training loss have regular spikes?

Question: I'm training the Keras object detection model linked at the bottom of this question, although I believe my problem has to do neither with Keras nor with the specific model I'm trying to train (SSD), but rather with the way the data is passed to the model during training.
Here is my problem (see image below):
My training loss is decreasing overall, but it shows sharp regular spikes:

The unit on the x-axis is not training epochs, but tens of training steps. The spikes occur precisely once every 1390 training steps, which is exactly the number of training steps for one full pass over my training dataset.
The fact that the spikes always occur after each full pass over the training dataset makes me suspect that the problem is not with the model itself, but with the data it is being fed during the training.
I'm using the batch generator provided in the repository to generate batches during training. I checked the source code of the generator and it does shuffle the training dataset before each pass using sklearn.utils.shuffle.
I'm confused for two reasons:

The training dataset is being shuffled before each pass.
As you can see in this Jupyter notebook, I'm using the generator's ad-hoc data augmentation features, so the dataset should theoretically never be same for any pass: All the augmentations are random.

I made some test predictions to see if the model is actually learning anything, and it is! The predictions get better over time, but of course the model is learning very slowly since those spikes seem to mess up the gradient every 1390 steps.
Any hints as to what this might be are greatly appreciated! I'm using the exact same Jupyter notebook that is linked above for my training, the only variable I changed is the batch size from 32 to 16. Other than that, the linked notebook contains the exact training process I'm following.
Here is a link to the repository that contains the model:
https://github.com/pierluigiferrari/ssd_keras
Comment: @djk47463 I agree it's hardly a compact example, but how do you create a compact example if you have a complex object detection model and the problem could lie in any part of the model? Anyways, I've solved it myself, it was a Keras-specific issue after all. Maybe this will be useful for someone at some point.
Comment: This is hardly a [mcve], I think this question could go on a nice diet, which will increase the probability of getting an answer :)
Answer: I've figured it out myself:
TL;DR:
Make sure your loss magnitude is independent of your mini-batch size.
The long explanation:
In my case the issue was Keras-specific after all.
Maybe the solution to this problem will be useful for someone at some point.
It turns out that Keras divides the loss by the mini-batch size. The important thing to understand here is that it's not the loss function itself that averages over the batch size, but rather the averaging happens somewhere else in the training process.
Why does this matter?
The model I am training, SSD, uses a rather complicated multi-task loss function that does its own averaging (not by the batch size, but by the number of ground truth bounding boxes in the batch). Now if the loss function already divides the loss by some number that is correlated with the batch size, and afterwards Keras divides by the batch size a second time, then all of a sudden the magnitude of the loss value starts to depend on the batch size (to be precise, it becomes inversely proportional to the batch size).
Now usually the number of samples in your dataset is not an integer multiple of the batch size you choose, so the very last mini-batch of an epoch (here I implicitly define an epoch as one full pass over the dataset) will end up containing fewer samples than the batch size. This is what messes up the magnitude of the loss if it depends on the batch size, and in turn messes up the magnitude of gradient. Since I'm using an optimizer with momentum, that messed up gradient continues influencing the gradients of a few subsequent training steps, too.
Once I adjusted the loss function by multiplying the loss by the batch size (thus reverting Keras' subsequent division by the batch size), everything was fine: No more spikes in the loss.
Comment: Usually it is safer to just drop the last batch in such cases. Even if the loss is independent of the batch size, a very small batch is more likely to contain non-representative data and hence mess up the gradients. When using mini-batch gradient descent the loss *landscape* is not fixed but it's changing with every batch. This can cause typical fluctuations in the loss if the batch size is too small. Only if the batch size is large enough to be representative of the whole data set the loss will stabilize. A small(er) batch at the end of the epoch can thus have a similarly negative effect.
Comment: @a_guest I'm not sure I agree, for three reasons. 1) Any typically used mini batch sizes are so small relative to the overall dataset that the loss manifold fluctuates extremely between mini batches no matter what. The mini batch does not need to be very representative of the whole dataset. 2) Any optimizer that uses momentum or a similar mechanism doesn't depend much on individual batches anyway. Only the average of many batches matters. 3) From an empirical perspective, training with mini batch size 1 or very small batch sizes works very well in practice.
Comment: To elaborate on the first argument: Whether the last batch has 32 or 7 samples, both is tiny relative to, and therefore neither will be representative of, your dataset of 50k samples (or 500k, or 5 million).
Comment: I am struggling with a very similar issue, but my data size is an integer multiple of the batch size, so the explanation does not hold for me. I am unable to explain the sharp spikes and the steep drops in training loss at the end of each epoch.
Answer: For anyone working in PyTorch, an easy solution which solves this specific problem is to specify in the DataLoader to drop the last batch:
train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=False,
                                          pin_memory=(torch.cuda.is_available()),
                                          num_workers=num_workers, drop_last=True)

Comment: Sure, this gets the job done, but it's also just a crutch that treats the symptom rather than the cause. The point is that the loss magnitude simply should not depend on the mini batch size, because that makes no sense. And if you get this right, then there is no need to drop the last mini batch or worry about anything else.
Answer: I would add gradient clipping because this prevents spikes in the gradients to mess up the parameters during training.
Gradient Clipping is a technique to prevent exploding gradients in very deep networks, typically Recurrent Neural Networks.
Most programs allows you to add a gradient clipping parameter to your GD based optimizer.
Comment: Gradient clipping might get the job done, but I'd argue it would not be a great idea in this case, because it would treat the symptom rather than the cause (the loss shouldn't be exploding in the first place). Besides, the solution has already been provided: In most cases like mine above, the problem will be that the loss magnitude depends on the batch size, which it shouldn't.
Answer: For me, the solution and cause of the issue were a bit different.
The problem would still occur when defining the batch size to 1 or a proper divisor of the total number of samples (so that the last batch would still be full).
TL;DR;
This is a metric artifact due to averaging that doesn't affect training. The default loss metric is the average loss over the entire current epoch, up to the point where you read it. You can see the per-batch loss by using
def on_batch_begin(batch, logs):
    model.reset_metrics()
    return

lambda_callback = tf.keras.callbacks.LambdaCallback(on_batch_begin=on_batch_begin)

and passing it when training with
model.fit(..., callbacks=[lambda_callback])

Note that this will obviously make all metrics report only the last training batch loss for each epoch.
Explanation
I managed to pinpoint the problem once I noticed this reset_states() call in the model.fit() method. I also realized that it wouldn't happen over each batch in the batch loop code or train_step, so I went to investigate further.
Looking at the LossesContainer class, it's possible to see that the _total_loss_mean is accumulated with in the __call__() function, and weighted by the batch_size as @Alex mentioned. This value is later then reduced to a mean value (as it's a Mean instance) which finally renders the loss metric we see.
The good news is that _total_loss_mean isn't actually fed into the training process, as it's not the value returned by the LossesContainer's __call__().
The image below shows a training experiment with (gray) and without (pink) the 'fix'. It was done on top of the MNIST Tensorflow example. In it, it' possible to see the averaging effect over the actual data.
Answer: I faced a similar problem when using tensorflow. In my case, the issue was not related to mini batch size. The actual problem was that tensorflow wasn't shuffling the data completely due to limitation set by buffer_size as described here https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle
For perfect shuffling, the buffer_size should be greater than or equal to the full size of the dataset.


How to setup Airflow > 2.0 high availability cluster on centos 7 or above

Question: I want to setup HA for airflow(2.3.1) on centos7. Messaging queue - Rabbitmq and metadata db - postgres. Anybody knows how to setup it.
Answer: Your question is very large, because the high availability has multiple level and definition:

Airflow availability: multiple scheduler, multiple workers, auto scaling to avoid pressure, high storage volume, ...
The databases: a HA cluster for Rabbitmq and a HA cluster for postgres
Even if you have the first two levels, how many node you want to use? you cannot put everything in the same node, you need to run one service replica per node
Suppose you did that, and now you have 3 different nodes running in the same data center, what if there is a fire in the data center? So you need to use multiple nodes in different regions
After doing all of above, is there a risk for network problem? of course there is

If you just want to run airflow in HA mode, you have multiple option to do that on any OS:

docker compose: usually we use it for developing, but you can use it for production too, you can create multiple scheduler instances, with multiple workers, it can help you to improve the availability of your service
docker swarm: similar to docker compose with additional features (scaling, multi nodes, ...), you will not find much resources to install it, but you can use the compose files and just do some changes
kubernetes: the best solution, K8S can help you to ensure the availability of your services, easy install with helm
or just running the different services on your host: not recommended, because of manual tasks, and applying the HA is complicated


Hook_form_alter in Views header PHP code

Question: I have a content type called 'Checklist' which I bring forward using Views, in it's node-edit form.
I wrote PHP code within the header in order to do this, here it is:
<code><?php

    $view = views_get_current_view();
    $arg0 = $view->args[0];

    $nodeToLoad = $view->result[0]->nid; //I get the node to load from Fields within my view
    module_load_include('inc', 'node', 'node.pages');

    $node = node_load($nodeToLoad);
    dsm($nodeToLoad);
    $edit_page = drupal_get_form('page_node_form', $node);

    $edit_page['additional_settings']['#access'] = FALSE;
    $edit_page['buttons']['submit']['#submit'][] = 'redirect_submit';
    print drupal_render($edit_page );

    function redirect_submit($form, &$form_state){
        $form_state['redirect'] = 'www.google.com';
    }
?>
</code>
I'd like to remove the 'Delete' button of this form(which normally deletes the content).
I have read that this is easily done with hook_form_alter() but my code resides within the header of a View and I'd like to keep it that way.
Is there a way to do this?
Answer: Found a solution, but if there is a better way I'd like to hear more.
<code>$edit_page['actions']['delete']=false; //removed delete button
$edit_page['title']=false; //removed title display as well
</code>


Problemas com o Collapse Bootstrap

Question: Tenho um Collapse Bootstrap dentro de um while, onde é implementado várias vezes com diferentes informações dentro. Porém, quando eu rodo o código, quando dois collapses estão juntos, apenas um abre. Literalmente, se eu tenho 4 collapses, não importa em qual deles eu clico, somente um abre com as informações.
Aqui vai o código:
listaProposta.php ->
<?php
include ("valida_paginas_adm.php");

?>
  <?php include_once("navbar.php") ?>

Serviços <?php echo $resn ?>

                                    <?php

                                        while ($exibirRegistros = mysqli_fetch_array ($consulta)){
                                            $idProposta = $exibirRegistros[0];
                                            $numero = $exibirRegistros[1];
                                            $servico = $exibirRegistros[2];
                                            $descricao = $exibirRegistros[3];
                                            $data = $exibirRegistros[4];
                                            $valor = $exibirRegistros[5];
                                            $idEmpresa = $exibirRegistros[6];

                                            echo "

                                                        Serviço: $servico

                                                        Número da Proposta: $numero

                                                        $descricao

                                                        Data: $data | Valor: $valor

                                                            <button class='btn btn-danger collapsed' type='button' data-toggle='collapse' data-target='#lista02' aria-expanded='false' aria-controls='lista02'>
                                                            Status da Proposta
                                                            </button>

";
                                                            include("listaStatus.php");
                                                        echo "

                                            ";

                                        }

                                    ?>

  <?php include_once("rodape.php") ?>

listaStatus.php ->
<?php

include_once("conexao.php");

    $sqli = "select * from status where idProposta='$idProposta'";
    $consultar = mysqli_query($conexao,$sqli);

    while ($exibirR = mysqli_fetch_array ($consultar)){
            $idStatus = $exibirR[0];
            $statusProposta = $exibirR[1];
            $data = $exibirR[2];
            $idProposta = $exibirR[3];

        echo "

                        $statusProposta

                        $data

        ";

            }

?>
Answer: Você precisa criar os ids da sua div e os atibutos data-target e aria-controls dinamicamente, exemplo:

Serviços <?php echo $resn ?>

                            <?php
                            $idx = 0;
                            while ($exibirRegistros = mysqli_fetch_array($consulta)) {
                                $idProposta = $exibirRegistros[0];
                                $numero = $exibirRegistros[1];
                                $servico = $exibirRegistros[2];
                                $descricao = $exibirRegistros[3];
                                $data = $exibirRegistros[4];
                                $valor = $exibirRegistros[5];
                                $idEmpresa = $exibirRegistros[6];

                                echo "

                                                    Serviço: $servico

                                                    Número da Proposta: $numero

                                                    $descricao

                                                    Data: $data | Valor: $valor

                                                        <button class='btn btn-danger collapsed' type='button' data-toggle='collapse' data-target='#lista02$idx' aria-expanded='false' aria-controls='lista02$idx'>
                                                        Status da Proposta
                                                        </button>

";
                                include("listaStatus.php");
                                echo "

                                        ";
                                $idx++;
                            }
                            ?>

desculpa qualquer erro, estou editando pelo celular
Comment: Muito obrigada, resolveu meu problema!
Answer: isso provavelmente ocorre por conta do data-target e do id estarem sempre os mesmo. você esta criando eles todos com o mesmo id, acredito que seja esse o motivo. tenta adicionar uma variável pra mudar o id e o data-target conforme o while cria os collapse

Problemas com o Collapse Bootstrap


Woocommerce - campo especial dentro do jquery e usar opreadores de divisão

Question: estou em um projeto pessoal, e estou com dificuldade em entender como funciona o jquery no woocommerce, eu utilizo o woocommerce como um sistema de cotação e estou tentado gerar uma tabela de preço dinamica de apresentação >>sem integração (eu nao vou salvar salvo os dados no BD )<< com o sistema do woocommerce, quero apenas uma tabela de apresentação na pagina do produto que utiliza os dados captados da loja como preço e alguns atributos.
O que eu tenho ate agora é o código abaixo:
<code><script>
jQuery(function($) {
    var pc = 'p.price' //preco de custo
        q = $(pc).html();

    $('form.cart').on('show_variation', function( event, data ) {
        if ( data.price_html ) {
            $(pc).html(data.price_html); // mostra o valor
        }
    }).on('hide_variation', function( event ) {
        $(pc).html(q);

    });
});
</script>
</code>
Na pagina do produto do woocommerce eu tenho que captar o preço do produto (o código acima ja faz isso) e fazer uma formula com os campos atributo "pa_custo" e "pa_qdy"
a formula que eu gostaria de aplicar a este código seria
Preço do produto $(pc) + ("pa_custo" / (dividido) "pa_qdy") = x
alguém poderia me ajudar por favor?
Answer: Se os seletores são classes seria neste exemplo:
<code>  var ex = parseFloat($('.price').val()) + (parseFloat($('.pa_custo').val()) / parseFloat($('.pa_qdy').val());
</code>


Why is aptitude not installing python-mysqldb properly? ( error: No module named MySQLdb )

Question: username@servername Tue Nov 02 22:08:28 ~/public_html/IDM_app
$ sudo aptitude install mysql-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
The following NEW packages will be installed:
  libdbd-mysql-perl{a} libdbi-perl{a} libhtml-template-perl{a} libmysqlclient16{a}
  libnet-daemon-perl{a} libplrpc-perl{a} mysql-client-5.1{a} mysql-client-core-5.1{a} mysql-common{a}
  mysql-server mysql-server-5.1{a} mysql-server-core-5.1{a}
0 packages upgraded, 12 newly installed, 0 to remove and 44 not upgraded.
Need to get 0B/24.3MB of archives. After unpacking 60.9MB will be used.
Do you want to continue? [Y/n/?] Y
Writing extended state information... Done
Preconfiguring packages ...
Selecting previously deselected package mysql-common.
(Reading database ... 20736 files and directories currently installed.)
Unpacking mysql-common (from .../mysql-common_5.1.41-3ubuntu12.6_all.deb) ...
Selecting previously deselected package libnet-daemon-perl.
Unpacking libnet-daemon-perl (from .../libnet-daemon-perl_0.43-1_all.deb) ...
Selecting previously deselected package libplrpc-perl.
Unpacking libplrpc-perl (from .../libplrpc-perl_0.2020-2_all.deb) ...
Selecting previously deselected package libdbi-perl.
Unpacking libdbi-perl (from .../libdbi-perl_1.609-1build1_amd64.deb) ...
Selecting previously deselected package libmysqlclient16.
Unpacking libmysqlclient16 (from .../libmysqlclient16_5.1.41-3ubuntu12.6_amd64.deb) ...
Selecting previously deselected package libdbd-mysql-perl.
Unpacking libdbd-mysql-perl (from .../libdbd-mysql-perl_4.012-1ubuntu1_amd64.deb) ...
Selecting previously deselected package mysql-client-core-5.1.
Unpacking mysql-client-core-5.1 (from .../mysql-client-core-5.1_5.1.41-3ubuntu12.6_amd64.deb) ...
Selecting previously deselected package mysql-client-5.1.
Unpacking mysql-client-5.1 (from .../mysql-client-5.1_5.1.41-3ubuntu12.6_amd64.deb) ...
Selecting previously deselected package mysql-server-core-5.1.
Unpacking mysql-server-core-5.1 (from .../mysql-server-core-5.1_5.1.41-3ubuntu12.6_amd64.deb) ...
Processing triggers for man-db ...
Setting up mysql-common (5.1.41-3ubuntu12.6) ...
Selecting previously deselected package mysql-server-5.1.
(Reading database ... 21101 files and directories currently installed.)
Unpacking mysql-server-5.1 (from .../mysql-server-5.1_5.1.41-3ubuntu12.6_amd64.deb) ...
Selecting previously deselected package libhtml-template-perl.
Unpacking libhtml-template-perl (from .../libhtml-template-perl_2.9-1_all.deb) ...
Selecting previously deselected package mysql-server.
Unpacking mysql-server (from .../mysql-server_5.1.41-3ubuntu12.6_all.deb) ...
Processing triggers for ureadahead ...
Processing triggers for man-db ...
Setting up libnet-daemon-perl (0.43-1) ...
Setting up libplrpc-perl (0.2020-2) ...
Setting up libdbi-perl (1.609-1build1) ...
Setting up libmysqlclient16 (5.1.41-3ubuntu12.6) ...

Setting up libdbd-mysql-perl (4.012-1ubuntu1) ...
Setting up mysql-client-core-5.1 (5.1.41-3ubuntu12.6) ...
Setting up mysql-client-5.1 (5.1.41-3ubuntu12.6) ...
Setting up mysql-server-core-5.1 (5.1.41-3ubuntu12.6) ...
Setting up mysql-server-5.1 (5.1.41-3ubuntu12.6) ...
mysql start/running, process 26154

Setting up libhtml-template-perl (2.9-1) ...
Setting up mysql-server (5.1.41-3ubuntu12.6) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done

username@servername Tue Nov 02 22:09:21 ~/public_html/IDM_app
$ sudo aptitude install python-mysqldb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
The following NEW packages will be installed:
  python-mysqldb python-support{a}
0 packages upgraded, 2 newly installed, 0 to remove and 44 not upgraded.
Need to get 0B/112kB of archives. After unpacking 504kB will be used.
Do you want to continue? [Y/n/?] y
Writing extended state information... Done
Selecting previously deselected package python-support.
(Reading database ... 21184 files and directories currently installed.)
Unpacking python-support (from .../python-support_1.0.4ubuntu1_all.deb) ...
Selecting previously deselected package python-mysqldb.
Unpacking python-mysqldb (from .../python-mysqldb_1.2.2-10build1_amd64.deb) ...
Processing triggers for man-db ...
Setting up python-support (1.0.4ubuntu1) ...

Setting up python-mysqldb (1.2.2-10build1) ...

Processing triggers for python-support ...
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done

username@servername Tue Nov 02 22:10:26 ~/public_html/IDM_app
$ python manage.py syncdb
Traceback (most recent call last):
  File "manage.py", line 13, in <module>
    execute_manager(settings)
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 67, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/commands/syncdb.py", line 7, in <module>
    from django.core.management.sql import custom_sql_for_model, emit_post_sync_signal
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/sql.py", line 5, in <module>
    from django.contrib.contenttypes import generic
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/contrib/contenttypes/generic.py", line 6, in <module>
    from django.db import connection
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/db/__init__.py", line 77, in <module>
    connection = connections[DEFAULT_DB_ALIAS]
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/db/utils.py", line 91, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/db/utils.py", line 32, in load_backend
    return import_module('.base', backend_name)
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/db/backends/mysql/base.py", line 14, in <module>
    raise ImproperlyConfigured("Error loading MySQLdb module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: No module named MySQLdb

username@servername Wed Nov 03 03:24:00 ~/public_html/IDM_app
$ find / -name python-mysqldb 2>/dev/null
/usr/share/doc/python-mysqldb

username@servername Wed Nov 03 12:46:55 ~/public_html/IDM_app
$ ls /usr/local/mysql/bin/mysql_config
ls: cannot access /usr/local/mysql/bin/mysql_config: No such file or directory

username@servername Wed Nov 03 13:21:02 ~/public_html/IDM_app
$ find / -name mysql_config 2>/dev/null
username@servername Wed Nov 03 13:23:04 ~/public_html/IDM_app
$ find / -name python-mysqldb 2>/dev/null
/usr/share/doc/python-mysqldb

username@servername Wed Nov 03 13:29:01 ~/public_html/IDM_app
$ find / -name python-mysql 2>/dev/null
username@servername Wed Nov 03 13:29:21 ~/public_html/IDM_app
$ find / -name mysql 2>/dev/null
/etc/init.d/mysql
/etc/mysql
/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/contrib/gis/db/backends/mysql
/usr/local/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/db/backends/mysql
/usr/share/mysql
/usr/bin/mysql
/usr/lib/perl5/auto/DBD/mysql
/usr/lib/perl5/DBD/mysql
/usr/lib/mysql
/var/log/mysql
/var/lib/update-rc.d/mysql
/var/lib/mysql

username@servername Wed Nov 03 13:30:21 ~/public_html/IDM_app
$ ls -la  /usr/bin/mysql
-rwxr-xr-x 1 root 183376 Jul 29 22:54 /usr/bin/mysql

username@servername Wed Nov 03 13:35:01 ~/public_html/IDM_app
$ find / -name site.cfg 2>/dev/null
username@servername Wed Nov 03 13:44:04 ~/public_html/IDM_app
$ ls -la /usr/lib/mysql/
total 16
drwxr-xr-x  3 root 4096 Nov  2 22:09 .
drwxr-xr-x 42 root 8192 Nov  2 22:10 ..
drwxr-xr-x  2 root 4096 Nov  2 22:09 plugin

Answer: The fix was:
sudo aptitude install libmysqlclient-dev

sudo easy_install MySQL-python

Answer: First, try dpkg -L python-mysqldb, then you'll see what's installed.
Second, you are obviously using a manully compiled Python 2.7, while Ubuntu installs its packages for Python 2.6, which is the current default. You'll need to use easy_install to install python-mysql to your 2.7 installation.
Comment: This was helpful, but did not solve my problem. sudo aptitude install libmysqlclient-dev; sudo easy_install MySQL-python     did fix it.


Chamar uma função depois de validar um formulário no Bootstrap

Question: Estou a tentar fazer um exercício que consiste em mostrar uns valores após o utilizador fornecer os dados correctos num formulário. Tentei usar
<form class="needs-validation" onsubmit="calcular()" novalidate>

Mas apenas limpa os dados do utilizador em vez de fazer a função calcular.
Está aqui o código completo:
https://jsfiddle.net/ctanunes/c1nj2vwd/
Comment: Mas vc quer submeter o formulário ou apenas calcular?
Comment: apenas calcular, mas queria as validações também
Answer: Tem alguns problemas no código que está impedindo de funcionar. Um deles é o botão:
<button type="clean" class="btn btn-primary mb-2 mr-3" id="limpar" onclick="limpar()">Limpar</button>

Esse type="clean" não existe nas especificações HTML. Como é inválido, o botão vai funcionar no padrão type="submit", submetendo o formulário.
Esse novalidate invalida os required. Como o próprio nome sugere, quer dizer que os campos não precisam ser validados pelo HTML.
Altere o type do botão "Limpar" para button:
<button type="button" class="btn btn-primary mb-2 mr-3" id="limpar" onclick="limpar()">Limpar</button>

Também remova os ids dois dois botões. Ao usar o mesmo nome no id de um elemento dentro de um form e de uma função, gera conflito. Se não precisa dos ids, remova-os ou troque de nome.
No form.addEventListener('submit', function (event) { creio que apenas o preventdefault() seja suficiente, para impedir que o formulário seja submetido.
Na função limpar() é preciso também remover a classe .was-validated do form.
Vamos ver funcionando:

<!doctype html>

    <title>Ficha6</title>

    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm"
        crossorigin="anonymous">

Cálculo de Escalão e IRC

        <form class="needs-validation" onsubmit="calcular()">

                Insira o nome da empresa
                <input type="text" class="form-control" id="nomeEmpresa" placeholder="Nome Empresa" required>

                    Bom nome!

                Insira o valor do lucro (euros)
                <input type="number" class="form-control" id="lucro" min="0" placeholder="Lucro" required>

                    Insira um valor positivo!

                Escalão
                <input type="text" class="form-control" id="escalao" readonly>

                Taxa de IRC
                <input type="text" class="form-control" id="taxa" readonly>

                Valor imposto (euros)
                <input type="text" class="form-control" id="imposto" readonly>

            <button type="button" class="btn btn-primary mb-2 mr-3" onclick="limpar()">Limpar</button>
            <button type="submit" class="btn btn-primary mb-2 float-right">Calcular</button>
        </form>

Comment: Muito obrigada pela explicação! Era mesmo isto :)
Answer:
Ao anexar o manipulador de eventos ao elemento de formulário, o escopo do manipulador de eventos é o formulário e não a janela.
Como os elementos de entrada dentro de um formulário são anexados como propriedades ao objeto form, onde o nome é a chave, chamar calcular()o manipulador de eventos, onde o escopo é o formulário, seria igual a chamada form.calcular(), mas o formulário já tem um elemento com esse nome form.calcular é o botão de submit, não a calcular() função no escopo global.

Toque o iddo button id="btn_calcular".
Fonte
Comment: Infelizmente isso não resolve o problema
Comment: @CatarinaNunes, o que exatamente você quer a função calcular retorne?
Comment: A função calcular muda os valores dentro das caixas do Escalão, Taxa e Valor.
Comment: Veja, a função calcular está a funcionar se fizer a alteração no `id`, entretanto o elemento responsável por exibir o retorno do calculo é `readonly` (apenas leitura), então você precisar acertar essa lógica.


Por que esse Css não funciona?

Question: eu tenho algumas dúvidas neste código. Eu gostaria de usar fonte open sans no texto, que o texto fosse alinhado para cima na tabela. Sabem dizer pq o css nao funciona aplicado ao html? Lembrando que a outra td da tabela tem imagens uma em cima da outra que esticam a coluna.

texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto textotexto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto textotexto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto textotexto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto textotexto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto textotexto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto texto

css:
.texto {
    font-family: "Open Sans", sans-serif;
    font-size: 12px;
    margin-top: 0px;
    margin-left:30px;
    margin-right: 15px;
}

Comment: Reformate sua pergunta e deixe seu problema mais claro por favor, assim poderemos lhe ajudar!
Comment: Você só consegue visualizar uma fonte em um site, se aquela fonte (um arquivo de extensão .ttf) estiver salva no seu computador. Do jeito que font-family na classe texto está definido quer dizer que se no seu computador não tiver "Open Sans" instalado o texto vai ser com sans-serif e se esta tb não estiver instalada ai vai ser apresentado na font default do navegador. Faça o que Thiago Custodio recomenda que seu texto vai funfar!!!
Answer: Você precisa importar a fonte antes de usá-la:
<link rel="stylesheet" type="text/css" href="//fonts.googleapis.com/css?family=Open+Sans" />

Answer: Insira no topo de seu css o @import url que problema vai ser resolvido
Veja o exemplo:

@import url('https://fonts.googleapis.com/css?family=Open+Sans');

.texto {
    font-family: "Open Sans", sans-serif;
    font-size: 12px;
    margin-top: 0px;
    margin-left:30px;
    margin-right: 15px;
}

Por que esse Css não funciona?


Where and how to conditionally redirect old aliases to new aliases

Question: We are replacing a website with a new version. The domain is the same, but URL aliases in the new website are different.
In many cases, I can use an old alias to calculate its corresponding new alias. There are many thousands of old aliases, so I don't want to just use the Redirect module. I know how to write the calculations and how to check if a calculated new alias exists.
I need to know at what point I can intercept each incoming URL to calculate a new alias.
I also need to know how to redirect to the calculated new alias if it exists.
Concrete example:
A book chapter on the old site is at example.com/recent/secrets-to-success/10-knowing-when-to-quit. This chapter on the new site is at example.com/secrets-to-success/knowing-when-quit. The difference is (a) we no longer add the recent subfolder, (b) we've removed chapter numbers from chapter titles, and (c) we've changed which words are ignored when an alias is auto-generated.
When someone uses an old alias, I want to intercept it, calculate the new alias, check if it exists, and redirect to it if it does. This is only one example. There are several other ways of calculating, so there's some trial and error involved. But I know how to take care of that.
Comment: Sounds like a job for the [redirect](https://www.drupal.org/project/redirect) module.
Comment: By this description you don't need access to Drupals DB at all, this is just a one fixed part changing to another fxed part, I'd recommend  batmans answer using .htaccess.
Comment: It would be better to provide an example that better matches your actual requirements. The specifics of those may well determine what the best approach to take is
Comment: I agree with @Hudri: Changing _old_ with _new_ in a URL doesn't require any access to the database used from Drupal, except in the case _old_ and _new_ used in the question are placeholders, and the values you really use/need are taken from the database. If the latter is the case, you should edit the question to explain that, since the simplification simplified too much.
Comment: Thanks to @kiamlaluno for taking the time to help me see that I needed to fix my question. I have done so.
Comment: Thanks also to @Clive for the same.
Answer: This solution is based on a suggestion by @4k4 at Redirect from within controller always goes to home page.
My custom module is named rsc.
In modules/custom/rsc the file rsc.services.yml includes the following:
services:
  rsc.event.Redirect404:
    class: Drupal\rsc\EventSubscriber\rscRedirect404
    tags:
      - {name: event_subscriber}

In modules/custom/rsc/src/EventSubscriber the file rscRedirect404.php contains the following:
<?php
namespace Drupal\rsc\EventSubscriber;

use Drupal\Core\EventSubscriber\HttpExceptionSubscriberBase;
use Drupal\Core\Url;
use Symfony\Component\HttpFoundation\RedirectResponse;
use Symfony\Component\HttpKernel\Event\GetResponseForExceptionEvent;

class rscRedirect404 extends HttpExceptionSubscriberBase {

  protected function getHandledFormats() { // required
    return ['html'];
  }

  public function on404(GetResponseForExceptionEvent $event) {

    $path = \Drupal::request()->getPathInfo();
    $path = [code to modify the path from old site to new site]

    if (($url = \Drupal::service('path.validator')->getUrlIfValid($path))) {
      $event->setResponse(new RedirectResponse(URL::fromUserInput($path)->toString()));
    }
  }
}

With this in place, I see 404 redirects before they get passed to the default handler. If I do the $event->setResponse(), the user is redirected as I specify. If not, normal processing occurs.
In reality, the line that says [code to modify the path from old site to new site] is multiple lines where I do several trial-and-error tests to figure out the correct new URL,  but everything else is just as I'm using it.
Answer: This has nothing to do with drupal. Need to edit the .htaccess file
RewriteEngine On
RewriteRule ^test/(.*)$ /new/$1 [R=301,L]

This will redirect example.com/test/1 to example.com/new/1
Comment: Thanks for this comment. It highlights a weakness in my description which I have now corrected. A solution that works for me requires me to have access to the Drupal database when I calculate the redirection, but I hadn't said this. I gave you +1 for helping me fix my description.
Answer:
For example, what was example.com/old-style-alias might now be
  example.com/a-very-different-alias.

Redirect module is exactly what you need.
This can automatically add a redirect when you change the node's alias or you can manually add them to the redirect list, which gets saved to the DB.
Comment: thanks for your kind suggestion. I already use the Redirect module, but since all content has already been moved to the new site with modified aliases, I don't think I can use the automatic feature. There are many thousands of nodes with changed aliases, so adding them manually would take a long time. This is why I'm trying to find a solution for redirecting using code I can control. Or am I missing a feature of Redirect? I've added a "concrete example" to my original post that may help explain.


How to find conditional distributions from joint

Question: I want to learn about how to do Gibbs sampling, starting with finding conditional distributions given a joint distribution.  While looking for examples, I found this blog post that I wanted to replicate on my own, but I am having trouble understanding how to algebraically find the conditional distributions given the joint distribution.
The joint distribution is
$$ f(x, y) = kx^2 exp(-xy^2 - y^2 + 2y - 4x) $$ where $k$ is a normalization constant.  The marginal distributions are
$$x \mid y \sim Gamma(3, y^2 + 4) $$
and
$$y \mid x \sim N(\frac{1}{x + 1}, \frac{1}{2(x + 1)}) $$
What are the steps to derive the two marginals from the joint?  Is there a trick I missing?  I feel like since it was unexplained in the example it should be somewhat simple but I am just not seeing it...
Thanks!  Also, bonus points if anyone can point me to any (free to access) examples with solutions similar to this.  I want to practice deriving conditional distributions from joint ones.
Answer: Those distributions you call "marginal" are not marginal.  They are conditional distributions because you wrote $x \mid y$.  The marginal distribution of $X$, for example, is necessarily independent of the value of $Y$.
To see how the conditional distribution is gamma, all you have to do is write $$f_{X \mid Y}(x) = \frac{f_{X,Y}(x,y)}{f_Y(y)} \propto f_{X,Y}(x,y).$$  That is to say, the conditional distribution is proportional to the joint distribution, appropriately normalized.  So we have $$f_{X \mid Y}(x) \propto x^2 e^{-x(y^2+4)},$$ completely ignoring any factors that are not functions of $x$.  Then we recognize that the gamma distribution has density $$f_S(s) \propto s^{a-1} e^{-bs},$$ so the choice of shape $a = 3$ and rate $b = y^2+4$ demonstrates that the conditional distribution $X \mid Y \sim \operatorname{Gamma}(a = 3, b = y^2+4)$.
The conditional distribution of $Y \mid X$ is done in a similar fashion.  Just ignore constants of proportionality:  $$f_{Y \mid X}(y) \propto e^{-(x+1)y^2+2y},$$ but this one requires us to complete the square to get it to look like a normal density:  $$-(x+1)y^2 + 2y = (x+1)\left(-\left(y - \tfrac{1}{x+1}\right)^2 \right) + \tfrac{1}{x+1},$$ and after exponentiating and removing the $e^{1/(x+1)}$ factor, comparing this against $$f_W(w) \propto e^{-(w-\mu)^2/(2\sigma^2)},$$ we see that we have a normal density with mean $\mu = 1/(x+1)$ and variance $\sigma^2 = 1/(2(x+1))$.
Now, if you wanted the marginal distributions, you would need to integrate:  $$f_X(x) = \int_{y=-\infty}^\infty f_{X,Y}(x,y) \, dy,$$ for example.  And as you can see, this expression will not be a function of $Y$.  The difference is that if I simulated realizations of ordered pairs $(X_i, Y_i)$ from the joint distribution, the marginal density for $X$ would be what you would see if I only told you the values of $X_i$.  The conditional distribution of $X$ given $Y = y$ would be what you would see if I only gave you the $X_i$ for which the corresponding $Y_i$ was equal to $y$.
Answer: The "trick" is to observe that $f(x\mid y)=f(x,y)/f(y)$ is proportional to $f(x,y)$ up to terms that do not involve $x$. Hence, $f(x\mid y)\propto x^2\exp(-(y^2+4)x)$, and this is the "kernel" of a $\mathrm{Gamma}(3,y^2+4)$ density. The other full conditional $f(y\mid x)$ is obtained similarly after completing the square in the exponent.


Optimizing MySQL query with SUM, date range and Group By

Question: I have following tables:
CREATE TABLE IF NOT EXISTS stats (
    date date NOT NULL DEFAULT '0000-00-00',
    cid int(8) NOT NULL DEFAULT '0',
    v bigint(15) NOT NULL DEFAULT '0',
    c bigint(15) NOT NULL DEFAULT '0',
    a bigint(15) NOT NULL DEFAULT '0',
PRIMARY KEY (date,cid),
KEY date (date),
KEY cid (cid),
KEY date_cid_vca (date,cid,v,c,a)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

this table has 30,842,712 rows
and
CREATE TABLE IF NOT EXISTS camp (
id int(8) NOT NULL AUTO_INCREMENT,
name varchar(80) NOT NULL DEFAULT '',
PRIMARY KEY (id,name)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

this table has 1985 rows
I have the following query:
SELECT
    c.id,
    c.name,
    SUM(s.v) AS sumv,
    SUM(s.c) AS sumc,
    GREATEST(((SUM(s.c)/SUM(s.v))*100.00), 0.00) AS cratio,
    SUM(s.a) AS suma,
    GREATEST(((SUM(s.a)/SUM(s.c))*100.00), 0.00) AS aratio
FROM
    stats s, camp c
WHERE
    s.date >= '2012-02-01' AND
    s.date <= '2012-02-29' AND
    c.id=s.cid
GROUP BY s.cid;

EXPLAIN shows:
+----+-------------+-------+-------+-------------------------------+--------------+---------+---------------------+---------+-----------------------------------------------------------+
| id | select_type | table | type  | possible_keys                 | key          | key_len | ref                 | rows    | Extra                                                     |
+----+-------------+-------+-------+-------------------------------+--------------+---------+---------------------+---------+-----------------------------------------------------------+
|  1 | SIMPLE      | s     | range | PRIMARY,date,cid,date_cid_vca | date_cid_vca | 3       | NULL                | 1010265 | Using where; Using index; Using temporary; Using filesort |
|  1 | SIMPLE      | c     | ref   | PRIMARY                       | PRIMARY      | 4       | db.s.cid            |       1 | Using index                                               |
+----+-------------+-------+-------+-------------------------------+--------------+---------+---------------------+---------+-----------------------------------------------------------+

Problem is that the query takes around 50 seconds to complete even though it uses indexes. Is there any other way to optimize the query?
Thanks!
Answer: You've done well in optimizing this query with your indexes. I suppose you really do have over 1 million rows in stats in that date range. Unfortunately, joining (and then grouping) 1 million rows, even with a covering index, is a lot to ask from a database. For better performance, you will need to beef up the hardware, start denormalizing (put camp inside of stats to avoid the join), or keep running totals for each camp instead of calculating it on the fly.
Edit
Since removing the 1 million+ joins seemed to have made a large impact, you can try something like this:
SELECT c.*, a.* FROM
(SELECT
    SUM(s.v) AS sumv,
    SUM(s.c) AS sumc,
    GREATEST(((SUM(s.c)/SUM(s.v))*100.00), 0.00) AS cratio,
    SUM(s.a) AS suma,
    GREATEST(((SUM(s.a)/SUM(s.c))*100.00), 0.00) AS aratio,
    s.cid
FROM
    stats s
WHERE
    s.date >= '2012-02-01'
   AND s.date <= '2012-02-29'
GROUP BY s.cid) a
JOIN
  camp c
  ON c.id = a.cid

This query does the join on the smaller result set.
Comment: I think hardware is not the issue. Server has 4 CPUs with 8 cores (32 cores, 64 threads) and 64GB RAM. It has 6x SSD disks in RAID 10. my.cnf is maximum optimized (based on tuning-primer.sh). Can you please give more info on denormalizing. Thanks!
Comment: I've tried removing join and then pulling data from camp with separate query within data loop from main query. I guess it is the way to go because now total execution time reduced enormously. It takes around 10 seconds to complete.
Comment: Sounds like nice, beefy, hardware. :)
Comment: this query is superb. You only forgot to add GROUP BY in the subquery. It takes the same time as my query without JOIN. Thanks!
Comment: @Paxxil, I added the missing `GROUP BY`. Thanks.
Answer: The following query should allow it to use indexes more efficiently
SELECT
    c.id,
    c.name,
    SUM(s.v) AS sumv,
    SUM(s.c) AS sumc,
    GREATEST(((SUM(s.c)/SUM(s.v))*100.00), 0.00) AS cratio,
    SUM(s.a) AS suma,
    GREATEST(((SUM(s.a)/SUM(s.c))*100.00), 0.00) AS aratio
FROM
    camp c
INNER JOIN
    stats s
ON
    s.cid = c.id
    AND s.date BETWEEN '2012-02-01' AND '2012-02-29'

GROUP BY c.id;

Also I would consider deleting the date_cid_vca key as it is just containing the entire table and so not particularly useful.  The above query should use the PK to match rows from stats to camp based on date and cid and so while it'shard to be 100% certain without access to your database, I'm fairly confident that the above will improve your response times
Comment: I've tried your suggestion but unfortunately it takes around 5 seconds longer to execute your version of the query. Without index date_cid_vca it takes twice as much time.
Answer: You can create a Inner join with your C table, and in the join use the dates conditions, it should reduce the time of your query.
You could probably do more optimizations but thats the first 1 i could see.


Imbalanced classes and possible ways to increase precision, recall and f1-score of the prediction model

Question: I've just started my data science internship, and this is my first time in the field. I'm sure I'll face challenges in the future where I might need your help. It's also my first time asking a question on Stack Exchange, so I hope the community will be friendly and understanding. I have a binary classification problem of performing and non performing loans of customers. My ultimate goal is to predict whether a customers is performing or non performing when they come to take a loan. My dataset is imbalanced as there are lot of performing customers(74%) than non performing(26%). I have applied logistic regression and as a remedy for the imbalanced data set I have used SMOTE  which is a oversampling method but it generates data synthetically and surprisingly give very good prediction accuracy, precision and recall. But as it is not appropriate to oversample in this context my supervisor asked me to apply cost sensitive learning. I have applied cost sensitive learning to both logistic and random forest but logistic regression perform better. Here is the confusion matrix and prediction report of the logistic regression when applied cost sensitive learning.

As you can see here precision and f1-score of non performing customers(1)  not satisfying. Also most critical in the context is to identify non performing customers(1). From most methods I used these metrics increased for the majority class(0) but further decrease for the minority class(1). I have seen similar questions but most of them are not clear enough. So I need some advice on what are the possible methods that I can use to overcome the imbalance problem and increase the precision, recall and f1-score of both classes. Thank you in advance!
Comment: Please take a look at the following threads: https://stats.stackexchange.com/q/312780/1352 (precision, recall and F1 suffer from precisely the same issues as accuracy), https://stats.stackexchange.com/q/357466/1352 on "unbalanced" data and oversampling such as SMOTE, and, e.g., https://stats.stackexchange.com/q/312119/1352 on including costs into the decision analysis.
Comment: [Are you in the same class/course as this poster?](https://stats.stackexchange.com/q/629692/1352)
Answer: Like many other claimed "classification" models, logistic regressions return predictions on a continuum. That is, logistic regressions do no classification. The classification comes from a two-step pipeline of getting the continuous predictions from the logistic regression and then applying a decision rule to bucket those continuous predictions into categories, typically by applying a threshold above which the categorical predictions are category $1$ and below which the predictions are category $0$ (or $-1$, depending on how you've coded them). Thus, if you're unhappy with the classification performance, there are two possible culprits: the logistic regression is doing a poor job of distinguishing between the categories or the decision rule is inappropriate despite fine performance by the regression.
Unfortunately, much data science training treats this two-stage pipeline as one stage where the decision stage is fixed as assigning to the most probable category, and all work is for the first stage. While this at first does not sound outrageous, in imbalanced problems, it's reasonable for the minority category never to be more likely. In this case, such a decision rule will lead to no predictions of the minority category. However, nothing forces you to use that decision rule. There are reasons to look at the direct probability outputs without bucketing them, but seeing how your classification metrics perform over a range of thresholds can help you feel better about how your model is performing (at least in terms of ability to distinguish between the two categories). Maybe your precision, recall, and $F_1$ score are poor at the software-default decision rule, but how are they for other decision rules? You can answer this by looping over many thresholds, bucketing into categories according to these thresholds, and plotting the performance by threshold.
This still misses out on the richness of the probability predictions and what you can do when they are good, but you will learn more about how well the model distinguishes between categories by doing this than you will by using the software-default decision rule. Further, this neither requires you to lie to the model about the data by running techniques like SMOTE or under/overaampling, and you don't have to fiddle with weighted loss functions whose effectiveness you cannot assess until after you've trained the model (though the computing time for a logistic regression is probably not so great to make this unrealistic), as assessing the performance at many thresholds is performed after fitting the model, not before.
I still encourage you to get out of the classification mindset and get into a mindset where the goal is to accurately predict probability, but seeing what happens for many possible second stages of the pipeline at reveal that your ability to classify is not as bad as you thought.
To give an idea of what the probabilities can do for your problem with lending money, the lender doesn't decide just to lend or not to lend. The lender considers an interest rate, and a person with a higher probability of defaulting would get a loan with a higher interest rate. This is basically how credit scores work: when you have a low credit score, your default probability is assumed to be high and that you are a risk, so your interest rate is high to compensate the lender for taking that risk.
LINKS OF POSSIBLE INTEREST
Predictions vs Decisions [especially Kolassa's answer]
Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?
Why is accuracy not the best measure for assessing classification models? [Especially Kolassa's Answer]
Proper scoring rule when there is a decision to make (e.g. spam vs ham email) [Especially Kolassa's Answer]
Regression on imbalanced and zero-inflated data. How to deal with less frequent values? [This is where I opine that an inability to predict unusual events is expected behavior unless you have some characteristic of those events that distinguishes them from "business as usual".]
How to calculate accuracy of a logistic regression? [Data Science Stack Exchange]
Harrell's Blog: Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules
Harrell's Blog: Classification vs. Prediction


How do I calculate uplift in a marketing campaign?

Question: I have a campaign that had two interventions A and B that were run in 3 markets each. I, also, had a control in 15 markets where no interventions were run.
For the markets where intervention A was run we had 5 units sold in the time period before intervention and 5 units during intervention.
For the markets where intervention B was run we had 10 units sold in the time period before intervention and 15 units during intervention.
Now, the control markets had 100 units prior to intervention and 200 during.
So...
<code>            Prior         During
Control      100            200
Group A        5              5
Group B       10             15
</code>
The marked increase in the Control was due to seasonality.
Now, I want to calculate uplift (or downlift with this example data) by saying Group A should have expected to have 10 unit sales during intervention thus
<code> (5-10)/5  = -1
</code>
While Group B has an uplift of
<code>(15-20)/10 = -0.5
</code>
One of my colleagues wants to divide by the control values to account for the seasonality while I think we should subtract by the expected unit sales to account for the seasonality.
Sorry, I cannot give more details.
Answer: Usually uplift is a dimensionless entity (expressed in percentages) which implies a percentage observed change in reaction after an application of a treatment, adjusted to external factors (seasonality, etc)
In this case, the impact of factors external to the intervention is a 2x increase in sales.
Hence the uplift of group A = Observed change - change due to external factors
 uplift (A) = 5/5 - 200/100
= 100% - 200%
= -100%, which could mean either of
a) The impact of the treatment is negative
b) The Control selection is not representative as some of the factors (such as critical mass of your product sales) are not applicably similar to the test
Answer: Depends on your definition of uplift, I guess. In my book, uplift would be the difference between the actual and the anticipated, related to the anticipated. Assuming seasonality equally effects all markets (by multiplying by 2), the anticipated value of group A is 10, so lift is (5-10)/10=-0.5 (or -50%). For group B, (15-20)/20=-0.25 (or -25%).
Again, be sure to know your definitions (I find that they differ widely among companies even in the same sector). Also, I used a simplified seasonality "model" - if you have more data points, you may come up with something better than just "multiply by 2".
Comment: No idea, I am a newbie, too. I am not even sure the guidelines encourage this kind of meta-discussion in comments :)
Comment: Andris,

Thanks so much. I appreciate the help.

This was my first post, Is there a way for me to mark it.  I don't have enough karma for an up vote.


Similar Eigenvalues

Question: Let $A,B \in \mathbb{C}^{n \times n}$ be related by $B=PAP^{*}$, where $P \in \mathbb{C}^{n \times n}$ is a unitary matrix. Furthermore, $A$ has an eigenbasis $f_{j}$, and corresponding eigenvalues $\lambda_{j}$ ordered such that $\lambda_{1} \leq \lambda_{2} \leq \cdots \leq \lambda_{n}$.
What is the simplest way to show that $A$ and $B$ have the same eigenvalues? Is proof of Schur's theorem sufficient for this task?
Comment: Since $P$ is unitary $P^*=P^{-1}$ and therefore this matrices are similar.
Answer: Using the characteristic polynomial:
$$\chi_B(x)=\det(B-xI)=det(PAP^*-xI)=\det P\det(A-xI)\det P^*=\chi_A(x)$$
and I leave for you some justifications to do.

Similar Eigenvalues


Understanding the GRS test of the Capital Asset Pricing Model

Question: I posted a similar question on Quantitative Finance Stack Exchange a while ago. It has not received any answers, thus I am reposting a version of it here in the hope of finding a larger pool of experts.
Suppose we are interested in testing the Capital Asset Pricing Model (CAPM) using data on $N$ assets observed for $T$ time periods. The CAPM states that
$$
E(R^{ei})=\beta_i E(f) \tag{*}
$$
where $R^{ei}$ is the excess return on an asset $i$, $f$ is the excess return on the market portfolio and $\beta_i\equiv\frac{\text{Cov}(R^{ei},f)}{\text{Var}(f)}$. While the CAPM is a single-period model, let us assume it works in $T$ time periods so that $E(R^{ei}_t)=\beta_i E(f_t)$ where $E(f_t)\equiv E(f)\equiv\lambda$. Let us also assume the relevant covariances and variances and thus also $\beta$s are constant over time, too.
According to Cochrane "Asset Pricing" (2005) Chapter 12, the Gibbons, Ross and Shanken (GRS) test of the CAPM (GRS, 1989) amounts to running $N$ time series regressions of the form
$$
R^{ei}_t=\alpha_i+\beta_i f_t+\varepsilon^i_t \tag{12.1}
$$
and testing the joint hypothesis $H_0\colon \alpha_1=\dots=\alpha_N=0$.
Now, I do see how setting $\alpha_1=\dots=\alpha_N=0$ in $(12.1)$ and integrating w.r.t. $f_t$ implies $(*)$. However, I do not think that $(*)$ implies $\alpha_1=\dots=\alpha_N=0$ in $(12.1)$. I am only able to get the following:
$$
R^{ei}_t = 0+\beta_i E(f)+\varepsilon^i_t. \tag{12.1'}
$$
It is as if we have measurement error in $(12.1)$ compared to $(12.1')$, as $f_t$ gets in place of $E(f)$.
How do I reconcile $(12.1)$ with $(12.1')$ so that the GRS test makes sense?
Comment: You are correct. Testing the CAPM is fraught with difficulties, largely due to it being a one-period model. However, if we don't model as in 12.1 when applying as a multiperiod model, then we are assuming all assets' expected returns are constant through time, and that is clearly false.
Comment: The one-period capm assumes investors adjust starting prices so that the capm will hold in the future single period. Such price adjustments are not included in the (one-period) capm period. (In effect, the one-period capm is untestable.) Yet multi-period testing is forced to include all price movements by necessity, contradicting the capm. Hence, the reason why the capm frequently fails when tested this way.
Comment: @GrahamBornholt, thanks for you input! Your points make sense in general. However, I wonder if they help solve this exact issue. It seems to me the CAPM + the GRS test amount to a well defined set of assumptions (the CAPM) and a well defined procedure (the GRS test) that purports to achieve a specific goal (test the validity of the CAPM). It is my concern that the assumptions + the procedure do not actually facilitate goal achievement for reasons outlined above. Could you point out exactly which part of my argument is mistaken?
Comment: I don't think you are mistaken. When testing the Capm, it is normal to use what used to be called the 'market model' (your 12.1)
Comment: The market model is used in a variety of tests of the capm, not just the GRS test.
Comment: @GrahamBornholt, well, my problem with $(12.1)$ is that it is at odds with the CAPM implications as I try to explain in my post. $(12.1)$ is implied by $E(R^{ei})=\beta_i f$ that is a single factor model or a single index model (if I got my terminology right), not $E(R^{ei})=\beta_i E(f)$ that is the CAPM.
Comment: If the expected excess market return is constant over time, and the alphas of 12.1 are all zero, then the expectation of both sides of 12.1 reduces to the capm formula *.
Comment: @GrahamBornholt, this is about sufficient versus necessary conditions. To test A via B, we must have A imply B. Then if B gets rejected, we get to reject A, too; if not, not. Now, here we have a different case: we are testing A via B where B implies A. If we reject B, this does not mean we can reject A. Thus rejecting $(12.1)$ does not allow us to reject the CAPM.
Comment: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/142877/discussion-between-graham-bornholt-and-richard-hardy).
Comment: @GrahamBornholt, see Matthew Gunn's answer. My mistake was the following. It is true that we *could* consider testing A via B where B implies A. If we reject B, this does not mean we can reject A. However, that is not the piece of information we should care about. What is relevant is that not B (namely, $\alpha_i\neq 0$ in $(12.1)$) implies not A (CAPM does not hold), and that is what gets exploited in the test.
Comment: His answer expands on my 5th comment above. 12.1 is the market model (MM). I thought your issue was that any test showing not all alphas are zero in the MM relied on the assumption that the MM holds, and you were concerned that the capm did not imply the MM assumption upon which the test is based. Glad that is clarified.
Comment: @GrahamBornholt, I am glad, too. Thanks for your help!
Answer: For each asset $i$, consider the time-series regression to estimate $\alpha_i$ and $\beta_i$:
$$ R^e_{it} = \alpha_i + \beta_i f_t + \epsilon_{it} $$
Take expectations to get rid of the error term and get:
$$ \operatorname{E}[R^e_{i}] = \alpha_i + \beta_i \operatorname{E}[f]$$
Meanwhile (*) says $\operatorname{E}[R^e_{i}] = \beta_i \operatorname{E}[f]$. This can only be true if $\alpha_i = 0$. If $\alpha_i \neq 0$ then $\operatorname{E}[R^e_i] \neq \beta_i \operatorname{E}[f]$. If you can reject $\forall_i \alpha_i = 0$ then you can reject the factor model that you're testing.
As a practical matter, the way these tests work out is that if you have portfolios or assets with any interesting and significant spread in expected returns then you can typically reject any asset pricing model. You can statistically reject the Fama-French three factor model. The more practical but fuzzy test is whether the model is telling you something useful about the data! Even though you can reject the Fama-French 3 Factor Model with the right test assets, that model captures something about the cross-section of expected returns that's empirically there while the CAPM is just entirely in tension with the data.
Comment: Nice! Now it seems pretty obvious in retrospect ;) Thank you so, so much for your answers to this and other questions of mine!!! You have provided a helping hand where I really needed one! (And as always, I will wait a few days before accepting the answers to give other users a chance.)


Transfer files directly from FTP to Azure File Storage without keeping them locally in memory or disk

Question: I have to transfer files from FTP to an Azure File Storage. My code works fine, but I'm transferring those files in memory which is not a best practice. So first I read the stream to an Byte array in memory. Then I upload the output to an Azure file storage.
Now I know it's better to do this asynchronicaly. But I don't know if this is possible and how to do it.
My code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage;
using System.Configuration;
using Microsoft.WindowsAzure.Storage.File;
using System.IO;
using Microsoft.Azure;
using System.Net;

namespace TransferFtpToAzure
{
    class Program
    {
        public static void Main(string[] args)
        {
            List<FileName> sourceFileList = new List<FileName>();
            List<FileName> targetFileList = new List<FileName>();

            string targetShareReference = ConfigurationManager.AppSettings["AzureShare"];
            string targetDirectoryReference = ConfigurationManager.AppSettings["Environment"] + "/" + Enums.AzureFolders.Mos + "/" + Enums.AzureFolders.In;
            string sourceURI = (ConfigurationManager.AppSettings["FtpConnectionString"] + ConfigurationManager.AppSettings["Environment"].ToUpper() +"/"+ Enums.FtpFolders.Mos + "/").Replace("\\","/");
            string sourceUser = ConfigurationManager.AppSettings["FtpServerUserName"];
            string sourcePass = ConfigurationManager.AppSettings["FtpServerPassword"];

            getFileLists(sourceURI, sourceUser, sourcePass, sourceFileList, targetShareReference, targetDirectoryReference, targetFileList);

            Console.WriteLine(sourceFileList.Count + " files found!");

            CheckLists(sourceFileList, targetFileList);
            targetFileList.Sort();

            Console.WriteLine(sourceFileList.Count + " unique files on sourceURI" + Environment.NewLine + "Attempting to move them.");

            foreach (var file in sourceFileList)
            {
                try
                {
                    CopyFile(file.fName, sourceURI, sourceUser, sourcePass, targetShareReference, targetDirectoryReference);
                }
                catch
                {
                    Console.WriteLine("There was move error with : " + file.fName);
                }
            }
        }

        public class FileName : IComparable<FileName>
        {
            public string fName { get; set; }
            public int CompareTo(FileName other)
            {
                return fName.CompareTo(other.fName);
            }
        }

        public static void CheckLists(List<FileName> sourceFileList, List<FileName> targetFileList)
        {
            for (int i = 0; i < sourceFileList.Count; i++)
            {
                if (targetFileList.BinarySearch(sourceFileList[i]) > 0)
                {
                    sourceFileList.RemoveAt(i);
                    i--;
                }
            }
        }

        public static void getFileLists(string sourceURI, string sourceUser, string sourcePass, List<FileName> sourceFileList, string targetShareReference, string targetDirectoryReference, List<FileName> targetFileList)
        {
            string line = "";
            /////////Source FileList
            FtpWebRequest sourceRequest;
            sourceRequest = (FtpWebRequest)WebRequest.Create(sourceURI);
            sourceRequest.Credentials = new NetworkCredential(sourceUser, sourcePass);
            sourceRequest.Method = WebRequestMethods.Ftp.ListDirectory;
            sourceRequest.UseBinary = true;
            sourceRequest.KeepAlive = false;
            sourceRequest.Timeout = -1;
            sourceRequest.UsePassive = true;
            FtpWebResponse sourceRespone = (FtpWebResponse)sourceRequest.GetResponse();
            //Creates a list(fileList) of the file names
            using (Stream responseStream = sourceRespone.GetResponseStream())
            {
                using (StreamReader reader = new StreamReader(responseStream))
                {
                    line = reader.ReadLine();
                    while (line != null)
                    {
                        var fileName = new FileName
                        {
                            fName = line
                        };
                        sourceFileList.Add(fileName);
                        line = reader.ReadLine();
                    }
                }
            }
            /////////////Target FileList
            CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
            CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
            //var test = fileClient.ListShares();
            CloudFileShare fileShare = fileClient.GetShareReference(targetShareReference);
            if (fileShare.Exists())
            {
                CloudFileDirectory rootDirectory = fileShare.GetRootDirectoryReference();
                if (rootDirectory.Exists())
                {
                    CloudFileDirectory customDirectory = rootDirectory.GetDirectoryReference(targetDirectoryReference);
                    if (customDirectory.Exists())
                    {
                        var fileCollection = customDirectory.ListFilesAndDirectories().OfType<CloudFile>();
                        foreach (var item in fileCollection)
                        {
                            var fileName = new FileName
                            {
                                fName = item.Name
                            };
                            targetFileList.Add(fileName);
                        }
                    }
                }
            }
        }

        public static void CopyFile(string fileName, string sourceURI, string sourceUser, string sourcePass, string targetShareReference, string targetDirectoryReference)
        {
            try
            {
                FtpWebRequest request = (FtpWebRequest)WebRequest.Create(sourceURI + fileName);
                request.Method = WebRequestMethods.Ftp.DownloadFile;
                request.Credentials = new NetworkCredential(sourceUser, sourcePass);
                FtpWebResponse response = (FtpWebResponse)request.GetResponse();
                Stream responseStream = response.GetResponseStream();
                Upload(fileName, ToByteArray(responseStream), targetShareReference, targetDirectoryReference);
                responseStream.Close();
            }
            catch
            {
                Console.WriteLine("There was an error with :" + fileName);
            }
        }

        public static Byte[] ToByteArray(Stream stream)
        {
            MemoryStream ms = new MemoryStream();
            byte[] chunk = new byte[4096];
            int bytesRead;
            while ((bytesRead = stream.Read(chunk, 0, chunk.Length)) > 0)
            {
                ms.Write(chunk, 0, bytesRead);
            }

            return ms.ToArray();
        }

        public static bool Upload(string FileName, byte[] Image, string targetShareReference, string targetDirectoryReference)
        {
            try
            {
                CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
                CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
                //var test = fileClient.ListShares();
                CloudFileShare fileShare = fileClient.GetShareReference(targetShareReference);
                if (fileShare.Exists())
                {
                    CloudFileDirectory rootDirectory = fileShare.GetRootDirectoryReference();
                    if (rootDirectory.Exists())
                    {
                        CloudFileDirectory customDirectory = rootDirectory.GetDirectoryReference(targetDirectoryReference);
                        if (customDirectory.Exists())
                        {
                            var cloudFile = customDirectory.GetFileReference(FileName);
                            using (var stream = new MemoryStream(Image, writable: false))
                            {
                                cloudFile.UploadFromStream(stream);
                            }
                        }
                    }
                }
                return true;
            }
            catch
            {
                return false;
            }
        }
    }
}

Answer: If I understand you correctly, you want to avoid storing the file in memory between the download and upload.
For that see:
Azure function to copy files from FTP to blob storage.
Comment: Indeed, that is what I want to avoid. Thought you needed to work with async methods :). Thank you!!
Comment: "Operation is not valid due to the current state of the object.", any idea?
Comment: @AaYy Consider asking a new question and make sure you include [mcve].
Comment: I actually did: https://stackoverflow.com/questions/54009633/file-transfer-from-ftp-directly-to-azure-storage-error
Comment: @AaYy, hello, can you un-delete the post? is it acceptable to use file.UploadText() to transfer file from FTP to file storage?
Comment: @IvanYang How does ` file.UploadText()` help? It requires you to keep the whole file contents in the memory the same way as the code in this question does. Plus string is even more memory inefficient than the byte array and has a more limited use.
Comment: Thanks Martin, it's just a workaround. And really don't know why the stream from FTP does not work with file storage, but can work with blob storage.
Comment: @IvanYang Workaround for what? The OP has a working code already (although memory-inefficient). OP does not need any workaround.
Comment: how about blob directly to sftp? is this possible without downloading the contents locally?
Comment: @l--''''''---------'''''''''''' You have posted a question on this: [Uploading a Azure blob directly to SFTP server without an intermediate file](https://stackoverflow.com/q/57065811/850848).
Answer: Using Azure Storage File Share this is the only way it worked for me without loading the entire ZIP into Memory. I tested with a 3GB ZIP File (with thousands of files or with a big file inside) and Memory/CPU was low and stable. I hope it helps!
var zipFiles = _directory.ListFilesAndDirectories()
    .OfType<CloudFile>()
    .Where(x => x.Name.ToLower().Contains(".zip"))
    .ToList();

foreach (var zipFile in zipFiles)
{
    using (var zipArchive = new ZipArchive(zipFile.OpenRead()))
    {
        foreach (var entry in zipArchive.Entries)
        {
            if (entry.Length > 0)
            {
                CloudFile extractedFile = _directory.GetFileReference(entry.Name);

                using (var entryStream = entry.Open())
                {
                    byte[] buffer = new byte[16 * 1024];
                    using (var ms = extractedFile.OpenWrite(entry.Length))
                    {
                        int read;
                        while ((read = entryStream.Read(buffer, 0, buffer.Length)) > 0)
                        {
                            ms.Write(buffer, 0, read);
                        }
                    }
                }
            }
        }
    }
}

Comment: You can use something like `await extractedFile.UploadFromStreamAsync(entryStream)` instead of a great part of your code.
Comment: Actually, that was my first attempt. But this method requires a MemoryStream already loaded in memory, otherwise the error "Operation is not valid due to the current state of the object." is fired.
Comment: Also, `UploadFromStreamAsync` cannot be used when the stream originates from a `WebRequest` (as I had) since the response stream has `CanSeek` set to false.


Probability of rolling 3 Dice .

Question: i have 3 fair cubic dice. I roll it for 3 times ( ie. 9 dice ) . What is the probability of getting atleast one dice facing with 6 in of those 3 trials ? Is that probability same for other numbers as well ?
Comment: Give us more information about what you tried!
Answer: Hint. Consider the complement: what is the probability of getting no $6$ in these $3\cdot 3$ rolls?
P.S. At the end you should find the required probability is $1-(5/6)^9$. This probability does not change if you replace $6$ with another number among $1,2,3,4,5$.

Probability of rolling 3 Dice .


Telegram bot send message in certain hour in ruby

Question: I would create a telegram bot in ruby that send a message in a determinate hour i decide. For example. It's 8 am and the bot send a message "Good morning". Of course every morning at the same time. It could be a kind of reminder. I can get the time in this way:
time = Time.now.strftime("%H:%M")

And i know that to send a message i can use the Telegram API like:
bot.api.send_message(chat_id: message.chat.id, text: "Hi")

I use this code when i have an interaction between the bot and a user. For example:
Telegram::Bot::Client.run(token) do |bot|

  bot.listen do |message|
    case message.text
    when "hi", 'hi@myBot'
      bot.api.send_message(chat_id: message.chat.id, text: "Hi, #{message.from.first_name}")
    end
....
....

but I don't know how do what i need because if i want to send a message i need the chat.id and in this case i get it only when i have an interaction with a user. So, is it possibile send a message for example when it's 8 am without any user interaction?
Comment: You have two options: store ids and send a message by a cron task or create a worker after every message received. Both of them requires user interaction.
Comment: The only way to send a message without interaction is to hardcode chat id. Do you want to send a message to a certain chat?
Comment: The bot could be in many groups. I can maybe store the chat id in some way and then use it but i don't know if could be the right way. Have you any idea how solve my problem?
Comment: So, shortly, I can't do what I want?
Answer: Use crono gem to perform asynchronous tasks.
Crono.perform(YourJob).every 1.day, at: {hour: 8, min: 00}
Comment: I have never use crono and it's my first experiment with ruby. Can you please explain how can I use it? Thank you very much
Comment: Just go the the gem git repository I linked above. The usage of the gem is described in details there.
Comment: Did I help you with your problem? Can you accept my answer? @End.Game


NSFetchRequest release causes app to crash

Question: The static analyzer keeps telling me that I have a +1 retain count for my request object and instruments tells me there is a leak there. However, no matter where I try to release it, it keeps crashing my app. It's the same with NSPredicate object. Please do help, I'm trying to meet a deadline.
// Fetch Requests

// Method that returns an array of NSManagedObjects in the managedObjectContext with a predicate of who ordered

- (NSArray *)fetchDataWithEntity:(NSString *)entity andSortKey:(NSString *)key andPerson:(Person *)whoOrdered

{

 NSFetchRequest *request = [[NSFetchRequest alloc] init];

 NSEntityDescription *entityDescription = [NSEntityDescription entityForName:entity

                                                                   inManagedObjectContext:managedObjectContext];

 request.entity = entityDescription;

 // Handling Sorting

 NSSortDescriptor *sortDescriptor = [[NSSortDescriptor alloc] initWithKey:key ascending:YES

                                                                                selector:@selector(compare:)];

 NSArray *sortDescriptors = [[NSArray alloc] initWithObjects:sortDescriptor, nil];

 [request setSortDescriptors:sortDescriptors];

 [sortDescriptor release];

 [sortDescriptors release];

 // Handling Predicate

 if (whoOrdered) {

      NSPredicate *predicate = [NSPredicate predicateWithFormat:@"%@ == whoOrdered", whoOrdered];

      [request setPredicate:predicate];

 }

 NSError *error = nil;

 NSArray *mutableFetchResults = [managedObjectContext executeFetchRequest:request error:&error];

 //[request release];

 [error release];

 if (mutableFetchResults == nil) {

      // Handle the error

 }

 return mutableFetchResults;

}
Answer: There is no need to release the error object since you are not the owner of it and you should release the request object just before returning the mutableFetchResults object since you are creating it using alloc and thus are the owner of it ...
Comment: Thanks for noticing that, but it still doesn't work. Where ever I try to release request, it crashes. I also get 56.73% [request setPredicate:[NSPredicate predicateWithFormat:@"...", whoOrdered]]; as a source of my leak.


Add space in string before each word

Question: I am looking for an elegant (at least a clean) way to add space between each word in a camel cased sentence.
Example ThisIsASentence should be transformed in This Is A Sentence
Answer: One of the many ways to do this: use a regular expression (NSRegularExpression) which matches a lowercase letter (or not an uppercase letter if you've got non-letters in the string) followed by an uppercase one and replaces them with the two matched characters with a space between them - i.e. exactly how you'd solve this problem in a decent text editor (TextWrangler et.al.)
If you have trouble implementing the code ask a new question showing your code and explaining your issues.
Answer: Simple solution with regular expression, it considers only the uppercase characters A-Z.
The regex pattern (?<!^)[A-Z] searches for a single uppercase character which is not the beginning of the string.
NSMutableString *camelCaseString = @"ThisIsASentence".mutableCopy;
NSString *pattern = @"(?<!^)[A-Z]";
NSRegularExpression *regex =  [[NSRegularExpression alloc] initWithPattern:pattern options:0 error:nil];
[regex replaceMatchesInString:camelCaseString options:0 range:NSMakeRange(0, camelCaseString.length) withTemplate:@" $0"];
NSLog(@"%@", camelCaseString);

Comment: Thanks for the suggestion, but regex are something I would like to avoid for such a simple thing.
Comment: You wrote *elegant* and *clean* ;-)
Comment: Finally I've just found this way http://stackoverflow.com/questions/1918972/camelcase-to-underscores-and-back-in-objective-c seems to do the job

Add space in string before each word


Is there a way to check if a member exists in a struct?

Question:
Possible Duplicate:
How to detect whether there is a specific member variable in class?

I'm adding features to a C++ library. A thing that'd come in handy was to check if a certain member exists in a struct (it exisits depending on the library version - unfortunately there is no "version" parameter in the library).
Here is a simplified example of what I'd like to do:
struct options {
    int option1;
    std::string option2;
    float option3; // might be included or not

    options();
    void random_method();
}

options::options() {
    option1 = 1;
    option2 = "foo";

    if( EXISTS(option3) ) { // Pseudo-Code -> can I do that at all?
        option3 = 1.1;
    }
}

Comment: See this: http://stackoverflow.com/questions/257288/is-it-possible-to-write-a-c-template-to-check-for-a-functions-existence/264088#264088. It's the same concept.
Comment: Does SFINAE work in compilers other than g++?
Comment: How can a struct "optionally" include a member?
Comment: Thanks a lot, that was helpful. @DanF : As I wrote: I want to write an addion that works with serveral versions of the library. In some version the struct has said member in others it does not. That's how.
Comment: @con-f-use That sounds like something that would be determined at compile-time, wouldn't it?
Answer: You could implement a pure virtual function in the parent structure and have the children implement it.
struct Parent
{
  virtual bool has_option(OPTION) const = 0;
};

struct Car : public Parent
{
  bool has_option(OPTION op) const
  {
     bool result = false;
     if (op == MOON_ROOF)
     {
         result = true;
     }
     return result;
   }
};


Why do we need "smaller $\sigma -$algebra" for random variable?

Question: Let $X$ a random variable. Why the $\sigma -$algebra generate by $X$ is interesting ? (i.e. $\sigma (X)$). For example, we throw a  dice. Let $X$ denote the result. Why $\sigma (X)$ is more significant than the Borel
$\sigma -$algebra ?
Comment: What is the "tribute" in this context?
Comment: What do you mean by "the Borel tribute"? Also I am not familiar with the term "tribute" in this context.
Comment: I guess a $\sigma$-algebra maybe?
Comment: Are you French ? The translation of tribu is $\sigma$-algebra, not tribute.
Comment: Sorry, yes, $\sigma -$algebra (yes I'm french).
Comment: If $X$ is a random variable defined on probability space $(\Omega,\mathcal A,P)$ then $\sigma(X)\subseteq\mathcal A$. In that sense $\sigma(X)$ is "smaller" than $\mathcal A$. But if $\mathcal A$ is not a collection of Borel sets (and mostly it is not) then there is no comparing with Borelsets (actually $\sigma(X)$ is the collection of preimages under $X$ of the Borel sets) at all.
Answer: If $E,F\in\sigma(X)$ and $E\neq F$ then exactly one of these events can occur.
If $B,C\in\mathcal B$ then $B\neq C$ does not exclude that the events $\{X\in B\}$ and $\{X\in C\}$ are the same (so that both occur or both occur not).
E.g. in the example you give we have $\{X\in(0.5,2)\}=\{X=1\}$.
It is more handsome to work with a collection $\sigma(X)=\{\{X\in B\}\mid B\in\mathcal B\}$ then with a family $(\{X\in B\})_{B\in\mathcal B}$. This especially in the knowledge that lots of family members can be the same.