Reload Keras-Tuner Trials from the directory

Question: I'm trying to reload or access the Keras-Tuner Trials after the Tuner's search has completed for inspecting the results. I'm not able to find any documentation or answers related to this issue.
For example, I set up BayesianOptimization to search for the best hyper-parameters as follows:
## Build Hyper Parameter Search
tuner = kt.BayesianOptimization(build_model,
                     objective='val_categorical_accuracy',
                     max_trials=10,
                     directory='kt_dir',
                     project_name='lstm_dense_bo')

tuner.search((X_train_seq, X_train_num), y_train_cat,
             epochs=30,
             batch_size=64,
             validation_data=((X_val_seq, X_val_num), y_val_cat),
             callbacks=[callbacks.EarlyStopping(monitor='val_loss', patience=3,
                                                restore_best_weights=True)])

I see this creates trial files in the directory kt_dir with project name lstm_dense_bo such as below:

Now, if I restart my Jupyter kernel, how can I reload these trials into a Tuner object and subsequently inspect the best model or the best hyperparameters or the best trial?
I'd very much appreciate your help. Thank you
Answer: I was trying to do the same thing. I was looking into the keras docs for an easier way than this but could not find one - so if any other SO-ers have a better idea, please let us know!

Load the previous tuner. Make sure overwrite=False or else you'll delete your trials.

workdir = "mlp_202202151345"
obj = "val_recall"
tuner = kt.Hyperband(
    hypermodel=build_model,
    metrics=metrics,
    objective=kt.Objective(obj, direction="max"),
    executions_per_trial=1,
    overwrite=False,
    directory=workdir,
    project_name="keras_tuner",
)

Look for a trial you want to load. Note that TensorBoard works really well for this. In this example, I'm loading 1a38ebaba07b77501999cb1c4ab9413e.

Here's the part that I could not find in Keras docs. This might be dependent on the tuner you use (I am using Hyperband):

tuner.oracle.get_trial('1a38ebaba07b77501999cb1c4ab9413e')
Returns a Trial object (also could not find in the docs). The Trial object has a hyperparameters attribute that will return that trial's hyperparameters. Now:
tuner.hypermodel.build(trial.hyperparameters)
Gives you the trial's model for training, evaluation, predictions, etc.
NOTE This seems convuluted and hacky, would love to see a better way.
Comment: Good catch! Should be max. I edited it.
Comment: Why are you minimizing recall?
Answer: j7skov has correctly mentioned that you need to reload previous tuner and set the parameter overwrite=False(so that tuner will not overwrite already generated trials).
Further if you want to load first K best models then we need to use tuner's get_best_models method as below
# This will load 10 best hyper tuned models with the weights
# corresponding to their best checkpoint (at the end of the best epoch of best trial).
best_model_count = 10
bo_tuner_best_models = tuner.get_best_models(num_models=best_model_count)

Then you can access a specific best model as below
best_model_id = 7
model = bo_tuner_best_models[best_model_id]

This method is for querying the models trained during the search. For best performance, it is recommended to retrain your Model on the full dataset using the best hyperparameters found during search, which can be obtained using tuner.get_best_hyperparameters().
tuner_best_hyperparameters = tuner.get_best_hyperparameters(num_trials=best_model_count)
best_hp = tuner_best_hyperparameters[best_model_id]
model = tuner.hypermodel.build(best_hp)

If you want to just display hyperparameters for the K best models then use tuner's results_summary method as below
tuner.results_summary(num_trials=best_model_count)

For further reference visit this page.
Comment: The output of `get_best_models` is a list sorted by objective metric. Calling the index `trial_id` is misleading.
Answer: Inspired by j7skov, I found that the models can be reloaded
by manipulating tuner.oracle.trials and tuner.load_model.
By assigning tuner.oracle.trials to a variable, we can find that it is a dict object containing all relavant trials in the tuning process.
The keys of the dictionary are the trial_id, and the values of the
dictionary are the instance of the Trial object.
Alternatively, we can return the best few trials by using tuner.oracle.get_best_trials.
To inspect the hyperparameters of the trial, we can use the summary method of the instance.
To load the model, we can pass the trial instance to tuner.load_model.
Beware that different versions can lead to incompatibilities.
For example the directory structure is a little different between keras-tuner==1.0 and keras-tuner==1.1 as far as I know.
Using your example, the working flow may be summarized as follows.
# Recreate the tuner object
tuner = kt.BayesianOptimization(build_model,
            objective='val_categorical_accuracy',
            max_trials=10,
            directory='kt_dir',
            project_name='lstm_dense_bo',
            overwrite=False)

# Return all trials from the oracle
trials = tuner.oracle.trials

# Print out the ID and the score of all trials
for trial_id, trial in trials.items():
    print(trial_id, trial.score)

# Return best 5 trials
best_trials = tuner.oracle.get_best_trials(num_trials=5)
for trial in best_trials:
    trial.summary()
    model = tuner.load_model(trial)
    # Do some stuff to the model

Answer: using
tuner = kt.BayesianOptimization(build_model,
                         objective='val_categorical_accuracy',
                         max_trials=10,
                         directory='kt_dir',
                         project_name='lstm_dense_bo')

will load the tuner again.
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answer: The answer from hgh worked perfectly for me using the Hyperband search but I had to add tuner.reload() as follows:
tuner = kt.Hyperband(
       hypermodel=build_model,
       objective= kt.Objective("val_categorical_accuracy", direction="max"),
       max_epochs=50,
       factor=3 ,
       seed = 17,
       hyperband_iterations=1,
       distribution_strategy=tf.distribute.MirroredStrategy(),
       directory='models',
       project_name=EXPERIMENT,
       tuner_id = EXPERIMENT,
       overwrite=False,
       logger=NeptuneLogger()
       )
# Return all trials from the oracle
tuner.reload()
trials = tuner.oracle.trials

# Print out the ID and the score of all trials
for trial_id, trial in trials.items():
       print(trial_id, trial.score)

hp = tuner.oracle.get_trial("0028").hyperparameters

model = build_model(hp)

Answer: I have found this way to load the keras_tuner trials into a tuner.
tuner = RandomSearch.load_from_dir('path/to/dir/tuner1')

and then you would continue with the training or select the best performing model as per your need.
N.B This works for RandomSearch but was not tested on your specific 'search' approach.


Tratamento de valor null c#

Question: Tenho um método que me gera um execption, como tratar isso?: agradeço
 public string CarregaProdutosDermaClube(string codigoproduto)
{
    //consulta os dados
    var tbuscar = new BuscaProdutosDermaClubeDAL();
    var retorno = tbuscar.ProdutoDermaClube(codigoproduto);

    if (string.IsNullOrEmpty(retorno.idproduto))
    {
        return "0";
    }
    else
    {
        return "1";
    }
}

Comment: `if (!string.IsNullOrEmpty(retorno))
return retorno.idproduto; else return 0;` não ficaria agradavel?
Comment: Esse código ta gerando exception? Ou você quer que gere se for Null? No else não deveria retornar o conteúdo da var `retorno`? Ta bem confuso a pergunta.
Comment: Precisa verificar se o retorno é nulo primeiro. Poderia fazer algo assim: retorno?.idProduto
Comment: gerando exception
Comment: Usando o VS 2015
Comment: @ if (!string.IsNullOrEmpty(retorno)) vou testar
Comment: Não deu certo, if (!string.IsNullOrEmpty(retorno)) a única que funcionou foi  if (retorno != null) {}
Answer: Utilizando if ternário
 public string CarregaProdutosDermaClube(string codigoproduto)
        {
            //consulta os dados
            return string.IsNullOrEmpty(new BuscaProdutosDermaClubeDAL().
                                 ProdutoDermaClube(codigoproduto)?.idproduto)? "0" : "1";
        }

Comment: Exatamente como eu faria.
Answer: Se você estiver utilizando Visual Studio a partir da versão 2015:
public string CarregaProdutosDermaClube(string codigoproduto)
{
    //consulta os dados
    var tbuscar = new BuscaProdutosDermaClubeDAL();
    var retorno = tbuscar.ProdutoDermaClube(codigoproduto);

    if (string.IsNullOrEmpty(retorno?.idproduto))
    {
        return "0";
    }
    else
    {
        return "1";
    }
}

Em versões anteriores:
public string CarregaProdutosDermaClube(string codigoproduto)
{
    //consulta os dados
    var tbuscar = new BuscaProdutosDermaClubeDAL();
    var retorno = tbuscar.ProdutoDermaClube(codigoproduto);

    if (retorno == null || string.IsNullOrEmpty(retorno.idproduto))
    {
        return "0";
    }
    else
    {
        return "1";
    }
}

Comment: Isso tem a ver com a versão da linguagem, não do Visual Studio.
Comment: De fato é verdade, mas é mais facil explicar assim do que fazer a pessoa identificar a versão do C# que está instalada no visual studio.
Answer: Normalmente deve-se utilizar um bloco try/catch quando há troca de informações entre aplciações, no seu caso, entre seu software e o banco de dados. É também uma boa prática utilizar um framework de logs para gravar exceções geradas no seu sistema.
Outro ponto é o retorno do seu método, como ele retorna "0" ou "1" nada melhor que utilizar o tipo bool (true/false) indicando se o produto foi ou não carregado com sucesso.
 public bool CarregaProdutosDermaClube(string codigoproduto)
        {
            try
            {
                //consulta os dados
                var tbuscar = new BuscaProdutosDermaClubeDAL();
                var retorno = tbuscar.ProdutoDermaClube(codigoproduto);
                return retorno != null ? true : false;

            }
            catch (Exception ex)
            {
                // você pode logar a exceção capturada em "ex"
                return false;
            }
        }

Answer: Avaliando o código pode estar ocorrendo erro no caso da variável "retorno" = null.
Um exemplo para tratar essa situação é utilizar o ? antes do ".". Exemplo:
string idProduto = Convert.ToString(retorno?.idProduto);

Para leitura do código também sugiro utilizar o ternário, conforme já foi explicado nos comentários. Exemplo:
return string.IsNullOrEmpty(idProduto) ? "0" : "1";

Tratamento de valor null c#


Assert interface to its type

Question: I can't gracefully get pixels of an image as array in general case.
f, err := os.Open(imgPath)
check(err)
defer f.Close()
img, _, err := image.Decode(bufio.NewReader(f))
check(err)
pixels, err := getPixels(img)
check(err)
// Logic with pixels.

Now function getPixels looks like this:
func getPixels(img image.Image) ([]uint8, error) {
    if i, ok := img.(*image.NRGBA); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.Alpha); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.Alpha16); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.CMYK); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.Gray); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.Gray16); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.NRGBA64); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.Paletted); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.RGBA); ok {
        return i.Pix, nil
    } else if i, ok := img.(*image.RGBA64); ok {
        return i.Pix, nil
    }
    return nil, fmt.Errorf("unknown image type %T", img)
}

But I think this is ugly. Golang knows type of image and I would prefer something like this:
func getPixels(img image.Image) ([]uint8, error) {
    if i, ok := img.(eval(fmt.Sprintf("%T", img))); ok {
        return i.Pix, nil
    }
    return nil, fmt.Errorf("unknown image type %T", img)
}

I also can't assert to reflect.TypeOf(img). Maybe there is a way to get type from reflect.Type interface?
Comment: What kind of processing are you trying to do with the `Pix` field? I'm curious because though the `Pix` field for different image types is of type `[]uint8`, the actual contents of the slice is not consistent. For example, the `CMYK` image type stores values for C, M, Y and K one after the other in the slice for each pixel, but `Alpha` stores just one value for each pixel. That's why the `image.Image` interface has an `At()` method returning a `color.Color` value for any given pixel.
Comment: Then I calculate average color. Getting color of every pixel with `img.At(x, y).RGBA()` is slower than processing `[]uint8`: ~530ms vs ~315ms on an image with resolution 3840x1200. I handle a lot if images so I have decided to use img.Pix. But Python with library "pillow" is faster: ~215ms :).
Comment: Do you need that performance gain? If yes, are you sure that you're getting the right results without taking into account the image type? Especially for image types that have multiple components (eg. CMYK, NRGBA64 etc) where I assume you need to compute the average for each component.
Comment: I could do it without optimization, but why not for fun. Images are png, jpeg or tiff, colorful (4 channels) or grey (1 channel). Results checked with the help of imagemagick.
Answer: Your big if ... else structure could be simplified by using a type switch like this:
func getPixels(img image.Image) ([]uint8, error) {
    switch i := img.(type) {
    case *image.NRGBA:
        return i.Pix, nil
    case *image.Alpha:
        return i.Pix, nil
    case *image.Alpha16:
        return i.Pix, nil
    case *image.CMYK:
        return i.Pix, nil
        // ...
    }
    return nil, fmt.Errorf("unknown image type %T", img)
}

Where you still have to list all possible types, but it's nicer.
Since all image implementations are struct pointers having a field named Pix, you may use reflection to get that field. This implementation will handle future image implementations without any change (if they will also be structs with a Pix field).
This is how it would look like:
func getPix(img image.Image) ([]uint8, error) {
    v := reflect.ValueOf(img)
    if v.Kind() == reflect.Ptr {
        v = v.Elem()
    }

    if v.Kind() == reflect.Struct {
        pv := v.FieldByName("Pix")
        if pv.IsValid() {
            if pix, ok := pv.Interface().([]uint8); ok {
                return pix, nil
            }
        }
    }

    return nil, fmt.Errorf("unknown image type %T", img)
}

Testing it:
fmt.Println(getPix(&image.NRGBA{}))
fmt.Println(getPix(&image.RGBA{}))

type unknownImage struct{ image.Image }
fmt.Println(getPix(unknownImage{}))

Output (try it on the Go Playground):
[] <nil>
[] <nil>
[] unknown image type main.unknownImage

Comment: @Andrey Go is a statically typed language. You can only type-assert to a concrete type or another interface type, but the type must be known at compile-time. Use the 2nd approach with reflection.
Comment: Yes, `switch` slipped from my mind. Switch looks a bit nicer. But is it possible in golang to assert interface to its type ("%T")? In general case there may be tens of different types.

Assert interface to its type


How to obtain a template parameter pack of integers int a function?

Question: Say I want to have a function like:
void reset()
{
    (std::get</*pack of integers from 0 to 4*/>(someTuple).reset(), ...);
}

I did it with a helper function like this:
template < size_t... Indexes >
void reset(std::integer_sequence<size_t, Indexes...>)
{
    (std::get<Indexes>(someTuple).reset(), ...);
}
void reset()
{
    reset(std::make_integer_sequence<size_t,5>{});
}

But can I achieve this with just one function? The attempt like:
template < size_t... Indexes >
void reset(std::integer_sequence<size_t, Indexes...> = std::make_integer_sequence<size_t,5>{})
{
    (std::get<Indexes>(someTuple).reset(), ...);
}

failed as the call reset() resolved to reset<>(std::integer_sequence<size_t>) rather than to reset<0,1,2,3,4>(std::integer_sequence<size_t,0,1,2,3,4>). I am using the latest snapshot of GCC-7.0.0
Comment: what you have is the way to go, imo
Answer: You can achieve something like this with a defaulted template parameter:
template <class S = std::make_index_sequence<5>>
struct reset; // intentionally undefined

template <size_t... Is>
struct reset<std::index_sequence<Is...>> {
    template <class Tuple>
    static void impl(Tuple& t) {
        (std::get<Is>(t).reset(), ...);
    }
};

reset<>::impl(someTuple);

but not in a function (since such a thing would require partial specialization of function templates, which is not supported by the language).
Instead, just create a helper lambda for inline parameter pack unpacking:
void reset() {
    indexer<5>()([&](auto I){
        std::get(someTuple).reset();
    });
}

Answer: You could make use of generic lambda and some helper struct like this:
#include <utility>
#include <type_traits>
#include <tuple>

template <std::size_t N, class = std::make_index_sequence<N>>
struct iterate;

template <std::size_t N, std::size_t... Is>
struct iterate<N, std::index_sequence<Is...>> {
   template <class Lambda>
   void operator()(Lambda lambda) {
      lambda(std::integral_constant<std::size_t, Is>{}...);
   }
};

struct Foo {
   void reset() {}
};

void reset() {
   std::tuple<Foo, Foo, Foo> bar;
   iterate<std::tuple_size<decltype(bar)>::value>{}([&](auto... is) {
      (std::get<is>(bar).reset(), ...);
   });
}

int main() {
   reset();
}


c# mvc how to validate 3rd column Value Based on Two column

Question: Here I have three properties Like
Public Class Addition{
public Int Val1{get;se;}
public Int Val2{get;se;}
public Int Val3{get;se;}
}

In .cshtml Val1=10;Val2=20; If User Entre Val3=30 Then How can i validate Val3 Value Should Be LessThan or Equal to 30 Beacause (10+20=30)
Comment: use a custom validation attribute. I recommend you fluent validation .NET. refer this link https://stackoverflow.com/questions/8906228/how-to-validate-one-field-related-to-anothers-value-in-asp-net-mvc-3
Comment: Could you please give hint how can i bind three values at a time
Answer: What is the problem are you facing to solve it.
it shouldn't be complex
You can use
if(Val3 > (Val1 + Val2))
{
  //Show Your Message
  return;
}

Upvote if helps


No R, Como calcular a média de uma coluna com base em critério em outra coluna?

Question: Tenho duas colunas (A e B) quero calcular a média da coluna A para os elementos correspondentes apenas para os que na coluna B forem maior que 10 por exemplo.
Answer: É um problema de seleção de linhas de uma data frame por uma condição lógica:
set.seed(6480)    # Para ter resultados reprodutíveis

n <- 50
dados <- data.frame(A = runif(n, 0, 100), B = runif(n, 0, 40))

mean(dados[dados$B > 10, "A"])    # índice lógico
#[1] 51.62713

mean(dados$A[dados$B > 10])       # equivalente
#[1] 51.62713

Mas se a coluna B tiver valores NA o índice lógico não resulta, temos que usar which.
is.na(dados$B) <- sample(n, 10)        # fazer alguns B iguais a NA

mean(dados$A[dados$B > 10])            # veja o que dados$B > 10 dá
#[1] NA

mean(dados$A[which(dados$B > 10)])
#[1] 52.17357

EDIÇÃO.
Como o Flávio Silva diz em comentário, também se pode usar o argumento na.rm.
mean(dados$A[dados$B > 10], na.rm = TRUE)
#[1] 52.17357

Comment: muito obrigado!
Comment: Na função `mean(dados$A[dados$B > 10])`, acrescentar o `na.rm = T` não seria o suficiente, ficando `mean(dados$A[dados$B > 10], na.rm = T)`?
Comment: @FlavioSilva Certo, obrigado pela sugestão, também dá. Mas o problema a que eu me estava a referir é diferente. Quando um índce lógico tem valores `NA` deve-se usar `which`. Vou editar a resposta com a sua sugestão.
Comment: @RuiBarradas Eu perguntei mais pelo fato de buscar entender a diferença entre o `na.rm` e quando usar o `which`
Answer: Quando é data.frame, e tenho de tirar médias de varias colunas, eu uso o código colMeans e dentro da função coloco tipo: colMeans(dados[dados$coluna1=="A" & dados$coluna2=="0.01",]) normalmente ele irá obter as médias por colunas dos dados que atendam a este critério só ressalvo que na parte ,] depois desta virgula você não está discriminando quais colunas entram no calculo por isso irá se realizar com todas, o problema é quando nas colunas tem caracteres ("A","NOME") pois o código costuma não realizar o calculo de médias com fatores, para isso voce pode simplesmente concatenar os respectivos números das colunas as quais você deseja os resultados. Ex: nova.tabela.contendo.as.médias<-colMeans(dados[dados$coluna1=="A" & dados$coluna2=="0.01",c(4,1,3,2,5,7)]),


AWStats log format for tomcat access logs which has X-Forwarded-For

Question: What should be the AWStats log format for below tomcat access logs ?
I tried these formats but the external IP addresses are not coming into AWStats reports.
LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %referer %other %other"

LogFormat="%other %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %host_proxy"

tomcat valve settings:
pattern="%h %l %{USER_ID}s %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"  "X-Forwarded-For=%{X-Forwarded-For}i"  "JSESSIONID=%{JSESSIONID}c" %D"

Log entry:
127.0.0.1 - - [04/Nov/2013:13:39:55 +0000] "GET / HTTP/1.1" 200 12345 "https://www.google.com/url?some_url" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36"  "X-Forwarded-For=real_ip, proxy_server_internal_ip"  "JSESSIONID=-" 12345

Answer: $ echo $STR
127.0.0.1 - - [10/Nov/2013:04:14:03 +0000] "GET /XXXX/js/service.js HTTP/1.1" 200 656 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0" "X-Forwarded-For=188.8.131.52, 10.2.0.121" "JSESSIONID=YYYY" 2

$ echo $STR | perl -pe 's/X\-Forwarded\-For=([\d\.]+).*?\"/X-Forwarded-For=$1"/g'
127.0.0.1 - - [10/Nov/2013:04:14:03 +0000] "GET /XXXX/js/service.js HTTP/1.1" 200 656 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0" "X-Forwarded-For=22.214.171.124" "JSESSIONID=YYYY" 2

Comment: Welcome to server fault!  While it's great to give examples, it's nice to explain what is being done so it's useful to future readers as well.  Additionally, I've cleaned up the formatting - note that you can mark code segments with a 4-space indent.
Comment: This perl one liner extracts first IP address from X-Forwarded-For field in access log
Answer: Thanks for the help.
Did one more trick, awstat wants host_proxy logs as 220.127.116.11, 10.2.0.121, so changed your perl command as below which changes "X-Forwarded-For=22.214.171.124, 10.2.0.121" to "188.8.131.52, 10.2.0.121" without quotes and extra spaces removed.
perl -pe 's/ "X-Forwarded-For=([\d.,\s]+).*?" /$1/g'
After the changes the logs are provided to awstat for processing.


Restrictions of measure converges to original measure?

Question: Let X be a subset of $\mathbb{R}^d$ and $(X,A,\mu)$ be a measure space. Let  $K_n$ be compact sets that exaust $X$, i.e. $X=\bigcup\limits_{n\ge 1}K_n$. I was wondering if the restriction $\mu_n:=\left.\mu\right\rvert_{K_n}$ was narrowly converging to $\mu$. By definition of narrow convergence, I have to show that $\int fd\mu_n\to \int fd\mu$ for every $f$ bounded and continuous but I am not sure if the claim is true and how to start.
Comment: weak converges (test functions are $\mathbb{C}_b(X)$ works for measures of total finite variation. In the case you re describing, I think value convergence is more appropriate (test functions are $\mathcal{C}_{00}(X)$.
Answer: If $\mu$ is an infinite measure then $\int f d\mu$ need not exist for $f$ bounded and continuous, so I will assume that $\mu$ is a  finite positive measure. I will also assume that $K_n$ increases to $X$. In this case $\int fd\mu_n=\int_{K_n} fd\mu \to \int fd\mu$ by DCT.
If $K_n$'s are not increasing the result is fasle since $K_1$ may be repeated infinitely many times in the sequence $(K_n)$.


Custom prompt using Net::Appliance::Session

Question: I am trying to ssh into a Cisco switch using the Perl module Net::Appliance::Session but I am running into an issue due to the nature of the expected prompt.  The switch prompt is the following

User Name:

so when the basic username prompt is expected it is not returned properly.  Looking at the documentation for Net::Appliance::Session it talks about custom prompts but I am unsure how to implement it.  It refers to Phrasebook.
If any one has any examples of how this may be achieved, it would be greatly appreciated.
To avoid cross posting I had asked a similar question at perlmonks yesterday, here's the link to that thread  http://perlmonks.com/?node_id=1199230
Thanks in advance.
Code example ---
#!/usr/bin/perl

use warnings;
use strict;
use Net::Appliance::Session;

my $host = '<my_host>';
my $user = '<my_user>';
my $pass = '<my_password>';

my $session = Net::Appliance::Session->new( personality => 'cisco', host => $host, transport => 'SSH', );
$session->connect( username => $user, password => $pass ) or die "Cannot Connect! : $!";

my @output = $session->cmd('show version');

$session->close;

print join("\n",@output);

UPDATE
I've changed my code to use a custom Phrasebook as described  Net::CLI::Interact::Phrasebook
My phrase book file looks like :
prompt user
    match /[Uu]ser\s*[Nn]ame:\s*?$/

prompt pass
    match /[Pp]assword: ?$/

prompt prompt
    match / ?$/

prompt basic
    match /> ?$/

macro paging
    send terminal length %s

And my updated script looks like :
#!/usr/bin/perl

use warnings;
use strict;
use Net::Appliance::Session;

my $host = '<my_host>';
my $user = '<my_user>';
my $pass = '<my_password>';

my $session = Net::Appliance::Session->new({
    add_library     => '/home/edunn/switch',
    personality     => 'custom',
    transport       => 'SSH',
    host            => $host,
    connect_options => { opts => [ '-q', ], },
});

$session->set_global_log_at('debug');

$session->connect($user,$pass ) or die "Cannot Connect! : $!";

my @output = $session->cmd('show version');

$session->close;

print join("\n",@output);

But now I get the following error

   [0.001516] p  finding prompt
   [0.006937] t  creating Net::Telnet wrapper for ssh
   [0.007171] t    command expands to:  ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -q -l cisco 192.168.2.17
   [1.325789] d    SEEN:

   [1.341702] p   reading phrasebook /home/edunn/switch/custom/login
   [1.342318] p    storing prompt user
   [1.343439] p    storing prompt pass
   [1.344178] p    storing prompt prompt
   [1.344881] p    storing prompt basic
   [1.345578] p    storing macro paging
   [1.346353] p    nope, doesn't (yet) match basic
   [1.346577] p    nope, doesn't (yet) match pass
   [1.346789] p    nope, doesn't (yet) match user
   [1.347021] p   hit, matches prompt prompt
   [1.347718] p   prompt has been set to (?-xism: ?$)
   [1.348436] e  running macro paging
   [1.348655] e   macro params are: 0
   [1.349293] e  executing actions
   [1.349814] e    dispatching to execute method
   [1.350736] t   callback received for send
   [1.350957] t    queueing data for send: "terminal length 0"
   [1.351277] t   callback received for match
   [1.351576] d    SEEN:

   [1.351772] t    output matched (?-xism: ?$), storing and returning
   [1.352222] p   setting new prompt to (?-xism: ?$)
   [1.352437] p   prompt has been set to (?-xism: ?$)
   [1.353120] e  running macro paging
   [1.353323] e   macro params are: 24
   [1.353860] e  executing actions
   [1.354344] e    dispatching to execute method
   [1.355083] t   callback received for send
   [1.355290] t    queueing data for send: "terminal length 24"
   [1.355535] t   callback received for match
   [1.438431] d    SEEN:
t
   [1.438731] t    output matched (?-xism: ?$), storing and returning
   [1.439212] p   setting new prompt to (?-xism: ?$)
   [1.439426] p   prompt has been set to (?-xism: ?$)

Looking at the error it looks like it is matching User Name: and Password: but then it fails looking for something called prompt which I have no idea where it is picking it up from.
Thanks for looking.
Answer: answer to update
From the Phrasebook docs I see

The last essential prompt is of course a simple command line prompt match, and this should be named generic.
# example only!
prompt generic
    match /> ?$/

Adding that should take care of your current issue.
I looked for how they handle inheritance so you could get all of the existing stuff like the basic prompt for free.  My best guess is that it is based on the directory structure.  If you place you custom directory with its pb in a directory below cisco with its pb or a copy of it that might work.
original answer
It looks like from the Phrasebook docs that personality => 'cisco' is sort of generic.  You might have better luck with IOS or one of the other variants.  I'm not that familiar with Cisco's lineage so you should look for what makes the most sense to you.
But upon further examination that may not help much.  From the source it looks like it is looking for Username or username, but not User Name.  Can you change the prompt to conform to one of the ones it is looking for?
Comment: I ended solving this problem by upgrading my switch firmware.  The updated firmware allows me user authentication via ssh so I no longer need to rely on the prompts to login.  Thank you for all your help.
Comment: I am trying to create a custom prompt as described in the Phrasebook docs but now get a different error.  I will post my changed above
Comment: updated the code one more time.  I no longer get an error but I also don't get any output from the issued command.
Comment: You should change `prompt basic` or `prompt prompt` into `prompt generic` depending on which one is the normal prompt.    Reread the part of the docs I quoted.
Comment: if I replace prompt prompt with prompt generic it errors out, it is still looking for prompt prompt
Comment: Have you tried starting with one of the existing profiles?


How can i rewrite the views table output for endless scroll

Question: I use the module views load more and for an horizontal infinite scroll i want only one row with many columns. When i click on the load more button, views rendered every new page as a table. Like this:

... 1. Page ...

... 2. Page ...

I try to rewrite the views-view--VIEWNAME.tpl.php and the views-view-grid--VIEWNAME.tpl.php but when i reduce this i get a output. Like this:

    ... 1. Page ...

    ... 2. Page ...

I reduce the views-view-grid--VIEWNAME.tpl.php to:
<?php foreach ($rows as $row_number => $columns): ?>
    <?php foreach ($columns as $column_number => $item): ?>
      <td <?php if ($column_classes[$row_number][$column_number]) { print 'class="' . $column_classes[$row_number][$column_number] .'"';  } ?>>
        <?php print $item; ?>

    <?php endforeach; ?>
<?php endforeach; ?>

and add the table and tr tags to the views-view--VIEWNAME.tpl.php.
Don`t know where the tbody tag comes. Same Problem here:
https://drupal.org/comment/4641640#comment-4641640
Answer: You can use this module Views Infinite Scroll. It will help to integrate  jquery.autopager plugin with your drupal website.

Views Infinite Scroll provides a Pluggable pagers that enables views infinite scrolling ( autopaging, depaging, endless pages ...you name it) by using jquery.autopager plugin. using this module with views page display will load another page whenever the user reaches bottom of the page.
Comment: Thanks for your comment. But, I want rewrite the views output for horizontal scroll. Views Infinity Scroll works only for vertical scrolling.
Comment: see this comment https://drupal.org/comment/6204150#comment-6204150


How to make Nginx redirect to 404 if subdomain doesn't exist in localhost domain

Question: This is my localhost server configuration for Nginx. This is at /etc/nginx/sites-enabled/development/localhost
server {
    listen 80 default_server;
    server_name localhost;
    root /var/www/html;
    index index.nginx-debian.html;

    location / {
      internal;
    }

    error_page 404 /404.html;
    location /404.html {
      root /var/www/html/errors;
      internal;
    }

}

If I call http://locahost in URL bar, a 404 page error is shown. But should show index.nginx-debian.html. But if I call http://localhost with a valid subdomain, it works. Like that http://users.localhost, defined in /etc/nginx/sites-enabled/development/users.
I want to show Nginx default page if I only call http://localhost, 404.html if subdomain doesn't exist, and another any page if I call http://localhost correctly with a defined subdomain.
Answer: I solved as follow.
At /etc/nginx/sites-enabled/development/default.
server {
  listen 80 default_server;
  server_name *.localhost *.<host-name>;
  root /var/www/html/errors/404;

  location / {
    index subdomain.html;
  }

}

At /etc/nginx/sites-enabled/development/localhost.
server {
    listen 80;
    server_name = localhost;
    root /var/www/html;
    index index.nginx-debian.html;

    # to URI errors caused by paths after domain
    error_page 404 /domain.html;
    location = /domain.html {
        root /var/www/html/errors/404;
        internal;
    }
}

I don't know if this solution is the best, so if someone has a better solution, post here. Thanks.

How to make Nginx redirect to 404 if subdomain doesn’t exist in localhost domain


Services module: what is the syntax of node.index parameters?

Question: I'm trying to get the list of certain type nodes using xmlrpc and POST method. I'm constructing xml like this:
<code><?xml version="1.0" encoding="UTF-8"?>
<methodCall>
<methodName>node.index</methodName>
<params>
   <param>
      <value>
         <string>0</string>
      </value>
   </param>
   <param>
      <value>
         <string>*</string>
      </value>
   </param>
   <param>
      <value>
         <string>type=forum</string>
      </value>
   </param>
</params>
</methodCall>
</code>
yet in return I get nodes of various types. I guess that type=forum is bad. What is correct syntax of this parameter?
Answer: I usually call node.index this way
<code>$nodes = xmlrpc($this->server_endpoint, Array('node.index' => Array(0,'nid,uuid,vid', array('type' => $contentType), PHP_INT_MAX)), $this->headers);
</code>
the parameters being:

Page number (usually I set it to zero and set the number of nodes per page to PHP_INT_MAX. This because I found page support buggy in a previous version, I haven't retested it recently)
Array with fields to retrieve (only that you find in the node DB table are available. Careful with it, less fields means better request speed)
Array with content type(s) to retrieve
Number of nodes in the index list per page

I dont'know how exacly goes in XML, but I think that it should be easy enough to adapt to that syntax, if necessary.


Python modding - prevent dangerous scripts to be imported?

Question: I want to allow users to make their own Python "mods" for my game, by placing their scripts in a special folder which the game "scans" for Python modules and imports.
What would be the simplest way to prevent "dangerous" scripts from being imported? I don't want people complaining to me that they used someone's mod and it erased their hard drive.
Things I would like to limit is accessing/modifying/creating any files outside of their folder and connecting to the internet/downloading/sending data. If you can thik of anything else, let me know.
So how can this be done?
Comment: There is no easy way to provide a reasonable amount of sandboxing. *Especially* if the files will require some amount of disk access.
Comment: Could you handle disk access exploits by using permissions? e.g., When the user installs your game, create an account that's only allowed to read/modify files in the game's directory, and make all extensions run on that account?
Comment: Then what are the hard ways? I thought there would at least be a way to prevent specific modules to be imported in the future.
Comment: pypy provides more sophisticated sandboxing, but I have no information on how to embed it into your game.
Comment: pypy is not an option as the game engine requires the official python version, sorry.
Comment: Permissions? What permissions? And what account?
Comment: I believe ocern is suggesting you create a very restricted os user that has due to OS restrictions a very limited access to harddrive. Which may be a good idea.
Comment: Ar you guys talking about an operating system user account and it's permissions? How does that make sense unless everyone is using my PC to play my game?
Comment: see: http://stackoverflow.com/questions/861864/is-there-a-safe-subset-of-python-for-use-as-an-embedded-scripting-language
Answer: Restricted Python seems to able to restrict functionality for code in a clean way and is compatible with python up to 2.7.
http://pypi.python.org/pypi/RestrictedPython/
e.g.

By supplying a different __builtins__ dictionary, we can rule out unsafe operations, such as opening files [...]
Answer: The obvious way to do it is to load the module as a string and exec it.  This has just as many security risks, but might be easier to block by using custom globals and locals.  Have a look at this question - it gives some really good guidance on this.  As pointed out in Delnan's comments, this isn't completely secure though.
You could also try this.  I haven't used it, but it seems to provide a safe environment for unsafe scripts.
Comment: Note that this will merely make things slightly harder for malicous modders. It's by no means a comprehensive sandbox.
Comment: @delnan: Correct. I'll add that to my answer.
Comment: @user975135: `execfile('modscript.py', custom_globals, custom_locals)` where `custom_globals` and `custom_locals` are dicts.  Read the link.
Answer: There are some serious shortcomings for sandboxed python execution.  aquavitae's answer links to some good discussion on the matter, especially this blog post.  Read that first.
There is a kernel of secure execution within cPython.  The fundamental idea is to replace the __builtins__ global (Note: not the __builtin__ module), which informs python to turn on some security features; making a handful of attributes on certain objects inaccessible, and removing most of the implementation objects from the interpreter when evaulating that bit of code.
You'll then need to write an actual implementation; in such a way that the protected modules are not the leaked into the sandbox.  A fairly tested "file" replacement is provided in the linked blog.  Getting a look on that might give you an idea of how involved and complex this problem is.

So now that you have understood that this is a challenge in python; you should take a look at languages with sandbox execution as a core feature, such as Lua, which is very popular in games.
Answer: Giving them python execution and trying to limit what they do is asking for trouble. See this SO question for discussion and a pointer to a good article. (You would presumably disable "eval", but it wouldn't make much difference in practice.
My suggestion: Turn the question around. Your goal is to provide them with scripting facilities so they can enhance the game. Find or define an interpreter for a suitable scripting language that has the features you need, and use it to execute their scripts. For example, you could support data persistence in a simple keystore model, without giving them file creation access. Or give them a command to create files but ensure it only accepts a path-less filename. The essential thing is to ensure that there is NO way for them to execute python commands directly.


Invariant ring is a finitely generated $K$-algebra, confusion regarding the proof

Question: I'm currently studying quotients of affine varieties by finite groups. I use the book "Algebraic Geometry" by J. Harris (http://userpage.fu-berlin.de/aconstant/Alg2/Bib/Harris_AlgebraicGeometry.pdf). On page 123, Harris shows that the quotient of an affine variety under the action of a finite group does exist. For that, he shows that the invariant ring $A(X)^G$ is a finitely generated $K$-algebra. However, a few steps are not very clear to me:

I understand why $K[x_1, ... ,x_m]^{S_m}$ is a finitely generated $K$-algebra: it gets finitely generated by the elementary symmetric polynomials $y_i$, okay. But now my question is: How can we see that $K[x_1, ... , x_m]$ is a finitely generated $K[y_1, ... , y_m]$-module? That's not clear for me at all. That would mean that we could write any $P\in K[x_1, ... , x_m]$ as $P = \sum_{i=1}^k s_i f_i $, where $s_i$ are symmetric polynomials, so $s_i \in K[y_1, ... , y_m]$, and the $f_i \in K[x_1, ... , x_m]$ are the generators. But what about polynomials of the form $P_n (x) = x_m^n + 1$? I am very sure that these polynomials cannot all be written in the from as above, because they are not symmetric, so the $s_i$ must be constant functions, but $K[x_1, ... , x_m]$ is not a finitely generated $K$-module, but that would contradict the highlighted sentence. Can please someone tell me what I am doing wrong?
Then, I am also not quite sure, why it follows that also $K[x_1, ... , x_m]^G$ must be a finitely generated $K[y_1, ... , y_m]$ as well. Is it maybe because it is a submodule of $K[x_1, ... , x_m]$, and maybe $K[x_1, ... , x_m]$ is a Noetherian $K[y_1, ... , y_m]$-module? I'm completely stumped.
Answer: Consider the polynomial $\prod_{i=1}^m (T-x_i) \in K[x_1,\dots, x_m]^{S_m}[T]$, this is monic and has $x_1, \dots, x_m$ as its roots. It follows that all the $x_i$ are integral over $K[x_1,\dots, x_m]^{S_m}$. A finitely generated integral ring extension is finite as a module. As $K[x_1,\dots, x_m]^{S_m}$ is a Noetherian ring, any finitely generated module over $K[x_1,\dots, x_m]^{S_m}$ is Noetherian, thus $K[x_1, \dots ,x_m]^G$ is finite over $K[x_1,\dots, x_m]^{S_m}$ as well.
Comment: Thank you very much!
Comment: Can I ask you one question? There are a few more details about Harris' proof which i dont understand. Would it be better if i created one big post containing all of my questions, or is it better to create a post for each question individually? I dont want to spam this site with my questions, but i also dont want to post a huge wall of text and pictures, because i think short and precise questions are much better.


How to Send Collection View Cell Text via Segue

Question: I am trying to send a label that is contained within a collection view cell to another view controller with a segue.
My plan is that when a user taps on the collection view cell, the app then segues to the next view controller where the navigation bar's title displays the text of the label in the collection view cell selected.
I have tried this:
func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath) {
    let cell = collectionView.dequeueReusableCell(withReuseIdentifier: "cell", for: indexPath) as! CCCollectionViewCell

    //itemSelected = items[indexPath.row] as String

    itemSelected = cell.pLabel.text!

    print(itemSelected)
}

and in prepareForSegue I have not written any code as I am not sure how this works.
I commented out the block '..items[indexPath.row] as String' because it won't show the label and added the print function to see what will output but it only outputs the name given in the storyboard.
I am very new to Xcode so am not familiar with didSelect and prepareForSegue. All I am trying to do is to send the text within a collection view cell to another view controller with a segue.
Answer:   func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath) {

        self.performSegue(withIdentifier: "contentVideoSegue", sender: indexPath)

}

override func prepare(for segue: UIStoryboardSegue, sender: Any?) {

    if segue.identifier == "contentVideoSegue"{
        let selectedIndexPath = sender as? NSIndexPath
        let videoContentVC = segue.destination as! VideoContentController
        videoContentVC.text = items[selectedIndexPath.row] as String
    }
}

in the hope of helping :)
Comment: please add some description explaining the answer
Comment: Firstly in the didSelectItemAt indexPath method is calling sugue. After this call override func prepare(for segue) method starts to work. @jjj
Answer: From your code you are not calling the performSegue(withIdentifier:sender:) so you probably have created segue from the CollectionViewCell to DestinationViewController. So get the indexPath using this cell in prepareForSegue method.
override func prepare(for segue: UIStoryboardSegue, sender: Any?) {

     if let cell = sender as? UICollectionViewCell,
        let indexPath = self.collectionView.indexPath(for: cell) {

         let vc = segue.destination as! SecondViewController //Cast with your DestinationController
         //Now simply set the title property of vc
         vc.title = items[indexPath.row] as String
     }
}

Answer: So you don't need to set up the cell in didSelect because you're already doing that in cellForItemAtIndexPath.
Rather you'll want to call performSegue(withIdentifier: "SegueName", sender: indexPath) in your didSelectItemAtIndexPath. Then in your prepareForSegue:
override func prepare(for segue: UIStoryboardSegue, sender: Any?) {

    guard let indexPath = sender as? IndexPath else { return }

    let collectionCell = collectionView.cellForItem(at: indexPath)
    let textToPass = collectionCell.textLabel.text

    let detailVC = segue.destination as? DetailViewController
    detailVC.passedInString = textToPass
}


Python: How do you specify the parameter type of a Class function to the current class?

Question: I'm writing a set of custom classes with inheritance using Python. Here is the code, simplified:
class Parent(object):
    __slots__ = ['a','b']

    def __init__(self, a:int, b:int):
        self.a = a
        self.b = b

    def myfunc(self,s)->int:
        return self.a + s.a

class Child(Parent):
    __slots__ = ['c']

    def __init__(self, a:int, b:int, c:int):
        super().__init__(a,b)
        self.c = c

    def myfunc(self,s)->int:
        return self.a + s.a + s.c

def main():
    dad = Parent(1,2)
    son = Child(4,8,16)

    print(dad.myfunc(son))
    print(son.myfunc(son))
    print(son.myfunc(dad))

if __name__ == "__main__":
    main()

Upon running the function, I get this output (which I understand):
5
24
Error: Parent has no attribute 'c'

The myfunc() definition in Child requires that both self AND s are instances of Child.
My ideal functionality for this code is that I want to implement method overloading. When son.myfunc(dad) is called, I want it to check the type of dad. If dad is also a Child, it should use Child's definition of myfunc(). Otherwise, if dad isn't a Child, son should fall back to using the definition of myfunc() in Parent. So, for the above code, I'd want:
5
24
5

I would've thought that I could implement this functionality as follows, specifying the parameter types of myfunc():
class Parent(object):
    __slots__ = ['a','b']

    def __init__(self, a:int, b:int):
        self.a = a
        self.b = b

    def myfunc(self, s:Parent)->int:
        return self.a + s.a

class Child(Parent):
    __slots__ = ['c']

    def __init__(self, a:int, b:int, c:int):
        super().__init__(a,b)
        self.c = c

    def myfunc(self,s:Child)->int:
        return self.a + s.a + s.c

def main():
    dad = Parent(1,2)
    son = Child(4,8,16)

    print(dad.myfunc(son))
    print(son.myfunc(son))
    print(son.myfunc(dad))

if __name__ == "__main__":
    main()

However, this doesn't work. Python and my IDE tell me that "Parent is not defined" by the time I use it to specify the input type of Parent's myfunc(), and the same for Child. I think I get what it's saying ('Parent' won't mean anything until the compiler finishes reading Parent, so it can't be specified as the parameter type), but I don't know how to work around this.
How can I implement my intended functionality? Is there some keyword like 'this' or 'self' that can be used to refer to the current class's type for these specifications? Even if method overloading doesn't work in this way- is there a way for a single class (Parent) to specify itself as the parameter type of one of its functions?
Edit: I'm aware that Python parameter types are annotations for readability and error-checking, and they don't actually cause errors/failures if called with a different type. The second code segment was created to help communicate what I want to have occur. I'm aware I might have to add a check like isinstance() at the start of myfunc, but
(a) I would like to avoid it if there is an alternative
(b) if I did discover in the body of Child's myfunc() that I wanted to call the Parent's instead, I'm not certain of how I'd do that.
Answer: You have two separate issues:

In order to type-hint a method that returns its own class (forward reference) you should use a string:
def myfunc(self, s: "Parent") -> int:

You can use isinstance to check for type, i.e.
def myfunc(self, s: "Child") -> int:
    return (self.a + s.a + s.c) if isinstance(s, Child) else (self.a + s.a)

As a side note the type of s in your Child.myfunc should probably be Parent.
Update: Per your comment, the following should be possible:
class Parent(object):
    # ...

    def _myfunc(self, s: "Parent") -> int:
        return self.a + s.a

    def myfunc(self, s: "Parent") -> int:
        return type(s)._myfunc(self, s)

class Child(Parent):
    # ...

    def _myfunc(self, s: "Parent") -> int:
        return self.a + s.a + s.c

Comment: 1. Thank you for referring me to forward referencing! That's something I've been looking for for a long time, but didn't know how to articulate it to search it.

2. I'd considered having to use isinstance(), but was hesitant about it. In my real code, each version of myfunc() is about 50 lines long. Is there a way I can just call the Parent's version of myfunc() on the Child from within Child's myfunc() (as opposed to having to copy all that code again as you've done)?
Comment: This sounds like an xy problem. Your simplified example is, well, too simple and smells like bad design to me. See my updated answer for a method that abuses Python's flexibility.


Is BsonRepresentation(BsonType.ObjectId) always unique?

Question: Just a straight forward question is
<code>[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
</code>
always unique? If not is there a way to generate a unique id for my user?
Answer: The rule formation for a objectid is:
The 12-byte ObjectId value consists of:

a 4-byte timestamp value, representing the ObjectId's creation, measured in seconds since the Unix epoch
a 5-byte random value
a 3-byte incrementing counter, initialized to a random value

If you want to know more, there is this article that explains in detail how to avoid that situation.


React Dropdown Value Not Set to default after Clear Filter

Question: I had a Company Dropdown Component and Sidebar.js to call the component. When i clicked on clear filter button, the products displayed are set to default value. But the value& text of Company dropdown still displaying old value. For example,"Honda".
The behaviour seems correct as the products will display to default value but doesnt work on dropdown
Sidebar.js
import Category from "./Category/Category";
import Model from "./Model/Model";
import Company from "./Company/Company";

import "./Sidebar.css";

const Sidebar = ({ handleChange, selectedCompany, resetPage }) => {
  const [companyValue, setCompanyValue] = useState("");

  useEffect(() => {
    console.log("companyValue:", companyValue);
  }, [companyValue]); // Log companyValue whenever it changes

  const handleCompanyChange = (event) => {
    const newValue = event.target.value;
    setCompanyValue(newValue); // Update the companyValue state
    handleChange(event);
    resetPage();
  };

  const handleModelChange = (event) => {
    handleChange(event);
    resetPage();
  };

  const handleCategoryChange = (event) => {
    handleChange(event);
    resetPage();
  };

  const clearAllFilters = () => {
    setCompanyValue("");
    handleChange({ target: { name: "company", value: "" } });
    handleChange({ target: { name: "model", value: "" } });
    handleChange({ target: { name: "category", value: "" } });
    resetPage();
  };

  return (
    <>

        <Company handleChange={handleCompanyChange} value={companyValue} />
        <Model
          handleChange={handleModelChange}
          selectedCompany={selectedCompany}
        />
        <Category handleChange={handleCategoryChange} />
        <button onClick={clearAllFilters}>Clear All Filters</button>

    </>
  );
};

export default Sidebar;

Company.js

import React from "react";
import "./Company.css";

function Company({ handleChange }) {
  const handleCompanyChange = (event) => {
    handleChange(event); // Invoke the handleChange function for company changes
  };

  return (

Company

        <select onChange={handleCompanyChange} className="company-dropdown" name="company">
          <option value="">All</option>
          <option value="Honda">Honda</option>
          <option value="Toyota">Toyota</option>
        </select>

  );
}

export default Company;

Comment: Hi Jesscy ! Have you checked the below answer? Let us know please :)
Answer: This is because you need to pass the state companyValue to Company and use it there as the value of select to be completely synced:
<Company handleChange={handleCompanyChange} value={companyValue} />

function Company({ handleChange, value }) {
  const handleCompanyChange = (event) => {
    handleChange(event); // Invoke the handleChange function for company changes
  };

  return (

Company

        <select  value={value} onChange={handleCompanyChange} className="company-dropdown" name="company">
          <option value="">All</option>
          <option value="Honda">Honda</option>
          <option value="Toyota">Toyota</option>
        </select>

  );
}


Plotting surface and curve in 3D and a curve in xy-plane, all in the same plot

Question: To illustrate an optimization problem, I want all of this in the same 3D plot:

A surface.
A curve in the xy-plane.
A curve/path on the surface which marks out the points on the surface that lies directly above the curve in the xy-plane.

This is my code so far:
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator
import numpy as np
from mpl_toolkits import mplot3d

fig, ax = plt.subplots(subplot_kw={"projection": "3d"})
X = np.linspace(-5,5,100)
Y = X
X, Y = np.meshgrid(X, Y)
Z = 50 - X**2 - Y**2

#Plotting curve on the surface
ax = plt.axes(projection='3d')

yline = np.linspace(-5,5,100)
xline = -np.sqrt(4/(2+yline**2)) #the x-values of the curve in the xy-plane
zline = 50 - xline**2 - yline**2

ax.plot3D(xline, yline, zline, "black")

surf = ax.plot_surface(X, Y, Z, cmap=cm.coolwarm)
ax.set_zlim(0, 50)

#Plotting curve in xy-plane
a = 5
g = 1 - 2*X - X*Y**2
plt.contour(X,Y,g, [a], offset=0)

plt.show()

Here is the plot from two different angles:

Some problems:

First of all, it seems like the axes have been numbered twice. Is that because I make a meshgrid, and later on use ax.plot3D? That I use two different ways of plotting something, and as a consequence make the 3D space twice?
The path on the surface appears weakly. Is there a way to make the path more visible?
From the picture in bird perspective, we see that the path does not lie directly above the curve in the xy-plane. What would be easier, was if Python had a built-in function who could project the curve in the xy-plane directly onto the surface. Am I missing something here? Does anyone know of such a function?

These questions might be dummy questions, but answers and tips are highly appreciated!
Answer:
The code creates two axes objects (both assigned to the ax variable) in the same figure. This is not needed and results in double ticks marks.

To make the path on the surface more visible, plot it with a higher zorder.

The curve on the surface does not overlap with the curve on the xy plane because these are different curves. To plot the projection of the surface curve on the xy plane, set all z-coordinates of the curve to 0.

Below is the code with these changes.
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator
import numpy as np
from mpl_toolkits import mplot3d

fig, ax = plt.subplots(subplot_kw={"projection": "3d"})
X = np.linspace(-5, 5, 100)
Y = X
X, Y = np.meshgrid(X, Y)
Z = 50 - X**2 - Y**2

yline = np.linspace(-5, 5, 100)
xline = -np.sqrt(4 / (2 + yline**2))
zline = 50 - xline**2 - yline**2
ax.plot3D(xline, yline, zline, "b", zorder=10)

surf = ax.plot_surface(X, Y, Z, cmap=cm.coolwarm, alpha=0.7)
ax.set_zlim(0, 50)

#Plotting curve in xy-plane
ax.plot3D(xline, yline, 0, "k")

plt.show()

Comment: I see my mistakes now. Thank you so much! Been stuck on this for days, so I really appreciate it!


If a song is in a certain key, can you add an instrument or vocal to the song / track as long as they are in the same scale of that key

Question: If a song is in a certain key, can you add an instrument or vocal to the song / track as long as they are in the same scale of that key.
So for example, if a song is in key D major, you can add any instrument/vocal as long as the key is in the D major scale, so E minor, B minor,
G Major, A Major, etc, etc.
I'm sure this is an easy question, but I'm very new to music and scales.
Thanks
Comment: Welcome! Basically, yes, when a song is in a certain key, which most are, then you can add a track, or play along with it, **in** that key. That will sound best. If what you're saying is that any track in a particular key will *only* contain those chords you mention, then not necessarily. There are many questions asked here about that subject, and so many are based on the premise that only diatonic notes and chords must be used in a song.If that's where you're coming from, then the answer's no! You're also confusing chords and scales. They're related, but different.
Comment: If it is a non-transposing instrument and you don't mind some weird sounds and strange voice leading: yes.
Comment: Have you tried it?
Comment: thirds and sixths: The Musical Dilettante: A Treatise on Composition by J. F. Daube (s. my answer ...)
Answer: You can add whatever you want, but it might not sound the way you expect it to.
I assume that the effect you want is for it to sound "consonant" (ie no dissonances) so that it sounds pleasing to the ear following our Western popular music convention.
In that case, it's not just about the key: you also need to follow the same tempo and the same chord progression.
Following your D major example. If you add a track/melody that's playing on the notes of a D major chord while your song is playing an A7 chord, the tension effect might be broken. Conversely, if your new track is playing on an A7 chord when your song playing over a D major you might have a dissonant effect.
Also, the tempo. If the tempo of the new tracks are mis-aligned, you will get an unpleaseant "off-tempo" feeling.
Many accomplished musicians, especially coming from a jazz background, can easily improvise a new tune ontop of an existing chord progression. However, this takes either some good musical hearing or a fair amount of training.
Answer: The instrument should not matter as long as it is tuned to western scales.  The whole concept of key as we use it typically refers to the 12 TET chromatic tuning in this day and age.  So as long as the instrument is capable of playing or being tuned in this way and the player knows how to play it then yes.  A counter example might be a Turkish instrument called the Saz.  This is a guitar like instrument that has 1/4 tone frets on it.  They even make modern electric versions of it.  So if you invited a Turkish Saz player to lay down tracks there may be some conflict, or it may sound very cool.  Another counter example might be fret less instruments where you can adjust your fingering and play in Just tuning rather than equal tempered.
The quality of the tone is related to the instrument and some instruments are better suited for certain ranges (octaves).  But this is not an issue of key signature and being in tune but rather of orchestration and having the whole band sound good together.  For example composers and arrangers (producers too) invest time in deciding which instruments should play which parts of a score.  Even in a small rock band guitar, bass, and keyboards can clash when playing in key if they the mix of ranges is not diverse, i.e. no contrast.  Even when everyone is playing in the same key it is no guarantee that they will sound good together.
Answer: To make your first experience of adding a second instrument to any melody you can play or write a seconde voice following the melody in thirds or sixths without any problem. It will sound a little bit more classical and folk music (school music)  than pop or rock style: e.g. Silent Night, O how joyfully, Panis Angelicus, or keep the second voice on the same tone and skip to the next chord like Something stupid (Frank and Nancy Sinatra).
Zauberflöte, Mozart :

Panis Angelicus, César Franck

Don't cry for me Argentina:

Swiss and Austria folk music:

or if you like this (old German hits:)

or find a second voice to this song:
Answer:
If a song is in a certain key, can you add an instrument or vocal to the song / track as long as they are in the same scale of that key?

If I understand this question correctly, the answer is no.
It seems that you are asking whether any two tracks will necessarily sound good together if they are in the same key.  Being in the same key is only a very small element out of the many that determine whether two parts fit together, whether they are melodies or chord progressions.
It seems like you might be under the impression that every pitch used in a given key sounds harmonious when combined with any other pitch from that same key, but that is not the case.  Two arbitrarily chosen pitches from the same key might therefore clash with one another.  Composers writing harmony parts know which notes to choose to avoid such clashes, or even to capitalize on such clashes to create dissonant tension that resolves to consonance, but if you pick just any other melody written in the same key it is possible that the dissonant moments will occur randomly, without structure, resulting in an incoherent mess.


Difficulty understanding Chi-Squared p-values in this case

Question: Forgive me, but I'm new to this. I have two columns of categorical data. The data represents access to a particular resource, where a 1 indicates that the individual accessed the resource and a 0 indicates they did not. Each column represents a different sample of the population. I am using a chi-squared test on the contingency table. I am hoping to investigate whether the two samples do in fact come from the same population. I find that my P value is large (like p = 0.65). Do I have evidence to suggest that the two samples come from the same population, or at least, do I have evidence to suggest that they do not come from different populations? I'm struggling to wrap my head around the interpretation of the various hypotheses.
Comment: Chi-square is a test of association. That's not exactly the same as whether they come from the same population. But your notion is one of "accepting the null" which is against the rules. You can only "fail to reject" the null.   So, you do not have evidence that sample is related to access.
Comment: I see, but I think I'm still not quite clear - I do not have evidence that sample is related to access - this would mean that I do not have evidence to say that because someone is from sample A, therefore their access is likely to be X? Or, more importantly, I do not have evidence to state that which sample you come from affects how you accessed?
Comment: Both of those. In fact, with only two samples, those two statements are equivalent.
Comment: @peterfloam thank you. If you post that as an answer I will accept it
Answer: While you're doing a test of association, that's the same thing as a test of homogeneity of proportions (in spite of many people suggesting there's a distinction; there isn't -- assuming none of the population proportions are exactly $0$, the hypotheses of independence and homogeneity of proportions are different ways to say the same thing about the population - each implies the other, it follows from the definition of independence in probability).
However, even if you could establish that two sets of proportions are homogeneous, that doesn't of itself seem to establish that the samples came from the same population, only that the population proportions happened to be the same. The distributions are the same, the populations they were drawn from may be distinct collections; they might be clearly separated - e.g. in time and space - and may have always been so. The two statements are not always logically equivalent so a little care may be required in how we express it.

I find that my P value is large (like p = 0.65). Do I have evidence to suggest that the two samples come from the same population, or at least, do I have evidence to suggest that they do not come from different populations?

Neither of those! You lack evidence against $H_0$. While the data are consistent (in a particular sense) with homogeneous proportions, the data are also consistent with a range of proportions that are not homogeneous (there are hypotheses distinct from that $H_0$ that would also not be rejected in a test on the data you have; if your samples are small there may be a very wide range of such potential hypotheses). Clearly if there are mutually-exclusive possible nulls that would not be rejected you can't reasonably claim that non-rejection of a particular one of them is the case.

I am hoping to investigate whether the two samples do in fact come from the same population.

Generally speaking, you can't establish that the population parameters (population proportions) are identical from a non-exhaustive sample.
Answer: Chi-square is a test of association. That's not exactly the same as whether they come from the same population. But your notion is one of "accepting the null" which is against the rules. You can only "fail to reject" the null. So, you do not have enough evidence that sample is related to access to be able to reject the null.
Comment: "Don't have evidence" isn't correct, either.  What if the underlying counts run into the billions (to take one extreme)?  Few people would seriously claim a large p-value is a lack of evidence!  More accurately, this result merely shows there isn't sufficient evidence to reject the null.
Comment: Thanks both. FWIW, I have only a few dozen data points, so not really a concern about billions, but I see your point :)
Comment: That's true. I edited my answer.


Como restaurar do Git um arquivo modificado que deu erro?

Question: Turma, fiz um commit (e depois o push) de um arquivo no git até onde estava tudo bem. Quando voltei para continuar o trabalho, fiz algumas alterações que resultaram em erros e bugs. Dessa vez não fiz commit ainda e quero restaurar tudo que fiz até o último commit pois me perdi nas alterações e quero recomeçar do último commit correto.
Comment: exato! quero recomeçar de onde parei no último commit. o que eu fiz depois deu erro e não consigo encontrar o erro.
Comment: git reset --hard, talvez?
Comment: Vc quer descartar as alterações atuais e deixar igual no último commit que vc deu push?
Comment: Certo! Caso alguma resposta tenha resolvido seu problema considere marcar como aceita.
Answer: Para descartar TUDO o que vc fez localmente:
git reset --hard

Para descartar todas as alterações de um arquivo específico:
git checkout -- <arquivo>

Para colocar na pilha tudo o que vc fez e ter possibilidade de recuperar posteriormente:
git stash

Answer: step 1: "esconder" as modificações que você ainda não comitou.
git stash save <descrição>

step 2: resetar o ultimo commit da branch levando as modificações dele para staging area.
git reset HEAD~

aqui (step 2) caso queira, você pode commitar novamente as modificações ou remove-las de vez com: git checkout -- < arquivo>
step 3 (opcional): recuperar as modificações "escondidas" no step 1 e aplicar novamente sob o código.
git stash list

lista todas as stashs feitas
git stash apply stash@{numero-da-stash}

aplica a stash selecionada sob o código
ao aplicar a stash, podem acontecer conflitos que devem ser resolvidos
WARNING
Depois que você fizer as alterações que deseja e fizer um próximo commit, ao tentar realizar o push novamente, você receberá um erro porque a arvore local estará diferente da remota dado que você fez um push e depois resetou o commit localmente. Para proseguir com o push você pode usar:
git push --force

antes de dar esse comando certifique-se de que sua branch está organizada com os commits que você deseja, porque essa branch substituirá a remota


ERRO LARAVEL: No application encryption key has been specified

Question: Bom dia, estou tentando baixar um template neste link: https://itsolutionstuff.com/post/php-laravel-57-create-admin-panel-exampleexample.html
 Porém na hora do comando : php artisan key:generate ele gera um erro No application encryption key has been specified, pesquisei e a única solução que encontro é  rodas o cmando php artisan key:generate que é justamente o erro, alguém pode dar um help?
Answer: Tente
php artisan key:generate
php artisan config:cache
php artisan cache:clear

Comment: Boa tarde, o erro ocorre quando executo justamente o php artisan key:generate , depois mais nehum comando funciona.
Answer: Existe o arquivo .env no projeto? Caso exista, existe nele a variável APP_KEY?
O que o comando php artisan key:generate faz é gerar uma chave aleatória para a variável APP_KEY.
Caso o comando php artisan serve esteja em execução, você precisará reiniciar o servidor.
Comment: Alguém consegue me ajudar com isto?
Answer: Você tem que instalar ou atualizar o composer antes:
composer install

ou
composer update


Distance between two categorical distributions

Question: I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same.
There are lots of distance measures that I am aware of (e.g. list of statistical distance measures), but many seem to be tailored to work with continuous distributions. Is there a go to distance measure or test with categorical data? (Literature welcome)
Also, ideally I would want to compute a formal test (say p-values), but I assume I can always run a permutation test?
Comment: Geometric and Harmonic mean similarities, Bhattacharyya distance, Hellinger distance, Chi-square distance for probabilities, Pearson/Neyman chi-square divergence (asymmetric and symmetric), Kullback–Leibler asymmetric divergence (Information gain), Kullback–Leibler symmetric divergence (Jeffreys divergence), K-divergence (asymmetric), K-divergence (symmetric, Topsoe distance), Jensen difference (Information radius), Taneja distance.
Comment: Do you want a distance between two probability vectors of the same lenght? Such as, to compare (.10, .25, .25, .20, .20) with (.15, .25, .10, .30, .20)? They are empirical distributions of some 5-category categorical distribution.
Comment: ... you may find formulas on my web-page (link on my profile), in collection "Various proximities", macro !KO_proxqnt, find in .docx document.
Comment: Great! My question was then whether any of those is better suited for categorical data than the rest?
Comment: @ttnphns yes. See my answer to BruceET below
Comment: All - for categorical data. k categories, two k-length vectors of frequencies or probabilities (proportions summing to 1) to compute a (dis)similarity between.
Answer: I assume you have two samples of categorical data and want to know if they were taken from the same distribution. Perhaps most commonly, to compare nominal categorical samples, one can use a chi-squared test.
For the fictitious data below, levels of the categorical variables are
denoted by $1$ through $5,$ but those are not ordered labels.
Suppose we have populations with two different probability distributions
$p_1$ and $p_2$ as follows:
p1 = c(.1,.1,.1,.2,.5)  # heavy emphasis on last category
p2 = c(.1,.2,.2,.2,.3)  # less heavy

Use R to sample data vectors x1 and x2 (of different sizes)
both from population with probabilities $p_1.$
set.seed(2021)
x1 = sample(1:5,1000,rep=T,prob=p1)
t1 = tabulate(x1, nbins=5); t1
[1] 105  99  85 196 515             # tabulation of first sample
x2 = sample(1:5,1500,rep=T,prob=p1)
t2 = tabulate(x2, nbins=5); t2
[1] 157 154 132 289 768             # tabulation of second

Then take a sample 'y` from the population with probabilities $p_2.$
y = sample(1:5,1500,rep=T,prob=p2)
t = tabulate(y, nbins=5); t
[1] 155 287 320 309 429

Use a chi-squared test to compare samples x1 and x2, using
the appropriate table TABs of counts:
TBLs = rbind(t1,t2);  TBLs
TBLs
   [,1] [,2] [,3] [,4] [,5]
t1  105   99   85  196  515
t2  157  154  132  289  768

With P-value above 5% the test finds no difference between these
two samples from the same population.
chisq.test(TBLs)

        Pearson's Chi-squared test

data:  TBLs
X-squared = 0.18744, df = 4, p-value = 0.9959

By contrast, with P-value near $0,$ a chi-squared test
overwhelmingly rejects the null hypothesis that the two
samples were taken from the same population.
TBLd = rbind(t2,t);  TBLd
TBLd
   [,1] [,2] [,3] [,4] [,5]
t2  157  154  132  289  768
t   155  287  320  309  429

chisq.test(TBLd)

        Pearson's Chi-squared test

data:  TBLd
X-squared = 215, df = 4, p-value < 2.2e-16

The chi-squared statistic $H = \sum_{i=1}^2\sum_{j=1}^5 \frac{(X_{ij} = E_{ij})^2}{E_{ij}}$ (labeled X-squared in output) can be used as a measure of disagreement between the two samples tested. Here, $i=1,2$ sample rows and $j=1,2,3,4,5$ column categories of the table and the $E_{ij}$ are found in the usual way,
using row and column totals.
For example, in the second chi-squared test above, the expected counts (assuming the null hypothesis to be true) of the ten cells are shown below:
chisq.test(TBLd)$exp
   [,1]  [,2] [,3] [,4]  [,5]
t2  156 220.5  226  299 598.5
t   156 220.5  226  299 598.5

In particular, $E_{11} = \frac{(1500)(302)}{3000} = 156.0.$ (In this simple example, this amounts to the average of $157$ and $155.)$ If all of the $E_{ij} > 5,$ then $H \sim\mathsf{Chisq}(\nu),$ where $n = (r-1)(c-1),$ for a table with $r$ rows and $c$ columns.
Comment: For samples, the chi-squared statistic $H$ is a measure of the _disagreement_ between the two samples. I did not refer to the Wikipedia article; Unless I missed something, it does not seem to say much about about the comparisons you want to make. Perhaps look for 'effect size' [here](https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/NCSS/Chi-Square_Effect_Size_Calculator.pdf)
Comment: Thanks! To clarify, rather than the samples, I was interested in actual measures of distance between (empirical) probability distributions.  E.g. should I use [total variation distance](https://en.wikipedia.org/wiki/Total_variation_distance_of_probability_measures) with such a discrete distribution? Or are there results suggesting some particular distance measure works better with categorical data? (things like Kolmogorov–Smirnov don't work great)


Create new product variation and add selected variation to cart

Question: Create new product with two variations and add selected to the cart
Desired outcome:
Form selects between variations and adds the corresponding variation to cart.
Current outcome:
The second variation is added to the cart regardless of which is selected
Notes: Adding the variation_id should be pretty straight forward but since the variation id has yet to be created I need a way to associate the variation terms in the form with the newly assigned variation_id
<form name="new-prod" method="post" action="">
<input type="radio" name="attribute_pa_size" value="11oz" id="pa_size_v_11oz">
<input type="radio" name="attribute_pa_size" value="15oz" checked="checked" id="pa_size">

PHP:
$post_size = $_POST['attribute_pa_size'];

Multiple variations created:
First Variation
$parent_id = $product_ID;
$variation = array(
'post_title'   => 'Product #' . $parent_id . ' Variation',
'post_content' => '',
'post_status'  => 'publish',
'post_parent'  => $parent_id,
'post_type'    => 'product_variation'
);
$variation_id = wp_insert_post( $variation );
update_post_meta( $variation_id, '_regular_price', 21.95 );
update_post_meta( $variation_id, '_price', 21.95 );
update_post_meta( $variation_id, 'attribute_' . $mythis->size_tax, '15oz' );
WC_Product_Variable::sync( $parent_id );

Second Variation
$parent_id = $product_ID;
$variation = array(
'post_title'   => 'Product #' . $parent_id . ' Variation',
'post_content' => '',
'post_status'  => 'publish',
'post_parent'  => $parent_id,
'post_type'    => 'product_variation'
);
$variation_id = wp_insert_post( $variation );
update_post_meta( $variation_id, '_regular_price', 18.95 );
update_post_meta( $variation_id, '_price', 18.95 );
update_post_meta( $variation_id, 'attribute_' . $mythis->size_tax, '11oz' );
WC_Product_Variable::sync( $parent_id );

Add variation to cart
$quantity     = 1;
$variation    = $post_size;
$woocommerce->cart->add_to_cart( $product_ID, $quantity, $variation_id, $variation  );

Answer: Replaced all references of $variation_id with $elevenvariation_id for the first variation.
Then replaced the "add variation to cart" block with the following, and it managed to work from my initial testing:
  $quantity     = 1;
  $variation    = $post_size;

$answer = $_POST['attribute_pa_size'];
if ($answer == "15oz") {
$woocommerce->cart->add_to_cart( $product_ID, $quantity, $variation_id, $variation  );
}
    else {
        $woocommerce->cart->add_to_cart( $product_ID, $quantity, $elevenvariation_id, $variation  );
        }


Is this set neccesarily to be a vector space?

Question: Suppose $F$ be a field and $S$ be a non empty set such that
1) $a+b \in S $
2) $ \alpha a \in S$
for all $a,b \in S$ and $ \alpha \in F.$
Is always $S$ to be a Vector space?
Comment: If $S$ is a subset of a vector space $V$, then the given conditions are sufficient to make $S$ a subspace of $V$. But without such $V$, the addition and multiplication are not related enough
Comment: Are you assuming any that the operations $+: S \times S \to S$ and $\cdot: F \times S \to S$ have any particular properties?
Comment: $S$ with addition must be an abelian group. $\phi: F\rightarrow\text{End}(S)$ determined by $\phi(\alpha)(a)=\alpha a$ must be a ring homomorphism.
Comment: @Travis: Ofcourse $S$ should be an abelian group under $+.$ Sorry I forget to mention that. Thanks drhab.
Answer: No. Take $F=\mathbb F_2$, $S=\mathbb Z$ and set $0\cdot a=0$ and $1\cdot a=a$ for all $a\in S$. Then you would have $0\cdot 1=(1+1)\cdot 1=1\cdot 1+1\cdot 1=2$, because in any vector space $(\alpha+\beta)a=\alpha a+\beta a$.
Answer: No, for we could take the multiplication map $\cdot: F \times S$ to be zero map---the resulting structure fails to be a vector space only because it does not have a multiplicative identity, i.e. an element $1 \in F$ such that $1 \cdot s = s$ for all $s \in S$.
Ferra's nice example shows that preventing this degenerate solution by demanding existence of a multiplicative identity still isn't enough to guarantee that the $S$ is a vector space over $F$.


BeautifulSoup to Data Frame With Pandas

Question: New to programming here, so forgive the silly questions.  I've been trying to work out how to Python for web scraping and a lot of the YouTube videos and other questions kinda get me there, I'm having a hard time relating the answer to my actual code.
My code so far is:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import csv
import time
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup as soup
import pandas as pd

url = "https://racing.hkjc.com/racing/information/English/Racing/LocalResults.aspx?RaceDate=2021/02/06&Racecourse=ST&RaceNo=1"
driver = webdriver.Chrome()
driver.get(url)

html = driver.execute_script("return document.getElementsByTagName('html')[0].innerHTML")

soup_level1=soup(driver.page_source, 'lxml')

race_soup = soup_level1.find("tbody", class_="f_fs13")
print(race_soup.text.strip())

results_soup = soup_level1.find("tbody", class_="f_fs12")
print(results_soup.text.strip())

datalist = [] #empty list
x = 0 #counter
print('good')

driver.close()

This will generate the parsed data, but now I am stuck as how to move it from text to a data frame with pandas.  I'm sure it is simple, but all of the instructional material I've seen isn't clicking for me.
Also, The code so far is just sort of copy and pasted chunks from different websites that I got to work with trial and error.  I'm not sure if any of it is redundant, so if there is a neater way to go about it, I would appreciate that feedback as well!
Thanks in advance,
Spencer
Comment: `dfs = pd.read_html(driver.page_source)` this will return a list of all the `` tags it finds as dataframes. Just a matter of pulling out the table you want (provided there are `` tags found)
Comment: Awesome, thanks.  I knew it had to be something simple, but I couldn't quite piece it together.

Another question is, how do I choose which table(s) I want and exclude the others?  For example, there are more than two tables on the page, but I just want the first two.  How can I stop after that?
Comment: so is that `'race'`, and `'result'`?
Comment: Yeah.  The 'race' table just has information about the race itself (date, track, time, etc).  The 'results" table has the actual times the horses ran, place finished, etc.  Then there are other tables about betting pay offs and other things that I do not care about and do not want pandas to work with.
Comment: ya ok. little trickier since it's the `` tags that have the specific class attribute. Give me a sec, I'll post the solution
Comment: Thanks, appreciate it.  It's not the end of the world, as I can just, you know, ignore those tables lol.
Answer: Give this a try:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import csv
import time
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup as soup
import pandas as pd

url = "https://racing.hkjc.com/racing/information/English/Racing/LocalResults.aspx?RaceDate=2021/02/06&Racecourse=ST&RaceNo=1"
driver = webdriver.Chrome()
driver.get(url)

html = driver.execute_script("return document.getElementsByTagName('html')[0].innerHTML")
WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "f_fs13")))
htmlStr = driver.page_source

soup_level1 = soup(htmlStr, 'html.parser')

race_soup = soup_level1.find('tbody',{'class':'f_fs13'}).find_parent('table')
results_soup = soup_level1.find('tbody',{'class':'f_fs12'}).find_parent('table')

df1 = pd.read_html(str(race_soup))[0]
print(df1)

df2 = pd.read_html(str(results_soup))[0]
print(df2)

datalist = [] #empty list
x = 0 #counter
print('good')

driver.close()

Comment: Ran into this error:

AttributeError: ResultSet object has no attribute 'find_parent'. You're probably treating a list of elements like a single element. Did you call find_all() when you meant to call find()?
Comment: Traceback (most recent call last):
  File "C:\Users\Spenc\Desktop\Python Stuff\selenium test.py", line 21, in
    race_soup = soup_level1('tbody',{'class':'f_fs13'}).find_parent('table')
  File "C:\Users\Spenc\AppData\Local\Programs\Python\Python39\lib\site-packages\bs4\element.py", line 2173, in __getattr__
    raise AttributeError(
AttributeError: ResultSet object has no attribute 'find_parent'. You're probably treating a list of elements like a single element. Did you call find_all() when you meant to call find()?
Comment: Traceback (most recent call last):
  File "C:\Users\Spenc\Desktop\Python Stuff\selenium test.py", line 21, in
    race_soup = soup_level1.find('tbody',{'class':'f_fs13'}).find_parent('table')
AttributeError: 'NoneType' object has no attribute 'find_parent'

Do you mind if I ask you to explain what you're doing to try and fix this?  It looks like the new code is saying "go to this element, and then find the parent, and then give me just that table".  Am I understanding it correctly?
Comment: hmmm. ok. let me debug on my end. what version of beautifulsoup you running?
Comment: I think bs4?  Whatever the most recent one is, I guess.
Comment: do `import bs4`
`print(bs4.__version__)` I'm on 4.6.0
Comment: it tells me version 4.9.3
Comment: ok. I think i found the issue. It was trying to parse the data/html before the page was rendered. So put an wait element in there. try now.

BeautifulSoup to Data Frame With Pandas


How to find nearest parent by tag_name with selenium

Question: I'm working with "Table styled" web site by using selenium-webdriver.
There are a lot of nesting of tables and there are no id or name attributes of them.
So I decided to get header text in a table to find a place of data like this.
 driver = Selenimum::WebDriver.for :firefox
 element = driver.find_element(:xpath,
   "//font[@color='#FFFFFF' and text()='some probably unique text']")

from HTML like this.

...

  some probably unique text

...

I want to get the inside table element from font element that I got in above code.
I know I can get with element.find_element(:xpath, "../../../.."), but it's a little disgusting.
I want to specify a tag name at least like this element.find_element(:xpath, "../*/table").
Is there way to do this?
Answer: You can use XPath axes ancestor.
element = driver.find_element(:xpath, ".//font[@color='#FFFFFF' and text()='some probably unique text']/ancestor::table")

Or nested XPath: not for nested tables situation like yours
element = driver.find_element(:xpath, ".//table[.//font[@color='#FFFFFF' and text()='some probably unique text']]")

Comment: The first is all right, but add a `(...)[1]` around it to fetch _nearest_ ancestor. The second just fetches all of them and doesn't solve the problem.
Comment: @JensErat: Yeah, you are right, I thought tables were parallel, but they are nested.
Comment: Just realized it might be more reasonable to remove the parenthesis, in case multiple tables for multiple font elements are to be found; thus return the first table ancestor for each font element.
Answer: If you're using Capybara, nodes have a method ancestor that accepts a CSS selector (similar to JavaScript's closest).
For example:
table = find("font[color=FFFFFF]", text: "some probably unique text").ancestor("table")

within table do
  click_on "stuff"
end


Handling categorical variables in various ML algorithms

Question: I have read at many places that Decision trees and Random forests, if deep enough, can handle categorical variables without one-hot encoding.
1) What is special about these algorithms that they can handle categorical variables without one-hot encoding?
2) Are there any specific algorithms in which we do Dummy encoding (n-1 columns created for a categorical variable) vs One Hot encoding (n columns created)? I fail to understand that when one column in One hot encoding is having information that can be gathered from other columns, why do we ever prefer One hot encoding and why does the concept even exist?
3) Why does it say "if deep enough"?
Any helpful resources, links, videos or your own explanations are welcome. I want to clear this doubt, once and for all.
Found a similar question but doesnt answer everything I want to know.
https://datascience.stackexchange.com/questions/5226/strings-as-features-in-decision-tree-random-forest/19829#19829
from AN6U5's answer - says Random Forest does not require One-hot Encoding and I have read many more similar answers saying this.
Also, What algorithms require one-hot encoding?
Comment: Could you add some references for that claim?
Comment: Whether software works requires one-hot encoding or not is entirely dependent on the software, but the comments **directly below AN6U5's answer** claim that sklearn's implementation requires one-hot encoding. Presumably you have access to sklearn, so you should be able to try it yourself and find out.
Answer:
If the categorical variable is non-numeric, most software (like R) will convert the one column into n columns. It could be that they are not referring to anything special about the mathematical algorithm, but simply that the function or program you are using handles the one-hot encoding automatically.
A reason for converting a categorical variable to n-1 dummy variables is because of an issue in OLS regression known as perfect multi-collinearity. In OLS regression, the excluded n-th dummy variable is now used as a reference for the n-1 other dummy variables.
Perfect multicollinearity is NOT an issue in decision tree algorithms, and so you can use one-hot encoding. In fact, it'd be better to use one-hot encoding, as excluding that n-th column actually excludes information in these algorithms.
You'll have trouble answering this question, as "depth" has an arbitrary meaning.
The only time it makes sense to leave a categorical variable as one column in your data is if the categorical variable is numeric and there is an order present. For instance, 2 is more than 1, 3 is more than 2, so on... At least this has been my experience.
Comment: Can you elaborate more on 1. I mean to ask how do these algorithms operate internally so as to be able to handle categorical variables even without one-hot encoding.
Please cite some sources to read more.
Comment: The mathematical algorithms (in this case, the decision tree algorithms) do NOT handle categorical variables, rather, the functions/programs you use are coded such that one-hot encoding is performed before running your data through the mathematical algorithm. That is to say, if you use Python, the decision tree function you use will perform the one-hot encoding BEFORE building the model. The model is then built with the n dummy variables, without you having to do the one-hot encoding yourself.


Azure Ubuntu VM direct S2S VPN - without VPN Gateway. Is it possible?

Question: I would like to connect on prem Fortigate FW with one azure Ubuntu VM - direct. Is it possible without Azure VPN Gateway?
Config works with other vendor but no in Azure. Is that kind of config blocked somehow in azure? Thank's for help.
https://drive.google.com/file/d/1JLdM2r5npe5e3ewaeXTd-Vi70RyyLQ_r/view?usp=sharing
My oversight, sorry for the lack of configuration. On the Azure side of Ubuntu 20.04 - strongswan. Identical configuration uploaded to another provider works fine. Only difference is no external IP on the ubuntu vm interface in azure.Azure NSG 500/udp and 4500udp incoming allowed.UFW inactive. Phase1 doesn't start.
//ipsec.conf
config setup
charondebug = "all"
uniqueids = yes
conn strongswan-to-fortigate
type = tunnel
auto = start
keyexchange = ikev2
authby = secret
left = xx
leftsubnet = 10.0.0.1/24
right = yy
rightsubnet = 192.168.1.1/24
ike = aes256-sha256-modp2048
esp = aes256-sha256
aggressive = no
keyingtries = %forever
ikelifetime = 28800s
lifetime = 3600s
dpddelay = 20s
dpdtimeout = 120s
dpdaction = restart
//ipsec.secret
xx yy : PSK "key"
Comment: Yes. Microsoft doesn't attempt to block VPN traffic. Given that you show absolutely no configuration, we can't tell you why it doesn't work.
Comment: [Edit] your question to add the information; don't add it in comments.
Answer: I think you are you using the incorrect sort of VPN, an s2s is a site to site, and what you are describing is a p2s vpn(point-to-site).
You can of course place that one vm in a seperate vnet, and then set up a s2s vpn to vpn to that vnet, and then resource lock that vnet so additional machines get added to that vnet.
And I think a vpn in Azure always requires a vpn gateway, there nust be something to tunnel and encrypt the traffic.
I would look into setting up a p2s vpn between that fortigate and your ubuntu vm.
Edit:
Owh yeah, If you use SSL based VPN from Fortinet, you can use openfortivpn software which is part of Ubuntu and Fedora.


Adding single hidden input value to each form input in ajax call

Question: I have a form with a button that can add dynamic input fields, and it's creating an ajax issue. My ajax post is giving me 500 errors
 But My console log for data right now is this:

but I need to insert these as
insert into ticker_content (ticker_id, content)
     values (1, 'one'), (1, 'two');

If that makes sense.
So basically, the problem is i have multiple inputs at any given time (containing text values) and a hidden input in the form that gives me my correct ticker ID.
However, I need to make my array contain elements that each have the input text value and the ticker ID. So for every input that's created and filled, I need to assign the value of that form's hidden input to it as well so I can sent them as pairs to my foreach loop and insert.
Here's my addticker.php that's being called for the insert:
$items = $_POST['Items'];
$tickerID = $_POST['tickerID'];

foreach ($items as $item){

    $addTicker = "
        INSERT INTO ticker_content (tickerID, content)
        values ('$tickerID', '$item');
     "
     $mysqlConn->query($addTicker);
}

So basically for every Items[] value, I need to insert with the same hidden field.
Here's my form and JS code for reference. The first JS block is mainly for the functionality of dynamically adding inputs, but the last JS block is the ajax using serializeArray();
<?php foreach($tickerDisplays as $key => $ticker):?>

    <form id="Items" method="post">
        Item 1:
        <input type="text" name="Items[]">

        <button type="button" class="moreItems_add">+</button>

        <input type="hidden" name="tickerID" id="tickerID" class="tickerIdClass" value="<?php echo $ticker['ticker'] ?>">
        <input type="submit" name="saveTickerItems" value="Save Ticker Items">
    </form>

<?php endforeach;?>

Comment: As for the debugging after you've added the semi-column, you can use the developer tools of your browser. See: https://www.browserstack.com/developer-tools Go to the 'network' tab and check the parameters and output to see what your ajax call did. For debugging purposes you might want to echo something in your `addticker.php` to tell yourself what is happening.
Comment: You `addticker.php` returns an error 500, which means: "Internal Server Error", usually due to an syntax error in PHP. I see a missing semi-column at the end of your query string. When you get an error 500 you can usually see what problem is in the PHP error log. Have you been warned about SQL-injection? Your code is vulnerable.
Comment: Ok, good catch. I added the semicolon and got a 200 that time but no insert. Is there a better way I can test this for results on the page or in the console? I just feel like it won't insert the way I expect, especially with  multiple inputs
Comment: On it, thanks so much!
Answer: Firstly, you are missing a semicolon in your code (which is likely causing your 500 error).
Secondly, if you want to bundle all the fields from the form as a single query, the following will build out a query similar to what you noted earlier in your question as:

INSERT INTO ticker_content (ticker_id, content) VALUES(1, 'one'), (1, 'two'), ...

$items = $_POST['Items'];
$tickerID = $_POST['tickerID'];

$addTicker = "INSERT INTO ticker_content (tickerID, content) values";
foreach ($items as $item){
    $addTicker .= "('$tickerID', '$item'),";
}
$addTicker = substr($addTicker, 0, -1); // Eat that last comma

$mysqlConn->query($addTicker);

Your HTML also needs some work because the id attribute should be unique on the page. Since you are duplicating the form, you should do something like the following:
<form id="Items<?php echo $ticker['ticker']?>" class="tickerform" method="post">

And then update your javascript:
// Using $(this) in Jquery allows you to access the
// element that is making the method call (in this case, the form)
$(".tickerform").submit(function(e) {
    e.preventDefault();
    var data = $(this).serializeArray();
    console.log(data);
    $.ajax({
       type: "POST",
       url: "addticker.php",
       data: data, // Don't need to serialize again, 'var data' is still in scope.
       success: function(data)
       {
           alert(data); // show response from the php script.
       }
    });
});

Comment: Ok so I actually got it to work with multiple fields on the same form! It inserted one record for each with the right ID. HOwever, the other forms have a submit button as well that doesn't work (probably due to an id on the submit input in my form?) Can you help me out there? So that no matter how many forms, the functionality will be the same?
Comment: So basically, submitting the first form works fine functionally but I may need a count or something on the submit button so that it will work on each form
Comment: @Whisou138 Updated answer.
Comment: You're the best! Thank you


Get elements of set excluding some

Question: dic = {'a': {1, 2, 3}, 'b': {5, 6, 7}}
print(dic['a'])

How can I get the printed result to exclude 1 (not to remove) to be:
{2,3}

Comment: Do you want to exclude number 1, or first item of every set?
Comment: @MauroBaraldi numer 1
Answer: You could subtract {1} from the inner set:
print(dic['a'] - {1})
{2, 3}

Or equivalently you can use difference:
print(dic['a'].difference({1}))
{2, 3}

You can learn more about the topic on sets — Unordered collections of unique elements
Comment: Nice and simple.
Comment: This is the best way since it has O(1) complexity.

Get elements of set excluding some