R Add previous month value as a column

Question: I have a data table with date and price by month:
set.seed(128)
dat <- data.table(DATE=c("2014-01-01","2014-02-01","2014-03-01","2014-04-01",
                "2014-05-01","2014-06-01","2014-07-01","2014-08-01",
                "2014-09-01","2014-10-01","2014-11-01","2014-12-01"),
              PRICE=rnorm(n = 12, mean=100, sd=15))
dat

      DATE     PRICE
1: 2014-01-01 108.83590
2: 2014-02-01  67.69042
3: 2014-03-01 122.28905
4: 2014-04-01  93.50617
5: 2014-05-01 109.57878
6: 2014-06-01 128.81537
7: 2014-07-01  82.67295
8: 2014-08-01  68.25281
9: 2014-09-01  82.98265
10: 2014-10-01 124.28960
11: 2014-11-01  84.61275
12: 2014-12-01  93.14938

I would like to add the previous months price as a column in the data table, so the table would look like this:
      DATE     PRICE    PRICE_PREV
1: 2014-01-01 108.83590
2: 2014-02-01  67.69042 108.83590
3: 2014-03-01 122.28905 67.69042
...
10: 2014-10-01 124.28960 82.98265
11: 2014-11-01  84.61275 124.28960
12: 2014-12-01  93.14938  84.61275

How would I accomplish this in R?
Comment: I don't see the point . . your data looks wrong now in your expected output format . . you have 108.83 against Feb instead of Jan . .
Comment: The new column would allow me to make calculations of month over month data such as price difference (e.g. dat$PRICE_DIFF <- dat$PRICE - dat$PRICE_PREV)
Comment: why don't you convert date to as.date and plot month over month change using `diff` function ?  refer this post: http://stackoverflow.com/questions/6876928/r-month-by-month-percent-growth-on-an-xts-objects
Answer: Use shift from data.table v >= 1.9.5 (on GitHub)
dat[, PRICE_PREV := shift(PRICE)]
dat
#           DATE     PRICE PRICE_PREV
#  1: 2014-01-01 108.95158         NA
#  2: 2014-02-01 107.23918  108.95158
#  3: 2014-03-01 124.96608  107.23918
#  4: 2014-04-01  99.61007  124.96608
#  5: 2014-05-01 118.13540   99.61007
#  6: 2014-06-01 108.70528  118.13540
#  7: 2014-07-01  93.12482  108.70528
#  8: 2014-08-01 100.06829   93.12482
#  9: 2014-09-01 103.14932  100.06829
# 10: 2014-10-01 122.66710  103.14932
# 11: 2014-11-01 104.84064  122.66710
# 12: 2014-12-01 113.37610  104.84064

If you have an older version (v < 1.9.5) you could also do
dat[, PRICE_PREV := c(NA, PRICE[-.N])]

Answer: The first term has no known price for previous month (I'm assuming - if it does, you can replace the NA with it).
The following code should work fine. The first line creates a new vector, the second adds it as a column to dat and the third removes the vector (as it's now in dat and we don't need it separately).
PRICE_PREV <- c(NA, dat$PRICE[1:length(dat$PRICE)-1])
dat <- cbind(dat, PRICE_PREV)
rm(PRICE_PREV)

R Add previous month value as a column


pip install in editable mode from a pyproject in a docker container, ModuleNotFoundError not found

Question: When I pip install -e . my python package in a container (using setup.py and pyproject.toml, not requirements.txt), python cannot find my package.
If I remove -e from the pip install, so its just pip install . then it works in the container.
I added a pip list and I see my module installed on the container, yet it cant be used.
Note: I am using a docker volume for my project on the container. I thought this would allow me to edit the files locally without having to restart the container in and editable mode.
Here is my docker-compose
version: '3.3'
services:
  myapp:
    container_name: myapp
    build:
      context: ./services/myapp
      dockerfile: Dockerfile.dev
    restart: always
    volumes:
      - ./services/myapp/:/usr/src/app/
    ports:
      - 5000:5000
    env_file:
      - ./.env/.env.dev.myapp
    networks:
      - myapp-network
networks:
  myapp-network:
    driver: bridge

My docker file
# pull official base image
FROM python:3.10.7-slim-buster

# set work directory
WORKDIR /usr/src/app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# install system dependencies
RUN apt-get update && apt-get install -y netcat ansible ssh sshpass

# Copy certs over
COPY certs/* /app/

# install dependencies
RUN pip install --upgrade pip
RUN pip install -e .

# Show all installed modules
RUN pip list

# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

After that my entrypoint calls a manage.py that is outside of my project and I try to import some moduels with it, But this is where it fails. It cant find those modules.
If I use pip install -e . in my python evn outside of the container it works.
Is there a reason why pip install -e . wont work in my docker container?
UPDATE:
I found that if I go directly into the container and uninstall the module and reinstall it with pip install -e . it works.
Is this just a race condition problem with docker volumes?
Comment: How are you creating the container? Where are the library files located? The `-e` flag does not copy the files to `site-packages` so if the repository isn't copied into the container (but site-packages are), then the library won't be available.
Comment: There is nothing in `/usr/src/app` at build time, is there? docker-compose instructs to mount a volume there, but this only happens at run-time, not at build time, right?
Comment: I create the container through docker-compose > Dockerfile, I use a docker volume to get my package onto the container.
Comment: Please be more specific - edit your post to include the relevant Dockerfile code and host file structure.
Comment: I updated my post with the docker-compose.yml entry, and the Dockerfile.
Comment: That was it,  also I needed to direct the pip install -e to my project/ folder. Thanks. Will post a solution.
Answer: Problem was two fold.

I needed to copy my directory over in the Dockerfile as the volume  is only mounted at run time.
Needed to pip install to the actual driectory. I noticed the wrong directory was being used form the pip list output.

Corrected Dockerfile:
# pull official base image
FROM python:3.10.7-slim-buster

# set work directory
WORKDIR /usr/src/app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# install system dependencies
RUN apt-get update && apt-get install -y netcat ansible ssh sshpass

# Copy certs over
COPY certs/* /app/

# install dependencies
COPY . /usr/src/app
RUN pip install --upgrade pip
RUN pip install -e project/.

# Show all installed modules
RUN pip list

# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]


Encontrar div que contém imagem e adicionar classe para reconhecimento do CSS

Question: Preciso de ajuda para encontrar as tags  que tem dentro da estrutura abaixo.
Quando encontrado, adicionar uma classe "hasimg" junto a classe "skuFiltro" da tag
 primária. Preciso reconhecer se tem imagem ou não para poder diferenciar o estilo CSS da estrutura de front-end de uma plataforma que estou trabalhando.

                380mm

Answer: Para percorrer todos os elementos com a classe 'skuFiltro', você pode utilizar:
$('.skuFiltro').each(function(index, element) {
    var img = element.querySelector("img");
    if(img != null){
       element.addClass("hasimg");
    }
});

A função .each() percorre cada elemento que possua a classe .skuFiltro. A querySelector() está verificando dentro do atual elemento do .each, se há alguma tag 'img', caso não tenha nenhuma dentro do elemento em questão, a variável img vai receber NULL.
Comment: Obrigado Vinicius, problema resolvido! :)
Comment: @IvanRichardRoncatto show! Caso minha resposta tenha contribuído para sua resolução, se puder marcar ela como "resposta".


jquery animation does not animate

Question: i have some jquery code that will load the chatbox of my site every second (so if any new posts arrive they become visible)
my code is here
function loadLog(){
    $.ajax({
        url: "/log.html",
        cache: false,
        success: function(html){
            $("#chatbox").html(html); //Insert chat log into the #chatbox div
            if($("#chatbox").attr("scrollHeight") + 20 > $("#chatbox").attr("scrollHeight") - 20){
                $("#chatbox").animate({ scrollTop: newscrollHeight }, 'normal'); //Autoscroll to bottom of div
            }
        },
    });
}

everything works fine, except it is meant to autoscroll to the bottom of the chatbox so you see the newest posts, instead it just stays at the top.
I am using the most recent version of jQuery
Comment: Try use .prop("scrollHeight"), not attr
Comment: also the if condition doesn't make sense.. `if (x + 20 > x - 20)`???????
Comment: Set up a stripped down Proof of Concept on [jsfiddle](http://jsfiddle.net). Otherwise, everyone here is just guessing.
Comment: the condition means that it will only scroll if it is possible to scroll
Answer: There is no such attribute scrollHeight (it's property). What if you try something like this instead:
$box.animate({scrollTop: $box[0].scrollHeight}, 'normal');

http://jsfiddle.net/dfsq/zBdas/
Another tip: make sure you cache your DOM queries like $box = $("#chatbox"), don't reselect elements again and again.
Comment: thankyou this works fine my one used to work so i dunno what happened lol

jquery animation does not animate


Melhorar inserção em banco de entidades relacionadas?

Question: Tenho no banco de dados as tabelas User, Profissional e Endereço. A chave estrangeira de User está em Profissional e chave estrangeira de Profissional está em Endereço. Preciso cadastrar um profissional, que é um usuario e tem um endereço.
Model User:
public function profissional()
{
    return $this->hasOne('App\Profissional', 'users_id');
}

Model Profissional:
public function endereco()
{
    return $this->hasOne('App\Endereco');
}

public function user()
{
    return $this->belongsTo('App\User', 'users_id');
}

Model Endereço:
public function profissional()
{
    return $this->belongsTo('App\Profissional');
}

A maneira que encontrei para inserir o profissional no banco juntamente com as outras entidades foi essa:
$user = new User($userData);
$profissional = new Profissional($profissionalData);
$endereco = new Endereco($enderecoData);

$user->save();
$user->profissional()->save($profissional);
$profissional->endereco()->save($endereco);

O código está funcionando normalmente, porém acredito que existam formas melhores de se fazer a mesma coisa. Gostaria de soluções alternativas ou melhorias que possam ser feitas nesse código.
Answer: É assim mesmo como tu fez, pode ser feito para criar assim também:
$user->create(array())
     ->profissional()->create(array())
     ->endereco()->create(array());

mas, a lógica sua está dentro do padrão do Laravel


OrientDB: Accessing query result data in Server side function

Question: OrientDB allowes to create server side javascript functions.
Is there any way to access intermediate query results inside server side javascript function. Eg. If I return query results, I get json response array as expected. But if I try to access query result inbetween function like:
var comps = db.query("SELECT code from Company");

db.begin();
for(var i=0; i<comps.length; i++){
  var c = comps[i];
  db.save({
    "@class":"Temp",
    col:  c.code
  });
}
db.commit();

return comps;

For above function I get blank values stored in column "col".
As per my observation, OrientDb returns context instead of actual results inside the function.
Thanks in advance.
Answer: Based on your remark "OrientDb returns context instead of actual results inside the function" I assume you are working with Java 8. The point is, behavior of OrientDB differs between Java 8 and Java 7 in this respect (because of the different underlying Javascript engines: Nashorn vs Rhino). There are several known (related) issues, see this thread for example.
Answer: Replace c.code with c.field("code").
Comment: Does accessing fields in query results with field(name) work in javascript functions for OrientDB 2.x ? I can't get it to work. I get null values instead.


Proper way to get data from multi dim array

Question: Hi following is my array data:

I am trying to get the highlighted part of the picture.
This is an associative array so far I have tried this:
<?php

for($i = 0 ; $i < count($ad['all_atr']) ; $i++)
{
    $atr_n= $ad[$i]['all_atr']['atr_n'];
    $atr_v= $ad[$i]['all_atr']['atr_v'];
?>

    <?php echo $atr_n;?>
    <?php echo $atr_v;?>

<?php
....

but gives error 'Undefined offset: 0'
Answer: To access it, point the array first, then the index:
for($i = 0; $i < count($ad['all_atr']); $i++)
{
    $atr_n = $ad['all_atr'][$i]['a_name'];
    $atr_v = $ad['all_atr'][$i]['a_val'];
}

Or with using foreach:
foreach($ad['all_atr'] as $values) {
    $atr_n = $values['a_name'];
    $atr_v = $values['a_val'];
}

Comment: @Umerm change it to `a_name`
Comment: it says Undefined index: atr_n
Answer: Just check that the data is set, and isn't empty before adding them to $atr_n & $atr_v :
if (isset($ad['all_atr'][$i]['a_name']) && isset($ad['all_atr'][$i]['a_val']) && !empty($ad['all_atr'][$i]['a_name']) && !empty($ad['all_atr'][$i]['a_val'])) {
    $atr_n = $ad['all_atr'][$i]['a_name'];
    $atr_v = $ad['all_atr'][$i]['a_val'];
}

Comment: Change `atr_n` and `atr_v` to `a_name` and `a_val` respectively.
Comment: @Amal Murali,Yes corrected,I have seen hilighted part of question so I have used those.Thanks for notified.


C# - Save the last selected file names after closing DialogBox

Question: I have a button which will open a file dialog and choose 2 certificates.
  After i click the button a windows will be shown from where i choose two certificates.
I click Open so i will get this two certs.  What i want to do is that after i click Open and click again the Select Certificate button instead the empty filename textbox like this  I want the last selection to be saved.
Is it possible such a thing?
Comment: You mean you want to automatically re-select the previously selected items?
Comment: Yes, you are right.
Comment: I know how to remember the directory, but not how to find the selected files.
Comment: Save the FileName string & set it before opening a second time. Note that there is a bug with this only setting part of the file name - workaround here http://stackoverflow.com/questions/17163784/default-name-with-openfiledialog-c
Answer: OpenFileDialog has a FileName property
Just assign it
Comment: File**Names** is ReadOnly, but FileName isn't.
Comment: I assign the file names but it says: Property FileNames is ReadOnly
Comment: Shoot - the documentation says get set
Comment: @drgmak was this the answer that helped you? then you might considering to mark it also.
Answer: Assign the FileName property when you create the dialog. It supports multiple files if you include quotes around the file names (full paths). See the example below:
<code>var dialog = new OpenFileDialog
{
    Multiselect = true,
    FileName = @"""C:\example.txt"" ""C:\example2.txt"""
};
</code>


Determinate $d^{k}f_{c}$ for function $f(x,y)$ with k and c given

Question: In my Calculus 3 class I have been given following problem:
Determinate: $d^{k}f_{c}$ for function $f(x,y)$ where $c = (c1,c2)$ and $k = 2$.
My problem is that I have no clue what does $d^{k}f_{c}$ notation mean, since I have never seen it before.
Answer: If $M$ is a real smooth manifold of dimension $n$, you can define a linear application on the space of $r$-forms
$$d^r:\Omega^r(M)\to\Omega^{r+1}(M)$$
called $\textit{exterior derivative}$. In local coordinates, for a smooth chart $(U_{\alpha},\phi_{\alpha})$, the general expression of $d^r$ is given by
$$d^r(\omega\vert_{U_{\alpha}})=d^r\left(\sum_I\omega_{\alpha,I}dx_{\alpha}^I \right)=\sum_Id^r(\omega_{\alpha,I})\wedge dx_{\alpha}^I.$$
In your case you have a smooth function $f_c(x,y)$ which is a zero-form, so an element of $\Omega^0(\mathbb R^2)$ (or $\Omega^0(U)$, with $U\subset \mathbb R^2$). In this case you'll have a unique representation of the $k$-th exterior derivative of the function since $\mathbb R^2$ has a global chart given by the coordinates $(x,y)$. The first exterior derivative maps $f_c\mapsto d^1f_c\in\Omega^1(\mathbb R^2)$, in particular
$$d^1f_c(x,y)=\frac{\partial f_c(x,y)}{\partial x}dx+\frac{\partial f_c(x,y)}{\partial y}dy\text{ and }\\
\Omega^2(\mathbb R^2)\ni d^2f_c(x,y)=d^1(d^1f_c(x,y))=d^1\left (\frac{\partial f_c(x,y)}{\partial x}dx\right )+d^1\left (\frac{\partial f_c(x,y)}{\partial y}dy\right)=\dots$$
Is the problem more clear now?
Comment: Yes, thank you.


Why does the Sum() function in (My)SQL return double the value I expect?

Question: I need sum of two different columns from two different tables, being tbl_bill and tbl_receipt.
tbl_receipt
 id receipt_no  member_id   receipt_amount
 1  21001         21        20500
 2  21002         21        200

tbl_member
  id    name
  21    priya

tbl_invoice
 id bill_no  order_id   member_id     bill_value
 1  30001       30          21          20928
 2  0002501     25          21          12030

I want answer in this way
member_id  member_name  receipt_sum  bill_sum
21         priya        20700         32958

I tried doing left join but it doubles the sum value. Can any one help me out?
$adapter = $this->tableGateway->getAdapter();
       $orderTable = new TableGateway('tbl_receipt', $adapter);
       $sqlSelect = $orderTable->getSql()->select();
       $sqlSelect->columns(array('id'=>'id','bill_amount'=>'bill_total','sum_receipt_amount'=>new \Zend\Db\Sql\Expression('SUM(tbl_receipt.receipt_amount)')));
       $sqlSelect->join('tbl_member','tbl_member.id = tbl_receipt.member_id', array('member_id'=>'id','member_name'=>'name'),'LEFT');
     //  $sqlSelect->join('tbl_receipt','tbl_receipt.member_id = tbl_bill.member_id', array('receipt_id'=>new \Zend\Db\Sql\Expression('(tbl_receipt.receipt_amount)')),'LEFT');

       $sqlSelect->group('tbl_member.id');
       $resultSet = $orderTable->selectWith($sqlSelect);
        //echo"
";

   // print_r($resultSet); die;
  return $resultSet;

Comment: *"I tried doing left join but it doubles the sum value."* - where is it? Maybe you did an error somewhere and we don't know that; "you" do.
Comment: Please providing your existing query for clarity.
Comment: *"I need sum of two different columns of two different tables"* - which 2 tables? There are 3 here.
Comment: is this codeigniter?
Comment: @FunkFortyNiner zend f2
Comment: tbl_bill and tbl_receipt
Comment: Share your tables and data, or examples of it doubling - please.
Answer: In SQL You  should left join on the result for sum
select m.id, m.name,t1.sum_amount, t2.sum_total
from tbl_member m
left join (
  select member_id, sum(receipt_amount) sum_amount
  from tbl_receipt
  group by member_id
) t1 on t1.member_id = m.member_id
left  join (
  select member_id, sum(total) sum_total
  from tbl_invoice
  group by member_id
) t2  on t2.t1.member_id = m.member_id

Comment: The main point here is that you're using a subquery rather than it just being a left join.
Comment: I'm using two left  join on aggregated  result  ..  so avoid the duplication related  to the join with the non aggregated  row  ..


Jsoup and getting data from a table

Question: I have tried similar code on the BBC and Wikipedia websites and I can access and handle data from the tables. What is my mistake when I try to do the same on this website? I can get some data with this code but not the specific figures inside the table when I change it to <code>.doc.getElementsByClass("tabela2");</code>.
<code>    String url = "https://www.itau.com.br/investimentos-previdencia/fundos/rentabilidade-uniclass";
    Document doc = null;
    try {
        doc = Jsoup.connect(url)
        .userAgent("Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36")
        .get();
    } catch (IOException e) {
        e.printStackTrace();
    }
    Elements table = doc.getElementsByClass("contentAllInt");
    System.out.println("table: " + table);
</code>
Thanks
Answer: The website in question uses AJAX to load its content dynamically after initial loading. So it uses JavaScript to render the content.
JSoup does not handle Javascript. It is not a browser after all. So JSoup can't be used here.
2 solutions:
1) Use a real brwoser, e.g. selenium to access this content in Java. Drawback: You add a new dependency. Real browsers are resource hungry. It may run slowly.
2) Investigate the AJAX calls directly and try to mimic those calls and interpret the results. Usually this is not in HTML format but in JSON, so you may need another interpreting library. I did not look into this option in your case, but usually this is the faster option.
Comment: If you find my answer helpful don't stop yourself from upvoting. If my answer brings a satisfactory solution, accept it and by that mark the question as solved :)
Comment: I accepted the answer but still don't have enough rep to upvote it :-)

Jsoup and getting data from a table


Working with PIX firewalls

Question: I have acquired a used Pix firewall. A PIX 506 to use as a firewall for my small home office, the netgear wi-fi was really not cutting it.
I'd like to get a pointer on where to start with it, like how do I reset it to factory specs,  log in, etc.
I'm an OS X, Linux, UNIX person and don't know IOS or Windows.
Comment: Cisco has documents on how to do factory resets: http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_password_recovery09186a008009478b.shtml
Comment: Be very careful with that device. The PIX line is end-of-life/end-of-support/end-of-patch with known vulnerabilities with public exploits. Particularly the 501/506 since they only support up to 6.3. You would be better served with a 2000 era PC running pfSense or something similar.
Comment: @ScottPack -- I actually also have a PIX 515, is that any better?
Comment: @user1172468 The 515e is the bottom grade, but most modern class, of PIXes. It'll take the newest PIXOS but is still abandoned technology. So better, but still a bit scary. That device does use standard desktop RAM, I can't remember which exactly, so it is easy to upgrade out of support.
Comment: @ScottPack, mmm I like pfSense but I really want a low power consumption system ... so I thought the PIXes would fit the bill ... so a 515e is scary ?
Comment: The entire PIX line is dead now, the 515e just supports a newer revision than the 506. Don't forget, the 515e bottom vents, so don't put it on a shelf.
Answer: Running the startup wizard if available in the OS you're running might be your best bet to start.
Comment: Hi Keith, I have access to Linux and OS X -- is there a start up wizard for these two OSes?
Comment: If you have the serial cable to match you can connect it.  9600 N/8/1 is the default connection.  Boot it to see what OS you are running.  Then you can run the factory reset procedure detailed in the other posts.  At that point you may have a startup wizard available when you reboot.
Answer: Password reset can be found on the Cisco site -  http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_password_recovery09186a008009478b.shtml
There is a quick start on the Cisco website - http://www.cisco.com/en/US/docs/security/pix/pix62/quick/guide/506quick.html
The web panel is Java based (If it is installed on your PIX) so will run on Windows, Linux, MAC.

Working with PIX firewalls


Marshall MG10CF and PC

Question: I have "Marshall MG10CF" with a headset output.
Can i somehow connect it to my pc as a microphone and record guitar?
Answer: You do not want to connect as a mic, as the output to a headset will be too high a level. Instead, you want to use a line-in input if your pc has one, or if not, use an external audio interface to connect.
USB audio interfaces can be very cheap.

Marshall MG10CF and PC


how to block access to include(x_file) from a file in a certain folder?

Question: I have a folder that contains many different files:
folder1:
somthing.php
somthingelse.php
blah.php
ect ect....

Now, I don't want the files in that folder to access (include) files outside the folder. So it's okay to include('blah.php') but not okay to include('../blah.php')..
Any ideas on how I'd go about doing this?
Answer: This is only possible on the CLI and in a Unix environment. You would have to use chroot to hide other folders.
http://php.net/manual/en/function.chroot.php


Why does my keras custom layer fit well on training data but gives bad results on validation?

Question: I am trying to understand how Keras custom layers works, but I am facing a problem with the validation accuracy of my model.
I tried to reproduce a simple convolutional network on MNIST dataset but with a custom layer combining the Conv2D operator and the BatchNormalisation.
First, the data I used :
from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = np.array([x.reshape(28, 28, 1) for x in X_train])
X_test = np.array([x.reshape(28, 28, 1) for x in X_test])
y_train = pd.get_dummies(y_train)
y_test = pd.get_dummies(y_test)

Here is the original implementation which works well :
def get_model():
    input_ = Input(shape=(28, 28, 1))
    x = Conv2D(filters=64, kernel_size=3, activation="relu", input_shape=(28,28,1))(input_)
    x = BatchNormalization()(x)
    x = MaxPool2D(pool_size=(2,2))(x)
    x = Conv2D(filters=128, kernel_size=3, activation="relu")(input_)
    x = BatchNormalization()(x)
    x = MaxPool2D(pool_size=(2,2))(x)
    x = Conv2D(filters=256, kernel_size=3, activation="relu")(input_)
    x = BatchNormalization()(x)
    x = MaxPool2D(pool_size=(2,2))(x)
    x = Flatten()(x)
    x = Dense(128, activation="relu")(x)
    x = Dense(64, activation="relu")(x)
    x = Dense(10, activation="softmax")(x)
    mod = Model(inputs=input_, outputs=x)
    return mod

optim = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, clipvalue=K.epsilon())
model = get_model()
model.compile(optimizer=optim, loss='categorical_crossentropy', metrics=["accuracy"])
model.fit(X_train, y_train, batch_size=128, epochs=3, validation_data=(X_test, y_test))

With this initial model, after 3 epochs, I get a train accuracy of 97% and validation 97%
And here is my custom layer :
class Conv2DLayer(Layer):
    def __init__(self, filters, kernel_size, dropout_ratio=None, strides=(1, 1), activation="relu", use_bn=True, *args, **kwargs):
        self._filters = filters
        self._kernel_size = kernel_size
        self._dropout_ratio = dropout_ratio
        self._strides = strides
        self.use_bn = use_bn
        self._activation = activation
        self._args = args
        self._kwargs = kwargs
        super(Conv2DLayer, self).__init__(*args, **kwargs)

    def build(self, input_shape):

        self.conv = Conv2D(self._filters,
                           kernel_size=self._kernel_size,
                           activation=self._activation,
                           strides=self._strides,
                           input_shape=input_shape,
                           *self._args,
                           **self._kwargs)
        self.conv.build(input_shape)
        self.out_conv_shape = self.conv.compute_output_shape(input_shape)
        self._trainable_weights = self.conv._trainable_weights
        self._non_trainable_weights = self.conv._non_trainable_weights

        if self.use_bn:
            self.bn = BatchNormalization()
            self.bn.build(self.out_conv_shape)
            self._trainable_weights.extend(self.bn._trainable_weights)
            self._non_trainable_weights.extend(self.bn._non_trainable_weights)

        if self._dropout_ratio is not None:
            self.dropout = Dropout(rate=self._dropout_ratio)
            self.dropout.build(self.out_conv_shape)
            self._trainable_weights.extend(self.dropout._trainable_weights)
            self._non_trainable_weights.extend(self.dropout._non_trainable_weights)

        super(Conv2DLayer, self).build(input_shape)

    def call(self, inputs):
        x = self.conv(inputs)
        if self.use_bn:
            x = self.bn(x)
        if self._dropout_ratio is not None:
            x = self.dropout(x)
        return x

    def compute_output_shape(self, input_shape):
        return self.out_conv_shape

Finally, here is the modified model :
def get_model():
    input_ = Input(shape=(28, 28, 1))
    x = Conv2DLayer(filters=64, kernel_size=3, activation="relu")(input_)
    x = MaxPool2D(pool_size=(2,2))(x)
    x = Conv2DLayer(filters=128, kernel_size=3, activation="relu")(input_)
    x = MaxPool2D(pool_size=(2,2))(x)
    x = Conv2DLayer(filters=256, kernel_size=3, activation="relu")(input_)
    x = MaxPool2D(pool_size=(2,2))(x)
    x = Flatten()(x)
    x = Dense(128, activation="relu")(x)
    x = Dense(64, activation="relu")(x)
    x = Dense(10, activation="softmax")(x)
    mod = Model(inputs=input_, outputs=x)
    return mod

For this model with custom layer, I managed to get the same train accuracy (97%), but the validation accuracy get stuck around 50%.
EDIT
Thanks to Matias Valdenegro, I achieved to solve the problem by modifying the call method :
def call(self, inputs):
    training = K.learning_phase()
    x = self.conv(inputs)
    if self.use_bn:
        x = self.bn(x, training=training)
    if self._dropout_ratio is not None:
        x = self.dropout(x, training=training)
    return x

With K the keras.backend module.
Answer: Both Dropout and Batch Normalization behave differently during training and testing/inference, and your layer does not have any of that behavior, so its using  those inner layers as training mode during inference, producing incorrect results.
I am not sure but I think you can fix this by passing the training parameter in the call function call through the layers, something like:
def call(self, inputs, training=None):
    x = self.conv(inputs)
    if self.use_bn:
        x = self.bn(x, training=training)
    if self._dropout_ratio is not None:
        x = self.dropout(x, training=training)
    return x

This should make the inner layers work differently during train and testing/inference phases.
Comment: Thanks a lot, I don't know why but the parameter `training=None` doesn't seem to work, so I used the function `keras.backend.learning_phase()` to get the training variable and now it works !
Comment: @RVaysse That is odd, you should not need to set learning phase manually, maybe there is another issue.


Where to learn the algebra behind the use of differential operators in calculus

Question: Algebra with differential operators is often used as a shortcut in calculus problems. I have previously asked about manipulations such as these:
$$\int x^5e^xdx =\frac1Dx^5e^x=e^x\frac{1}{1+D}x^5=e^x(1-D+D^2+...)x^5 + C$$
After reading some replies to my post, I have become interested in learning the abstract algebra that makes this magic work. Strangely enough, although the manipulation of $D$ is commonly taught as a method of solving differential equations, all the material (under the umbrella of real analysis) I have found about its usage did not touch on why or how it worked. I have come to the conclusion that if I am to understand the true mechanics of $D$, I am to look in texts about abstract algebra and not calculus.
However, the only area in math I have really worked with is calculus. In fact, the reason why I am interested is so I can apply $D$ to its fullest extent in calculus and make full usage of its properties, while feeling comfortable that I'm not breaking algebraic rules and abusing notation. Thus, with no background at all in abstract algebra, I really don't know what area to start learning, what texts to read, and what direction I need to go in.
As someone with no background in algebra, what should the itinerary of my mathematical journey be? Are there any introductory, prerequisite topics I should cover? After that, what specific topics relating to $D$? References to textbooks, lecture notes or even relevant wikipedia articles would be very much appreciated.
Comment: A related topic that might be useful for connecting operator theory with differential equations is Lie Theory, which deals with continuous groups of transformations which can be used to model solution spaces for systems of differential equations and is a fairly powerful tool for solving them via symmetry methods. I have Peter J. Olver's book, Applications of Lie Groups to Differential Equations and it covers this subject matter fairly well.
Comment: Also see Willard Miller's freely available book [Lie Theory and Special Functions,](https://www.ima.umn.edu/~miller/lietheoryspecialfunctions.html) which explains in more detail the Lie-theoretic methods used by Feinsilver.
Comment: You'll probably find helpful expositions on Umbral Calculus, e.g. see [links here](http://math.stackexchange.com/a/10161/242) to Roman and Rainville, and see Feinsilver: [Algebraic Structures and Operator Calculus.](http://www.springer.com/us/book/9780792321163)
Answer: I would say what you're interested in is Fourier multipliers.
The Fourier transform has the convenient property that for suitable functions, $\widehat{Df}(\xi)=\xi\hat f(\xi)$ where I've used the French convention $D=\partial/i$. This suggests that a factor of $\xi$ represents one derivative, so a more complicated function of $\xi$ represents a more complicated differential operator. In general, we write
$$a(D)f=\mathcal F^{-1}(a(\xi)\hat f(\xi))$$
and call $a$ the symbol of the operator. Lots of useful things can be written in terms of these symbols. For example:

Constant coefficient differential operators are polynomials in $\xi$:
$$a_0f+a_1f'+a_2f''+\cdots+a_nf^{(n)}=(a_0+a_1D+a_2D^2+\cdots+a_nD^n)f.$$
The Hilbert transform (among other singular integral operators, which are important in harmonic analysis) has symbol $sgn(\xi)/i$.
Even more broadly, there's the idea of a pseudo-differential operator, where you now allow the symbol to depend on both $\xi$ (frequency) and $x$ (space). As a simple example, polynomials in $\xi$ with coefficients in $x$ give differential operators with non-constant coefficients. These come up all the time in PDE. As a more interesting example, the operator which maps the Dirichlet data on a suitable region to the Neumann data can be expressed in this way, and it very nearly looks like what you would call $|D|$.

The utility of these things largely comes from the fact, as you observed, that algebra of operators is equivalent to algebra of symbols, which often simplifies things.
Answer: It sounds like you are interested in operator theory. The fact that you are asking this question suggests that you are pretty advanced for where you are but to really understand this, you need a background in functional analysis. I'm also not sure that abstract algebra is going to offer you any answers. Operator algebras generally fall under the umbrella of functional analysis. I'd recommend this text, though it may be overkill for your purposes.
Also just for the sake of motivating the development of operator theory a bit with some history, you might find this helpful.


iOS Get Date from Day of Year

Question: The common question on stackoverflow is how to get the Day of Year from a date but how to you get the date from the Day of Year?
I'm using the following code to generate the Day of Year but how to I do the converse?
<code>// Calculate Day of the Year
    NSCalendar *gregorian = [[NSCalendar alloc] initWithCalendarIdentifier:NSGregorianCalendar];
    NSUInteger dayOfYear = [gregorian ordinalityOfUnit:NSDayCalendarUnit inUnit:NSYearCalendarUnit forDate:[NSDate date]];
    setDayOfYear = dayOfYear;
    return setDayOfYear;
</code>
Answer: Assuming you know the year you want and the actual time of the date doesn't matter and you have the correct locale and time zones set up, you can just convert the day of the year and year from components back into a date. Again, you may need to decide what your application needs to do about the time component and you have to make sure that the day and year actually make sense for the calendar, but assuming the gregorian calendar you listed in your example, and the 200th day of 2013 (19 July, 2013), you could so something like this:
<code>NSDateComponents *components = [[NSDateComponents alloc] init];
[components setDay:200];
[components setYear:2013];
NSDate *july19_2013 = [gregorian dateFromComponents:components];
</code>
If I were to run this, I'd get a date that's 19 July, 2013 @ 07:00 since my current locale is in a time zone equiv of America/Los_Angeles and I didn't configure the calendar.

iOS Get Date from Day of Year


Arrange entites added to GAE by newest first when added?

Question: I created a google app engine client using eclipse and the android demo google hands out. I Created the back end and a few models. When I add an entities from android to my database on GAE it orders it by date not by newest created first. The key it just the current date and tie on android. Im not sure how to work with the back end, as google created it for me in my project. Is there a fast change I can make to it so instead or it ordering it by data when I add an item it will just keep the newest listings on top?
Edited question, this is my endpoint class Google generated for me. How can I modify it to receive the newest added entities first?
@Api(name = "quotesendpoint", namespace = @ApiNamespace(ownerDomain =     "projectquotes.com"           ownerName = "projectquotes.com", packagePath = ""))
 public class quotesEndpoint {

/**
 * This method lists all the entities inserted in datastore.
 * It uses HTTP GET method and paging support.
 *
 * @return A CollectionResponse class containing the list of all entities
 * persisted and a cursor to the next page.
 */
@SuppressWarnings({ "unchecked", "unused" })
@ApiMethod(name = "listquotes")
public CollectionResponse<quotes> listquotes(
        @Nullable @Named("cursor") String cursorString,
        @Nullable @Named("limit") Integer limit) {

 EntityManager mgr = null;
Cursor cursor = null;
List<quotes> execute = null;

try {
    mgr = getEntityManager();
    Query query = mgr.createQuery("select from quotes as quotes");
    if (cursorString != null && cursorString != "") {
        cursor = Cursor.fromWebSafeString(cursorString);
        query.setHint(JPACursorHelper.CURSOR_HINT, cursor);
   }

    if (limit != null) {
        query.setFirstResult(0);
    query.setMaxResults(limit);
}

execute = (List<quotes>) query.getResultList();
cursor = JPACursorHelper.getCursor(execute);
if (cursor != null)
    cursorString = cursor.toWebSafeString();

// Tight loop for fetching all entities from datastore and accomodate
// for lazy fetch.
for (quotes obj : execute)
    ;
} finally {
    mgr.close();
}

return CollectionResponse.<quotes> builder().setItems(execute)
        .setNextPageToken(cursorString).build();

Answer: The order you see in datastore viewer in GAE is not significant as it is just a display of the current data in your datastore and shown in the increasing order of entity id(if using auto id). This could coincidentally also have an increasing order of date. You cannot modify this display pattern.
What matters is the order seen by your queries and this is determined by indexes. So if you need to get your entities in the descending order of date, then if your date entry is left as indexed, GAE will be automatically having an index for date. You just need to query your entities by specifying a descending sort order on the date property.
EDIT:
Based on the code added, below modifications should be done to query the entities in descending order of date.
1, Add a new date property in your entity:
private Date entrydate;

2, While creating an entity, add the current date to this property
yourentity.setEntryDate(new Date())

3, While querying, set ordering based on descending order of date
query.setOrdering("entrydate desc");

Comment: I edited my question for you. I feel as if I'm close to getting this to work. There is my generated endpoint class. Can I modify it to send me the newest created entities first?
Comment: Ohh and the reason I ask like this is because I receive my entities as they appear on gae.
Comment: updated the answer to give some indications, regarding what to do. check the documentation to verify exact steps
Comment: great answer, you are getting me on the right track. set entry data is not a method in query... maybe i should modify Query query = entityManager.createQuery();  to send it back by newest date?


Spark ClassNotFoundException running the master

Question: I have downloaded and built Spark 0.80 using sbt/sbt assembly.  It was successful. However when running ./bin/start-master.sh the following error is seen in the log file
Spark Command: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -cp :/shared/spark-0.8.0-incubating-bin-hadoop1/conf:/shared/spark-0.8.0-incubating-bin-hadoop1/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop1.0.4.jar
/shared/spark-0.8.0-incubating-bin-hadoop1/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop1.0.4.jar -Djava.library.path= -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip mellyrn.local --port 7077 --webui-port 8080

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/master/Master
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.master.Master
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Update: after doing sbt clean (per suggestion below) it is running: see screenshot.
Answer: There can be a number of things which cause this error which are not specific to Spark:

Bad build, sbt clean compile that puppy again.
You have a cached dependency in your .ivy2 cache which conflicts with a dependency of that project version of Spark. Empty your cache and try again.
Your project which is building on Spark has a library version which conflicts with a dependency of Spark. That is, Spark may dependency on "foo-0.9.7" while your project put in "foo-0.8.4".

Try looking at those first.
Comment: I am not familiar with ivy2 dependency mgr - how to carry out emptying the cache as you mention in #2?


Changing the html and order of system fields(pass/email/locale/etc) of user edit form

Question: I have changed a number of forms accessible by users to fit a certain style. To do this I used an #after_build function that adds prefixes and suffixes to fields, as well as a bit of javascript. This was straightforward enough(and I hope the appropriate way to do that, though I'm not sure).
However, coming upon the user edit form, it does not behave as other forms do. The default system fields such as password, locale, email, etc, cannot be reordered and I can't seem to add prefixes or suffixes to them to insert my HTML.
I have read some things about using a template for forms via hook_theme() function, however I could not get Drupal to use my template file for the user profile edit form, though it works just fine for other forms. This was a last resort, however, and I prefer other means if possible.
How do I correctly alter the layout and html of the user profile edit form? Thank you for your time.
Answer: The reason the template method was not working on my site was the result of a caching issue.
To use a template file for the user profile form you can put this(or find the existing implementations of the hooks) in your template.php:
<code>function yourtheme_form_user_profile_form_alter(&$form, &$form_state, $form_id) {
  $form['#theme'] = 'user_profile_form';
}

function yourtheme_theme($existing, $type, $theme, $path){
  return array(
    'user_profile_form' => array(
      'render element' => 'form',
      'template' => 'templates/form--user_profile',
    ),
  );
}
</code>
Template file name, in this example, becomes form--user_profile.tpl.php


rule of thumb relating to access of classes from sibling packages

Question: Is there are any rule of thumb that relates to whether your package structure should allow access of a class from another class in a sibling package.
An example I have a class that represents a Login page:
project.page.login.LoginPage

And a class that represents an Account home page:
project.page.account.AccountHome

Both pages access the std chrome for the project (Header, Footer, Menu stuff and BasePage), is it better to put thoses classes in a sibling package e.g. project.page.chrome
project.page.chrome.BasePage
project.page.chrome.Menu
project.page.chrome.Footer
project.page.chrome.Header

or in the parent package:
project.page

e.g.
project.page.BasePage
project.page.chrome.Menu
etc

I know that this is a stylistic rule of thumb question, which  in a way  is subjective.
What I wish to know is if there is a commonly accepted rule for this sort of thing.  And if so what is the reasoning behind what are the problems or benefits associated with each approach.
Further to Vampire's answer.
My question is not whether you can reference classes from one sibling package to another.  It's whether you should and what are the reasons (either way).
Comment: See also http://stackoverflow.com/questions/1967229/java-subpackage-visibility
Answer: There is no such thing as a parent- or child-package. Each package in Java is completely stand-alone. Having them named hierarchically and stored like that in the filesystem is just a convention, but technically, those packages are all absolutely non-related stand-alone packages.
It is totally up to you how you like to organize your source-code in packages.
Comment: My question is not about whether you **can** but whether you **should**.  For example you can name your local method variables a1, a2, a3 etc but as a rule of thumb you shouldn't because it would be confusing and show's no intention or description of what you code is intended for.
Comment: Well, as I said in the second paragraph, it is totally up to you. And basically this question is heavily opinion-based and thus should be closed as off-topic like the SO rules suggest. ;-)
Answer: There is no conventional way to name your packages or grouping your classes in packages. But most of the time people tend to follow the standard Java APIs, and adopted the style from those.
(to name a few)
For e.g.: util, common, basic
Answer: you can use this but the best way is to review your specification and make it the most possibly readable and  extendable.
the best advice I can give you is learn the rules then forget them to make yours
Answer: There are maybe not exact rules but some guidelines. Link to answer to another similar question Since this answer appears on SO, pasting only the link here.  Reading the Uncle Bob articles may give you some pointers.
Answer: Package allows to combine the related classes/packages together with proper abstraction.
Technically, in Java there is no rule that sibling packages should allow class access from each other.
From design approach and good practices point of view, it is advised to group the classes/packages in a structured and intuitive way.
For eg.,
   project.page
   project.page.login
   project.page.chrome
   project.page.account

This facilitates in modularizing and unit testing of the application. Also, proper package hierarchy helps in quick debugging of the application using tools like log4j.


Highligh input default value and replace instead of writing before or after

Question: I have a form with default values for example 0.
When i click  i would like that 0 to highlight and when someone starts to write value replace the default value instead of writing before or after.
Comment: Yeah, that's really nice functionality, and you were very friendly when you told us about this feature. Did you have a programming specific question?
Answer: You can use HTMLInputElement.select() to select an inputs content. Also good to use focus() to focus it as well, as sometimes it doesn't.
const field = document.getElementById('yourId')

field.addEventListener('click', () => {
    field.select()
})


Which countries can participate in the Olympics?

Question: Which countries can participate in the Olympic games? Not to participate in a particular sport (like this question asks) but to participate in the Olympics at all. Is there any criteria for a country to participate in the Olympics?
Answer: In order for a country to send a team to the Olympics, the country needs a National Olympic Committee (NOC) that is recognized by the International Olympic Committee (IOC).  There are over 200 recognized NOCs, and Wikipedia has a list of all the recognized NOCs, sorted by the date the NOCs formed.
Under the current IOC rules, which were changed in 1996, new NOCs will only be recognized from nations that are recognized as an independent country by the United Nations.  However, before 1996, the IOC allowed NOCs from countries that were dependent territories of other nations, and these NOCs are allowed to continue.  Therefore, in the Olympics, you will see athletes from countries like Bermuda, Puerto Rico, and Hong Kong, which are all territories of other nations.  Currently, there are only two UN recognized countries that do not yet have NOCs: Vatican City and Niue.
There are unofficial NOCs that are not recognized by the IOC from various territories and groups of people, and Wikipedia lists some of those.  Some of these participate in the Paralympic games, but under the current rules they are not eligible for inclusion in the Olympics.


Linguagem em Português para Data em Joomla, PHP

Question: Estou trabalhando com um modulo de eventos em joomla e ao mostrar a data do evento o mesmo faz na linguagem Inglês. Possui o seguinte código em um dos arquivos do módulo:
<?php echo JHtml::_('date', $item->params->get('ctm_start',''), 'DATE_FORMAT_LC3'); ?>
        -
        <?php echo JHtml::_('date', $item->params->get('ctm_end',''), 'DATE_FORMAT_LC3'); ?>

Apenas por este código, eu conseguiria trazer a linguagem para português do brasil?
Comment: Pessoal, alguém da uma luz?
Comment: se minha resposta está certa, marque ela como correta, ou pelo menos dê um feedback.
Answer: Tem mais de uma forma de fazer isso. Uma delas seria alterar direto o código que você está apontando, outra é sobrepor linguagem.
Vá em Joomla > Administrator > Extensões -> Gerenciador de Linguagens > Sopreposição e então procure por DATE_FORMAT_LC3 e substitua por um formato que você quer usar.
O significado dos dígitos podem ser vistos em http://php.net/manual/en/function.date.php


Como adicionar dias em uma data com o formato dd/mm/yyyy em powerapps?

Question: Estou pegando uma data de uma tabela e adicionando dias na contagem no powerapps. Aparentemente no computador funciona, mas quando uso o aplicativo do powerapps no celular nao funciona, a data vem trocada, ao invés de vir dd/mm/yyyy vem mm/dd/yyyy.
O código abaixo cria o valor para a label 18:
Text(DateValue(Label17.Text);"[$-pt-BR]dd/mm/yyyy";"pt-BR")

E o código abaixo adiciona a quantidade de dias na data:
DateAdd(DateValue(Label18.Text);Value(ThisItem.Qtde_dias_para_contato);Days)

Não entendo o porquê de funcionar no computador e no celular não. É alguma configuração do celular?
Answer: é porque no texto você definiu o Padrão pt-BR, e no DateAdd vc precisa passar a formatação pois o padrão é US


Question referring to $\sup(D)$ and $\inf(D)$, for set $D$.

Question: Consider set $D$:
$$D = \left\{m + \frac1n\,\middle|\,m,n\in\mathbb{N}\right\}$$
So, the supremum of this set does not exist because  this set is NOT bounded above. I'm confused about the infimum in this case. The set is bounded below but I'm confused about what the greatest lower bound is.
(I am a new user so if you have any suggestions about my question writing, feel free to drop a comment)
Answer: Hint: Every element of $D$ is greater than $1$ but, if $a>1$ there is always an element $d\in D$ such that $d<a$.
Comment: Do both m and n have to be 1? What if m=1 and n=some large natural number?
Comment: @sophisticatedpeter Nice point! I made a mistake and I will edit my answer.


Issue integrating Selenium test suite using jenkins

Question: I am trying to integrate Jenkins with Selenium. I have prepared the following command:
java -jar "D:\CI\TWCP\Selenium plugins\selenium-server-standalone-2.32.0.jar" -htmlSuite *firefox "http://www.google.co.in"  "C:\Users\....\Desktop\Test suites\GoogleTestSuite.html" "C:\Users\.....\Desktop\results.html" -firefoxProfileTemplate "C:\....\Mozilla\Firefox\Profiles\s980v5zn.default"

When I execute the command in the command line, it works Perfectly fine. Then I called the same command through Jenkins and I got the following error:
    Started by user anonymous
Building in workspace D:\CI\Jenkins\jobs\Google Test\workspace
[workspace] $ cmd /c call C:\Windows\TEMP\hudson600399625489206831.bat

D:\CI\Jenkins\jobs\Google Test\workspace>java -jar "D:\CI\TWCP\Selenium plugins\selenium-server-standalone-2.32.0.jar" -htmlSuite *firefox "http://www.google.co.in"  "C:\Users\N_chandanKumar\Desktop\Test suites\GoogleTestSuite.html" "C:\Users\N_chandanKumar\Desktop\results.html" -firefoxProfileTemplate "C:\Users\N_chandanKumar\AppData\Roaming\Mozilla\Firefox\Profiles\s980v5zn.default"
Apr 24, 2013 10:54:07 AM org.openqa.grid.selenium.GridLauncher main
INFO: Launching a standalone server
10:54:08.647 INFO - Java: Sun Microsystems Inc. 1.6.0-b105
10:54:08.647 INFO - OS: Windows Vista 6.1 x86
10:54:08.662 INFO - v2.32.0, with Core v2.32.0. Built from revision 6c40c18
10:54:08.756 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub
10:54:08.756 INFO - Version Jetty/5.1.x
10:54:08.756 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
10:54:08.756 INFO - Started HttpContext[/selenium-server,/selenium-server]
10:54:08.756 INFO - Started HttpContext[/,/]
10:54:08.787 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler@13c1b02
10:54:08.787 INFO - Started HttpContext[/wd,/wd]
10:54:08.787 INFO - Started SocketListener on 0.0.0.0:4444
10:54:08.787 INFO - Started org.openqa.jetty.jetty.Server@7ced01
jar:file:/D:/CI/TWCP/Selenium%20plugins/selenium-server-standalone-2.32.0.jar!/customProfileDirCUSTFFCHROME
10:54:09.941 INFO - Preparing Firefox profile...
10:54:12.041 INFO - Launching Firefox...
10:54:14.242 INFO - Checking Resource aliases
10:54:37.079 INFO - Checking Resource aliases
10:54:37.079 INFO - Received posted results
GoogleTestCase.html
GoogleTestCase

10:54:37.297 INFO - Killing Firefox...
Tests failed, see result file for details: C:\Users\N_chandanKumar\Desktop\results.html
10:54:37.921 INFO - Shutting down...
10:54:37.921 INFO - Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=4444]

D:\CI\Jenkins\jobs\Google Test\workspace>exit 1
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE

result.html shows failure with below log
info: Starting test /selenium-server/tests/GoogleTestCase.html
info: Executing: |setTimeout | 1000000 |  |
info: Executing: |openAndWait | http://www.google.co.in/ |  |
info: Executing: |setTimeout | 1000000 |  |
info: Executing: |pause | 20000 |  |
info: Executing: |verifyTextPresent | India |  |
warn: currentTest.recordFailure: false
info: Executing: |assertTitle | Google |  |
error: Actual value 'Error Message' did not match 'Google'
warn: currentTest.recordFailure: Actual value 'Error Message' did not match 'Google'

If I run the same code adding debug tag i.e.
java -jar "D:\CI\TWCP\Selenium plugins\selenium-server-standalone-2.32.0.jar" -htmlSuite *firefox "http://www.google.co.in"  "C:\Users\....\Desktop\Test suites\GoogleTestSuite.html" "C:\Users\....\Desktop\results.html" -firefoxProfileTemplate "C:\Users\.....\AppData\Roaming\Mozilla\Firefox\Profiles\s980v5zn.default" -debug

I get the following logs at the end
GoogleTestCase.html
GoogleTestCase

10:58:56.737 DEBUG - Handled by org.openqa.selenium.server.htmlrunner.SeleniumHTMLRunnerResultsHandler@14b7453
10:58:56.737 DEBUG - RESPONSE:
HTTP/1.1 200 OK
Date: Wed, 24 Apr 2013 05:28:56 GMT
Server: Jetty/5.1.x (Windows Vista/6.1 x86 java/1.6.0
Cache-Control: no-cache
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/html
Transfer-Encoding: chunked

10:58:57.002 INFO - Killing Firefox...
10:58:57.018 DEBUG - java.net.SocketException: Connection reset
10:58:57.018 DEBUG - java.net.SocketException: Connection reset
10:58:57.018 DEBUG - java.net.SocketException: Connection reset
10:58:57.018 DEBUG - java.net.SocketException: Connection reset
10:58:57.033 DEBUG - java.net.SocketException: Connection reset
10:58:57.033 DEBUG - java.net.SocketException: Connection reset
Tests failed, see result file for details: C:\Users\N_chandanKumar\Desktop\results.html
D:\CI\Jenkins\jobs\Google Test\workspace>exit 1
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE

I have jenkins 1.509 and seleniumHQ2.32.0
Not able to make out how I am getting "java.net.SocketException: Connection reset" Any help in this regard will be appreciated.
Thanks & Regards,
cK
Comment: Some things to keep in mind while configuring UI tests are:  1) The slave should be connected to server preferably by Java Web Start and not let Jenkins control this Windows slave as a Windows service. 2) Use `start java -jar ...` instead of directly giving `java -jar ..` in the batch command. This will let the selenium server to be started in a new command window.
Comment: Shiva, Thanks for your suggestions. But sorry to say it didn't help. It didn't even run properly .

D:\CI\Jenkins\jobs\Google Test\workspace>start java -jar "D:\CI\TWCP\Selenium plugins\selenium-server-standalone-2.32.0.jar" -htmlSuite *firefox "http://www.google.co.in"  "C:\Users\N_chandanKumar\Desktop\Test suites\GoogleTestSuite.html" "C:\Users\N_chandanKumar\Desktop\results.html" -firefoxProfileTemplate "D:\twcpUserProfile" -debug

D:\CI\Jenkins\jobs\Google Test\workspace>exit 0
Finished: SUCCESS
Comment: Sorry it did not help. Just to crosscheck, is there a semicolon really after google.co.in or was it just garbled here?
Comment: No, there is no semicolon , it is just garbled
Answer: To check on the issue I ran the jenkins server in Interactive mode. The setting can be seen when you go to services.msc -> Jenkins process -> properties-> click on logon tab and you can select Local system account and tick "Allow service to interact with desktop".
This made jenkins to run the selenium test interactively instead of running in background.
There I could observe browser failing to open the website. This was tackled by following settings services.msc -> Jenkins process -> properties-> click on logon tab --> Select this account and give the account using which you have logged into this system.


Should I raise this chord up an octave?

Question: Okay, I asked somebody else for advice on my orchestration and he went a bit into lower interval limits. He did however give me advice on orchestration for the first chord. Here is what he gave me:
Low C: Half of the Double basses and the Contrabassoon
Low Eb: Bassoon by itself
Low G: Half of the Double basses and the Bass Clarinet
Octave C: Cellos
Just to clarify I'm talking about this chord:

Now, double basses with a C extension and with a 5th string are getting increasingly common. So it is likely that the double bassists will be able to play that low C with no problems. And I have seen a way that double bassists get an extremely low note to project out by flicking the string before bowing. But I'm wondering what I should do because here is what the lowest register of the double bass looks like in Musescore:

As you can see in the picture, anything below E natural is colored green. In Musescore, green means that the note can't be played by an amateur and red means that the note isn't even in the professional range. More specific to double bass, these low notes being green means that they can only be played if the double bass has either a 5th string or a C extension.
So I'm thinking of perhaps notating the double bass and contrabassoon an octave above the note I want, taking advantage of that built in octave transposition. So in other words I would notate it as a higher C(like the one at the top of the chord in the first image) and then because of the built in octave transposition, it would sound like the bottom note of the chord, C2. But then this raises the question of whether or not I should raise the notes in the other instruments by an octave as well to avoid muddiness in the sound(which is the reason why the person that gave me advice suggested the instrumentation above for the C minor chord).
So, should I raise the entire chord by an octave or just the low C in the double basses and Contrabassoon or should I leave it as is?
Comment: yes, you should raise it an octave. you can just adapt the law of the overtones to your orchestration and arrangements: root, octava 5th, 8th, 3rd, 7th, 9th,

why? look at  the triad in your example in this low position: the overtone of the 3rd  will be interferring with the overtones of the root. We find actually but very seldom triads in not quite as  low position by Beethoven or Bruckner and others.
Comment: As has been said in an answer below: Contrabasses should be notated one octave higher than they sound. As for the deep E flat. You don't need that in the orchestra. Have in mind that on a piano you can not double any notes, so in order to give a good forte sound with "body" you can have such a low E flat in the chord. But in an orchestra you can double any note in different instruments which means you don't have to include the very low E flat.
Comment: @LarsPeterSchultz But the Eb is the note that differentiates the C chord as being C minor. Without it, it would sound like a C major chord suddenly lead to a C minor melody within 2 beats of the chord. Thus in order for it to sound like a C minor chord as in the original piano piece, I have to include the Eb somewhere.
Comment: @Caters I did NOT say that you shouldn't include an Eb. I only said you can exclude that very low Eb. The Eb appears two places in the piano chord since it also appears in the right hand. That Eb (in the right hand) can be doubled with more than one instrument playing it. That is impossible on a piano. The very low Eb gives "body" or volume to the sound on a piano. In an orchestra you can get such "body" or volume with instrumentation.
Comment: @Caters An addition to my comment above. Just be careful with chords in a close position in a very deep register. I am not saying you should never do it, but it can easily be very muddy, so do it with caution. You could study the scores of some of Beethoven's symphonies and see what he does himself when writing orchestral music.
Answer: You seem to be getting very confused about written and sounding pitch.
If you want to play a double bass and contrabassoon to play Beethoven's two lowest notes, you write the notes an octave higher. Playing that is no problem at all for either instrument.
If you write the same notes as the original piano music, they will sound an octave lower, and the combination of the low C and Eb will just be an unpleasant noise.
FWIW three trombones and a tuba would make a much better (i.e. more homogeneous) sound from that chord than a mixture of strings and wind, and are more likely to be all in tune with each other than a mix of different instrument families as well.
You might also consider who you are really writing for here. On the one hand you are worries about "amateurs" whose double basses don't have a low C extension, and on the other you are scoring for contrabassoon which certainly isn't an "amateur" instrument. Half decent contrabassoons cost more than many new cars - and this is sometimes what they sound like on a bad day:
Answer: Do you really need it this low. If so, then write it down there with an ossia an octave higher.  Consider your target ensemble; if you expect an fifth (low) string, go ahead and write down there. If you are writing for high school or other ensemble which should not be expected to have 5-string basses, then re-design the whole passage (or however far you need to keep things smooth.) (I spend lots of time trying out different keys or voicings to match instrument ranges.)

Should I raise this chord up an octave?


Binary matrices such with distinct columns such that after the removal of any one row the columns are no longer distinct

Question: Let $A$ be an $m \times 2^n$ binary (i.e. with entries from $\{0,1\}$) matrix where $m \gg n$, such that all columns of $A$ are distinct $m$-bit bitwords. Suppose further that by removing any row of $A$ we obtain a $(m-1)\times 2^n$ matrix whose columns are no longer unique in the above sense.
For any such matrix one clearly has the upper bound $m < 2^{n}$. What is the best possible upper bound on $m$ in terms of $n$?
Example for $n = 3, m = 5$:
$$\begin{bmatrix}1 & 0 & 0 & 1 & 1 & 0 & 1 & 0 \\
0 & 1 & 1 & 1 & 0 & 0 & 1 & 1 \\
0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 \\
1 & 1 & 1 & 0 & 1 & 1 & 1 & 1 \\
0 & 0 & 0 & 1 & 1 & 1 & 1 & 1 \\ \end{bmatrix}$$

Edit: This is a very closely related question.
Comment: It seems impossible to have such a matrix
Comment: can you give an example?
Comment: @VezenBU I'm sorry I think you are right. I rewrote the question and added an example as well. Hopefully, this question makes more sense now.
Comment: Over what ring / field?
Comment: Does that matter? Why? @RodrigodeAzevedo
Comment: @gen Just curious.
Answer: If $A=[c_1,...,c_n]$ where $c_i$ is the $i^{th}$ column of $A$, Let  $E_k = \{(i,j): c_i + c_j = e_k \}$. So we have $|E_k| \geq 1$, $\forall k$ since removing $k^{th}$ row will make some two columns same. Further $i$ appears only once in $E_k$, since otherwise $c_i + c_j + c_i + c_l = e_k + e_k$ which implies $c_j = c_l$ contradicting distinctness of columns.  If there is a column index $i$ which is not present in any tuple in any set $E_k$ then we can remove that column and tighten the upper bound. Clearly rank$(A) = m$, since we can linearly combine the columns and generate $\{e_1,...,e_{m}\}$. Hence $m > 2^n$ is impossible since $rank(A) = m \leq 2^n$. $m = 2^n-1$ can be attained by matrix, $A=[0,e_1,...,e_m]$.
Comment: Thanks for this answer. As for your example, I'm not sure I follow why that would qualify. Take $n=2$ and consider the example $A$ you gave. Isn't the 4th row of this square matrix all zeros? How can this row discriminate two distinct columns that become identical after erasing this row?
Comment: Hi... there are only 3 rows in my matrix A. ie m=2^n-1 =3 for n=2. Check ur bound because the construction of A i gave beats the bound given in question.
Comment: The matrix A I gave has m+1 columns with first column all 0s and second column e_1 and third column e_2 .... last column e_m
Comment: The example A I gave and the upper bound solves the problem except m=2^n .
Comment: I see, I think you are right, and my earlier bound was wrong. Can you edit your answer a little bit so I can undo my downvote?
Comment: Edited...glad we agreed


does the progression VII > III in minor mean you have modulated to the relative major key?

Question: Is this progression possible in a modal minor chord progression or does it immediately make the III chord the new tonic via an authentic cadence?
Comment: Depends how long the piece remains with III *sounding* like it's a tonic. Could be several bars where III is sounding, but lots of pieces fluctuate between the relatives, without it being conceived that there's been even a modulation, least of all a 'key change'. Cadences are usually found at the ends or phrases, not *in* a progression.
Comment: @Tim indeed, the piece might even have III sounding like it's the tonic for no time whatsoever, for example if III is actually IV/VII.
Answer: This ambiguity is an important feature of the i-III pairing of first and second themes in classical sonatas and baroque (and pre-baroque) overtures. The progression c-f-Bb-Eb, taken in isolation, can be heard as i-iv-VII-III in C minor or vi-ii-V-I in Eb major. Schoenberg's suggestion of "neutralization" (emphasizing the notes that are different between two keys) doesn't work well between relative keys as they have the same notes. Likewise, it can be hard to make the i sound like a tonic; the major key III can sound a lot like a tonic. Perhaps the raising of scale step 7 in minor keys is helpful; V-i sounds cadential but v-i doesn't (though it's a good progression.)
One reason (some books on music history make this suggestion but I cannot remember which ones) that i-III is used for contrast is that the composer has to work a bit to get the III to sound like a different key. Simple ii-V7-I cadences could still be iv-VII-III in the relative. Similarly, it's not as easy to make a C to G modulation sound convincing in C major; the half-cadence V7/V to V7 is identical to the V7 to I in the key of the dominant.
As suggested in Tim's comment, placing cadences at phrase endings helps; a piece starting with i-V-i or I-V-I doesn't sound like a cadence but if rhythmically emphasized, it does.
Comment: Pre-baroque overtures?  What did you have in mind?
Comment: Taking care of things I may have missed.
Answer: From just VII III... no.
You need more context.
In some kind of circle of fifths or other sequential passage, it doesn't necessarily make sense to label any descending fifth progression as a key change.
Length matters too. If it's brief, tonicization is the term. If it's longer, like a whole phrase or period, then a modulation may make sense. Technically, I think you want to ask if there is a cadence, a real cadence. If so, then modulation is a sensible analysis.
Another thing to consider is III VII also does the job of setting up a potential change of tonal center, with all the caveats above about the scope of the change. Examples are Handel's sarabande in D minor, HWV 437. Or, the passamezzo antio form. Both of those work with a type of contracting of i V and III VII in minor. In other words you don't need a authentic cadence like descending fifth, you can also have a half cadence like descending fourth, to suggest a tonic/dominant or dominant/tonic pair.
Answer: The VII-III occurs in almost all minor songs that contain more than only tonic, dominant and subdominant and very often they are part of a fifth-fall progression. In this case music theorists don't speak about a modulation unless the music is not remaining for a longer section in the relative major key and isn't cadencing there e.g. on a fermata or the end of a section separated by a double barline.
Answer: First off, we need to clarify the terminology.    Thinking in C minor, I know a system where vii is B minor (though viiº - B dim - would be more likely),   VII is B major, ♭VII is Bb major.   And I know one where minor chords are notated with a - sign, C minor would be I- rather than i.
But I'm stuck with VII - III.    Is this B major to E major?  (If not, how WOULD you label this?)
See what I mean?  A system that only allows diatonic chords falls down when we stray from the diatonic strait-jacket.  And I'm afraid music DOES, all the time.
But I suspect you mean Bb to Eb.  A visit to the tonic of the relative major, via its dominant.
Yes, there's an authentic cadence in Eb.   Sort of, maybe.   'Cadence' is a structural idea, not just a pair of chords.   This may be a case of 'pass Go, do not collect £200' or 'Just visiting'.   Or the music may stick around the new tonal centre long enough to be considered a modulation.
And it's not really worth arguing over.   Perhaps it modulates, perhaps it just passes through. And when it's a case of relative major and minor, music often moves fluidly between the two with no particular borderlines.


Dependency injection to choose between local and remote data

Question: I need to create a model with dynamic choice of the source of data as a property of a model (code first from database. Code-second).
class MyModel{
public int MyModelId {get; set;}
...
public int PropertyId {get;set;}
public virtual Property Property {get;set;} // this what I need to choose.
...

The Property need to be get from Property table from a database if in a config file it's set <property>remote</property> and from a local container with the same structured as from the database if <property>local</property>.
class Property{
   public int PropertyId {get;set}
   public string name {get;set;}
   public virtual ICollection<MyModel> MyModels {get;set;}
   public Property()
   {
       MyModels = new List<Model>();
   }
}

and the local data is like
 List<Property> lProperty = new List<Property>()
{{PropertyId = 1,name = "val1"},
{PropertyId = 2,name = "val2"},
{PropertyId = 3,name = "val3"} ...}

Comment: Do you have the same horrible formatting in your code?
Answer: At this case you should refuse from EF relationship and write something like this:
public class MyModel
{
    public int MyModelId { get; set; }

    //PropertyId now is not real FK to PropertyTable at DB it is just another column,
    //but it still will store reference to Property
    public int? PropertyId { get; set; }

    [NotMapped]
    public Property Property
    {
        get
        {
            if (PropertyId.HasValue)
            {
                if (ForExampleStaticClass.config("property") == "remote")
                    using (var context = new Context())
                    {
                        return context.PropertyTable.Where(x => x.PropertyId == PropertyId.Value).FirstOrDefault();
                    }
                else
                    return ForExampleStaticClass.lProperty.Where(x => x.PropertyId == PropertyId.Value).FirstOrDefault();
            }
            else
                return null;
        }
        set
        {
            //I consider that Property(value) already exists at DB and/or Local Storage.
            PropertyId = value.PropertyId;
        }
    }
}


Given a 100mbit connection, what is your max downstream speed?

Question: I'm a (very) junior sysadmin and was asked this question on an interview. The answer was 96mbit/s.
I know this is related to network overhead but I cannot find any good information on why this should be 96mbit/s. For IPv4, network overhead accounts for 2% which means this should be 98mbit/s.
Can anyone explain this? Or provide any good info on network overhead? The topic is very hazy for me - why is 2% of the connection taken up by network overhead?
Comment: A pointless question for an interview.
Comment: This question is vague and ambiguous. Should we assume TCP? If so, should the "downstream speed" include only data or TCP address and headers too? If they really wanted a specific, numerical answer (rather than a discussion about what data rates you typically see and why) then it's a very bad question and reflects negatively on the interviewer. (If pressed for a number, I would have answered, "slightly over 90MB/s".)
Comment: Over the years I've heard different figures bandied around for "overhead", anywhere from 1-5%. Realistically it's in the eye of the beholder for example are you looking at network throughput to/from a client or the actual data received. For example if you think about download a file through FTP. You can measure the speed of data to/from the client's network interface or the actual data received via FTP. I think 2% is fair for general network overhead and perhaps another 2% is fair for protocol overhead giving roughly 4%
Answer: Unless this question was used "to see how you think," it's not a great interview question. If they only had one right answer then they're doing it wrong.
The crux of this question is whether you understand protocol overhead or not. The first thing I would ask if given this question is, at what level in the stack is this max downstream speed going to be measured at?
If they said at the Ethernet protocol level then 98Mb/s would have been a very close answer since it's actually 97.53Mb/s (without 802.1q tags) [sidenote: capitalization is important when talking about speeds (e.g. lowercase m is milli, uppercase M is Mega)]
If at the TCP/IP protocol level then things get trickier. You have more choices, is it TCP or UDP, is it IPv4 or IPv6, are TCP timestamps being used or not, are 802.1q tags being used, etc.
For IPv4/TCP/Timestamps off/No options/No 802.1q it would be 94.93Mb/s.
For IPv4/UDP/No 802.1q it would be 95.7Mb/s.
If you want to learn more about protocol overhead I highly recommend this post by Phillip Dykstra on the topic:
http://sd.wareonearth.com/~phil/net/overhead/


Deleting .png File through ListView

Question: I am making an application for personal use that will organize .png files and will let me remotely delete them from the directory through the application (through a ListView).
I have a snippet that will delete the file from the ListView, but not from the actual file directory. I want to be able to do both when I click delete.
private void deleteToolStripMenuItem1_Click(object sender, EventArgs e)
    {
        if (MessageBox.Show("This will delete the file from the folder. Are you sure?", "Warning", MessageBoxButtons.YesNoCancel, MessageBoxIcon.Warning) == DialogResult.Yes)
            for (int i = fileDisplayListView.SelectedItems.Count - 1; i >= 0; i--)
            {
                ListViewItem item = fileDisplayListView.SelectedItems[i];
                fileDisplayListView.Items[item.Index].Remove();
                File.Delete(fbd.SelectedPath + fileDisplayListView.Items.ToString());
            }
    }

Additional snippet for more information..
private void openToolStripButton_Click(object sender, EventArgs e)
    {
        fbd.ShowDialog();
        DirectoryInfo di = new DirectoryInfo(fbd.SelectedPath);
        directoryPath.Text = "Directory: " + fbd.SelectedPath;
        FileInfo[] Files =
            di.GetFiles("*.PNG*", SearchOption.AllDirectories);
        if (Files.Length == 0)
            MessageBox.Show("No .png files found in directory...", "Notification", MessageBoxButtons.OK, MessageBoxIcon.Question);
        fileDisplayListView.Items.Clear();
        foreach (FileInfo f in Files)
        {
            ListViewItem item = new ListViewItem(f.Name);
           this.fileDisplayListView.Items.Add(f.Name);
        }
        this.fileDisplayListView.View = View.Details;
        this.fileDisplayListView.Refresh();
    }

The last part of it, File.Delete(fbd.SelectedPath + fileDisplayListView.Items.ToString());
is not functional. Please help!
Comment: I'm guesing that instead of `File.Delete(fbd.SelectedPath + fileDisplayListView.Items.ToString());` you should have `File.Delete(Path.Combine(fbd.SelectedPath, item.Text));`
Comment: using the Debugger can work wonders. are you capturing the selected File and it's FilePath..?
Comment: @Kedrzu I have tried this and it does not work. MethodMan I have put MessageBox() checkers between the final strings and the file path seems accurate. This seems like a very simple solution but it keeps outputting `FILE_DIRECTORYSystem.Windows.Forms.ListView+ListViewItemCollection`
Answer: This code gets a list of all .jpg files in the directory, adds them to the ListView. By pressing the button it deletes the selected ListView elements and files:
private FileInfo[] files;
public Form1()
{
    InitializeComponent();

    files = new DirectoryInfo(@"C:\Users\User\Pictures").GetFiles("*.jpg", SearchOption.AllDirectories);
    foreach (var file in files)
    {
        listView1.Items.Add(file.Name);
    }
}

private void button1_Click(object sender, EventArgs e)
{
    for (int i = 0; i < listView1.SelectedItems.Count; i++)
    {
        var curentItem = listView1.SelectedItems[i];
        foreach (FileInfo file in files)
        {
            if (curentItem.Text == file.Name)
            {
                listView1.Items.Remove(curentItem);
                file.Delete();
                i--;
            }
        }
    }
}

Comment: I came up with a similar solution but this is functional so I will mark it. Thanks for the help.
Comment: You really shouldn't automatically delete the row from the ListView. There are *many* things that could prevent the `Delete()` call from actually working. Better to refresh the control with the contents of the folder, after the delete.
Answer: This is the snippet I used to fix the application. It is a lot more directly related to how I am approaching the problem.
if (MessageBox.Show("This will delete the file from the folder. Are you sure?", "Warning", MessageBoxButtons.YesNoCancel, MessageBoxIcon.Warning) == DialogResult.Yes)
            for (int i = fileDisplayListView.SelectedItems.Count - 1; i >= 0; i--)
            {
                ListViewItem item = fileDisplayListView.SelectedItems[i];
                string fpath = string.Empty;
                fileDisplayListView.Items[item.Index].Remove();
                fpath = fbd.SelectedPath.ToString() + "\\" + item.Text;
                File.Delete(fpath);
            }

Deleting .png File through ListView


How to correctly calculate forecast KPIs without specific test set?

Question: I have used the TBATS model on my data and when I apply the forecast() function, it automatically forecasts two years in the future. I haven't specified any training set or testing set, so how do I know how to calculate RMSE, MASE, MAE and MAPE?
The data I'm dealing with is Uber travel times data from Jan 2016 to Jan 2020. I have daily data (sampling frequency = 1) for 18 cities and each city has a different sample size (they range from 1422 days to 1459 days).
Here is the Kaggle link to my datasets. I'd recommend using the "final_" datasets instead of the "formatted_" ones.
Here is a subsample:
Date                        MeanTravelTimeSeconds
1   2016-01-02              1441.684
2   2016-01-03              1597.952
3   2016-01-04              1326.496
4   2016-01-05              1355.872
5   2016-01-06              1451.740
6   2016-01-07              1525.991
7   2016-01-08              1657.299
8   2016-01-09              1727.375
9   2016-01-10              1703.772

Here is the example for data for London:

I have set the vector of travel times as an msts object, for it has multiple seasonality, which is used by the TBATS model.
Below is a graph with the fitted values and with the forecast values:

I mainly have 3 questions:

How do I correctly calculate the KPIs for the forecast (RMSE, MASE, MAE, MSE)?
What is the difference between finding out the accuracy of the red line to the black line, and the accuracy of the blue line?
Is what I'm doing called an in-sample forecast or an out-of-sample forecast?

The values I'm getting for each are very low, here's my code:
data <- read.csv('C:/users/Datasets/Final Datasets/final_london.csv', TRUE, ",")
y <- msts(data$MeanTravelTimeSeconds, start=c(2016,1), seasonal.periods=c(7.009615384615385, 30.5, 91.3, 365.25))

fit <- tbats(y)
fc <- forecast(fit)

# RMSE
rmse <- sqrt(fit$variance)

# MAE
res <- residuals(fit)
mae <- mean(abs(res))

# MAPE
pt <- (res)/y
mape <- mean(abs(pt))

# MSE (Mean Squared Error)
mse <- mean(res^2)

Comment: please share your data.
Comment: Time series error can be a tricky thing. It's very easy to confuse yourself Into thinking it's a lot better than it really is. One of the ways that you can double check your error metric is To look at the ratio of absolute predicted difference to absolute lag difference. If your estimate is a lot larger than a one step lag as a predictor then it might not be that great of an estimator.
Comment: I made an edit above (to the post) where I include the link to the Kaggle dataset. There you have access to all cities and I'd recommend using the "final_" ones instead of the "formatted_" ones.
Comment: use `dput(data)` and copy and paste the output from the console into your question please.
Comment: @EngrStudent thanks! Bu my confusion is: when you say absolute difference, are you assuming I have split my data into train and test sets? Because what I am trying to figure out is how to evaluate the accuracy of my forecast for a future for which I don't even have data (2021 and 2022). If I did an out-of-sample forecast for 2020, I could compare the forecast for 2020 with real data for 2020. But I want to know how to measure the accuracy for a period in the not-yet-seen future.
Answer: Strangely, I can not get the tsCV() function to not return NAs. At any rate, the accuracy measures of the forecast can be obtained using the accuracy() function. We can also split into a training and test set as follows:
 library(forecast)
# Read in the csv:
final_london <- transform(read.csv('C:/users/Datasets/Final Datasets/final_london.csv', TRUE, ","),
                          Date = as.Date(Date, "%Y-%m-%d"))

# Expand the series to contain an entry for every day in the year:
final_london_expanded <-
  transform(merge(transform(final_london, estimate = FALSE),
    data.frame(Date = with(final_london, seq.Date(
    from = as.Date(paste0(gsub("\\-.*", "", min(Date)), "-01-01")),
    to = max(Date), by = "days"))), by = "Date", all = TRUE),
    estimate = ifelse(is.na(estimate), TRUE, estimate))

# Interpolate the data for the missing days:
final_london_expanded[] <- lapply(seq_len(ncol(final_london_expanded)), function(i){
    if(inherits(final_london_expanded[,i], "Date")){
      as.Date(final_london_expanded[,i])
    }else if(is.numeric(final_london_expanded[, i])){
      x <- final_london_expanded[,i, drop = TRUE]
      ifelse(is.na(x), approx(x, n = length(x))$y[which(is.na(x))], x)
    }else{
      final_london_expanded[,i]
    }
  }
)

# Isolate the year:
final_london_expanded$year <- as.integer(gsub("\\-.*", "", final_london_expanded$Date))

# Days in year counter:
final_london_expanded$day_in_year_ctr <- suppressWarnings(
  with(final_london_expanded, ave(year, year, FUN = seq.int))
)

# Store the series start year:
series_start_year <- min(final_london_expanded$year)

# Store the first day number of the minimum year:
series_sy_start_day <- with(final_london_expanded, min(day_in_year_ctr[year == series_start_year]))

# Coerce to multiple seasonality time series object:
y <-
  msts(
    final_london_expanded$MeanTravelTimeSeconds,
    start = c(series_start_year, series_sy_start_day),
    seasonal.periods = c(7.009615384615385, 30.5, 91.3, 365.25)
  )
# Chart the decomposition of the series:
plot(decompose(y))

# Isolate the year in which approximately 80 % of the series is represented;
# but still leaves at least a year for the test set:
year_eighty_perc <- with(final_london_expanded, year[round(length(y) *  .8, 0)])

# Determine the cut-off point of the test set:
train_max_year <- ifelse(year_eighty_perc >= (max(final_london_expanded$year) - 1),
                         year_eighty_perc - 1, year_eighty_perc)

# Find the final day in the year in which the test set is cut off:
series_my_end_day <- with(final_london_expanded, max(day_in_year_ctr[year == train_max_year]))

# Determine the starting year and period for the test set:
test_start_year <- train_max_year + 1

# Determine the day in year that the test set should begin:
test_sy_start_day <- with(final_london_expanded, min(day_in_year_ctr[year == test_start_year]))

# Partition the data into a test and training set;
# a training set:
train <- window(y, start = c(series_start_year, series_sy_start_day),
                end = c(train_max_year, series_my_end_day))

# Testing set:
test <- window(y, start = c(test_start_year, test_sy_start_day))

# Chart the decomposition of the series:
plot(decompose(train))

# Apply function to fit the tbats model:
fit <- tbats(train)

# Generate the forecast (to measure against the test set):
fc <- forecast(fit, h = length(test))

# Create a lookup table in order to extract the ts date values:
date_lkp <- data.frame(ts_rep = time(y), Date = as.Date(final_london_expanded$Date),
  mean_tt = final_london_expanded$MeanTravelTimeSeconds)

# Store a data.frame denoting the interpolated values:
date_values_interped <- transform(subset(final_london_expanded, estimate == TRUE,
                               select = c("Date", "MeanTravelTimeSeconds")),
                        Date = time(y)[match(Date, date_lkp$Date)])

# Chart the forecast, actuals and interpolated values:
autoplot(fc, xlab = "Date", ylab = "Number of Trips") +
  autolayer(test, series = "Actuals (test) data set") +
  autolayer(fc$mean, series = "Forecasts") +
  geom_point(data = date_values_interped,
    aes(x = Date, y = MeanTravelTimeSeconds, color = "Interpolated Values"))

# Calculate the accuracy metrics for the model on the trainin and test sets;
# https://stackoverflow.com/questions/11092536/forecast-accuracy-no-mase-with-two-vectors-as-arguments
# http://freerangestats.info/blog/2019/07/20/time-series-cv
# NOTE:  the accuracy measures for the test set are the ones that you want to report:
accuracy(fc, test)

# Regenerate the forecast model using the full data, but the previously selected model:
fcast_model <- tbats(y, model = fit)

# Generate the forecast:
fcast <- forecast(fcast_model)

# Chart the forecast:
autoplot(fcast, xlab = "Date", ylab = "Number of Trips")

# Chart the forecast / data partitions:
autoplot(fcast, xlab = "Date", ylab = "Number of Trips") +
  autolayer(train, series = "Training Data") +
  autolayer(test, series = "Test Data") +
  autolayer(fcast, series = "Forecasts")  +
  geom_point(data = date_values_interped,
             aes(x = Date, y = MeanTravelTimeSeconds, color = "Interpolated Values"))

# Calculate the accuracy metrics of the forecast model (on the training (entire data) set):
accuracy(fcast)

Comment: @LuizScheuer please see my revised answer above where accuracy metrics are calculated on the test set.
Comment: @LuizScheuer Please upvote and accept my answer if you have found it useful / it does what you require.
Comment: Thanks, but when I do `forecast(fc)`, I get the following error message: `Error in ce(actual, predicted) : argument "predicted" is missing, with no default`. Meaning, I need to tell it what data I'm testing it against, which I don't have because I'm not dividing my data into train and test. Simply forecasting the future (to which I don't have data: 2021 and 2022).
Comment: The visualization is a way better one then the one I have because of the inclusion of the test set. Three problems I ran into: (1) you use 365 to specify the last day of the year, but my datasets are not always 365 days for a year. is there a way to specify that the end is the last data point with year 2018? (2) Number 1 results in an error when I calculate the `accuracy(fc, test)` because the fc used training data with e.g. n = 364 but the test data has n = 363. (3) when I run `accuracy(fcast)` it tells me: ``Error in ce(actual, predicted): argument "predicted" is missing, with no default``.
Comment: @LuizScheuer The errors you are seeing you are introducing when translating my code back into your environment. If you want to ensure every year is full and complete please check my code above which expands the series to have values for every day in the series (using linear interpolation).


Creating synthetic data for time series, Hidden Markov Model

Question: Suppose that I have a task of classifying a time series. I decide to use Hidden Markov Model $\lambda(A, B, \pi)$, where $A$ is a transition matrix, $B$ is an emission probability, $\pi$ is an initial distribution. An observed stochastic process looks as below

and there are two unobserved states: State 1 and State 2, also shown on the plot. Let's say, that I would like to fit HMM and test how it performs in terms of states recognition. The best thing I can probably do is to split my data into the train and test set but in a specific manner: the last $n$ observations make up my test set (the part to the right of the dotted green line), as I cannot use crossvalidation due to the temporal structure of the data. But there is a problem: to reliably assess the performance (measured as i.e. accuracy), I need at least a few transitions between the states that this data doesn't provide.
Question: is there any method that allows creating synthetic data for this kind of data? One thing I was considering was to reverse the whole training set, add some gaussian noise and tack on the end of the original training set.
Answer: If you know the parameters $(\pi, A, B)$ of the system you want to simulate, then it should be a pretty straightforward process.
Simply draw a uniform random number and compare it to the probabilities from the starting distribution $\pi$ to generate an initial state. Then draw a random number and generate an outcome according to the probabilities for the current state by comparing it to the outcome probabilities in $B$. Draw another random number, and transition to a new state (or don't) according to the probabilities in $A$. Generate another random number and compare it to $B$ to generate an outcome for this second state. Draw a new random number to simulate a state transition… etc. Simply continue this process until you reach your desired sequence length.
I'm happy to elaborate on any of these steps in the comments if you have any questions. It's possible that I am misunderstanding your question.
Comment: Thanks for the response. The initial goal is to assess the performance of my classifier. In order to do that, I simply need more transitions between the states and it therefore more data, so I cannot really generate it from my model.
Comment: If you do the process that I described, you can then estimate the parameters from your new simulated dataset. If you need a lot of transitions, I would suggest simulating ~1000 sequences and doing the estimation on each one. Some will have more transitions and some will have less, and the parameters you estimate will vary on each simulation. But since you know the true parameters that generated them, you can use that knowledge to accuracy and precision of the estimator. This is called Markov-chain Monte Carlo
Comment: You can also use it to ask questions about classification accuracy, by looking at the hidden states that you infer and comparing them to the true states (that you know because the data are simulated.) The question this answers is roughly: Assuming that my parameter estimates are correct, how good will this classifier perform on other data generated by this process?


Como faço para imprimir a quantidade de vezes que um item de um array aparece

Question:
função deve ser escrita para receber somente um parâmetro: um array de notas. Ela deve retornar também um array com três elementos:

O primeiro, com a quantidade de notas iguais a 0 ou 1. Seriam os que não gostaram
O segundo, com a quantidade de notas iguais a 2 ou 3. Seriam os que acharam mediano
O terceiro, com a quantidade de notas iguais a 4 ou 5. Seriam os que gostaram
notas = [1,1,3,3,5,5]

function calculaGostos(notas){
   for(i =0; i <notas.length; i++){
       if(notas[i] >=4){
            var nGostaram = [notas[i] >=4]
        }if (notas <=1) {
            var nNaoGostaram = [notas[i] <=1]
        } else {
            var nMediano = [notas[i] ==2||notas[i] ==3]
        }
for each()

        return [nGostaram,nNaoGostaram,nMediano]
    }
}

Comment: Olá Bianca, sua pergunta não está clara, e o código que você colocou como exemplo, por estar errado não ajuda a esclarecer que é o problema, você poderia adicionar detalhes a sua pergunta para que possamos te ajudar
Comment: Desculpa Leo, é que nem eu entendi muito bem a lógica do exercício. Mas o que eu queria saber é como faço pra retonar quantas vezes um item aparece dentro de um array. Muito obriga pela atenção.
Comment: Você pode adicionar detalhes como por exemplo, como é este array, se é de números ou string, como você faz pra saber qual item você está procurando, se o item é passado como parâmetro... Coisas do tipo.
Comment: Reorganizei o código pra ficar mais fácil de ler e tentei utilizar o for each mas não sei a sintaxe, tentei multiplicar as variáveis dos if pela quantidade de elementos do array também.
Answer: O seu código faz uma coisa mas a sua pergunta diz outra coisa, de qualquer maneira irei demonstrar algumas maneiras de resolver o que acredito ser o seu problema, este é o nosso array de notas notas = [1,1,3,3,5,5], a sua função pode ser feita no seguinte modo:

const notas = [1,1,3,3,5,5];

function calculaGostos(notas){
  // Declare as 3 notas necessárias com valor inicial 0
  let nGostaram = 0;
  let nNaoGostaram = 0;
  let nMediano = 0;

  // Dentro ao for você não declara nenhum variável, mas usa as que acabamos de criar for do for, do contrário, variáveis criadas dentro do for, existem somente dentro do for
  for(i =0; i < notas.length; i++){
     if(notas[i] >= 4){
          // O operador ++ equivale a nGostaram = nGostaram + 1;
          nGostaram++;

     } else if (notas[i] <= 1) {
          nNaoGostaram++;
     } else {
          nMediano++;
      }
  }

  // Return é última instrução dentro a uma função, está a indicar que a função terminou a sua execução e deve retornar um valor
  return [nGostaram,nNaoGostaram,nMediano]
}

console.log(calculaGostos(notas)); // [2, 2, 2]

O exemplo acima é uma maneira que pode ser usada para fazer o que você precisa, mas você pode facilitar a tua vida, por exemplo a função termina retornando um array de números, você não vai sempre se lembrar o que cada número significa e todas as vezes você vai procurar a função para saber o que ela te retorna? Vamos melhorar isso!

const notas = [1,1,3,3,5,5];

function calculaGostos(notas){
  // Declare as 3 notas necessárias com valor inicial 0
  let gostaram = 0;
  let naoGostaram = 0;
  let mediano = 0;

  // Dentro ao for você não declara nenhum variável, mas usa as que acabamos de criar for do for, do contrário, variáveis criadas dentro do for, existem somente dentro do for
  for(i =0; i < notas.length; i++){
     if(notas[i] >= 4){
          // O operador ++ equivale a nGostaram = nGostaram + 1;
          gostaram++;

     } else if (notas[i] <= 1) {
          naoGostaram++;
     } else {
          mediano++;
      }
  }

  // Mudamos os nomes das variáveis só para serem mais legiveis
  // Retornamos um objeto e não um array
  return {gostaram, naoGostaram, mediano}
}

// Bem mais fácil de ler
console.log(calculaGostos(notas)); // {gostaram: 2, naoGostaram: 2, mediano: 2}

Reseve um pouco do teu tempo para ler um pouco sobre estes métodos que podem e serão úteis durante a tua vida de programadora Javascript:

Array filter
Array find
Javascript Operators
Javascript Objects
Guia de sobrevivência do Stack Overflow em Português
Comment: Muito obrigada! A primeira forma funcionou perfeitamente.