R package tmap: unable to overlay points on line & unable to show points in view mode

Question: I have developed code to identify where freeway ramps terminate on surface streets and I would like to show those points on a map which also shows the freeways and ramps to check whether my code is correct.
I created an simple features (sf) object of freeways (motorways) and ramps (motorway_links) for a city from open streets maps using the osm package. My code identifies the coordinates of the points where the ramps terminate on surface streets and creates an sf object of the points. The coordinates in both sf objects are geographic. They are consistent because the ramp termini nodes are selected from the link endpoint coordinates.
I have checked that both objects are of class 'sf'. I can plot both sf objects using the plot command:
plot(st_geometry(Hwy_sf))
plot(st_geometry(RampTerminiNodes_sf), add = TRUE, col = "red")

I would like to plot them using the tmap package in 'view' mode to be able to zoom into areas to confirm that the ramp termini are correctly identified but I am running into 2 problems.
First I can't get ramp termini points to plot over the highway lines in tmap. I can get each layer to plot separately but can't get the point layer plotted on top of the line layer. This works to show the highways as lines:
tmap_mode("plot")
tm_shape(Hwy_sf) + tm_lines(lwd = 2)

And this works to show the ramp termini as points:
tmap_mode("plot")
tm_shape(RampTerminiNodes_sf) + tm_bubbles(size = 0.1, col = "red")

But this doesn't work (the lines are displayed but the points are not):
tmap_mode("plot")
tm_shape(Hwy_sf) + tm_lines(lwd = 2) +
  tm_shape(RampTerminiNodes_sf) + tm_bubbles(size = 0.1, col = "red")

Second, I haven't been able to figure out how to display the point data in "view" mode. This doesn't work. The plot area shows the message: "Map data not yet available"
tmap_mode("view")
tm_shape(RampTerminiNodes_sf) + tm_bubbles(size = 0.1, col = "red")

System info is:
Windows 10
R version Rx64 4.0.2
R Studio version 1.3.1093
Answer: Solved the problem. I had an error in my code that changed the projection of the RampTerminiNodes_sf object. Correcting that solves the problem.


Parent does not wait for worker threads to complete task

Question: In my program, the main thread creates 4 (or more) worker threads. At some point the parent (main thread) has to wait for the workers in order to complete some calculations. The threads are running on an endless loop so I can't use the pthread_join(.., ..) POSIX function in order to wait until the workers are done. So I am using a global counter and a condition variable.
Main Thread Code
unsigned jobs = 0; // global variable

// global mutex and cv. They get initialised in my main.
pthread_mutex_t counter_mutex;
pthread_cond_t  counter_cv;

static void process(..){

    jobs = myArray.size();

    // I am using a function here that broadcasts a cv in order to
    // wake up the workers

    pthread_mutex_lock(&counter_mutex); // lock counter
        while (jobs > 0){
            pthread_cond_wait(&counter_cv, &counter_mutex); // PARENT SHOULD GET STUCK HERE TILL WORKERS ARE DONE
        }
        // cout << "Workers are done" << endl;
    pthread_mutex_unlock(&counter_mutex);   // unlock counter
}

Workers Code
 extern unsigned jobs;
 extern pthread_mutex_t counter_mutex;
 extern pthread_cond_t  counter_cv;

 void *run() {

    for (int i = 0;; i++) {

        // do some calculations here

        pthread_mutex_lock(&counter_mutex); // lock counter
            jobs--;
            if (jobs == 0){
                pthread_cond_signal(&counter_cv);
                cout << "All jobs are done" << endl;
            }
        pthread_mutex_unlock(&counter_mutex);   // unlock counter

    }
}

The problem is that sometimes my main thread does not get stuck at the condition variable in order to wait for the workers and that causes sometimes a segmentation fault. Is there any race condition that I can't see and that causes this problem?
Comment: How are you setting jobs initially?
Comment: jobs = myArray.size(); .. It's the number of the items in the queue/array that the workers pop and calculate.
Comment: Are there ever any threads active when you set jobs? There could be a race condition there since it's not locked. I'm not sure how that would cause the main thread to exit early though.
Comment: The threads are always active but stuck at another condition variable till the main thread pushes items to the queue/array.
Comment: Are you sure that the segfault is caused by the main thread not waiting? The code you've posted looks like it should work. In gdb you can use a breakpoint after the main thread gets past the condition and run `info threads` to see where are the threads are to confirm.
Comment: Yes I used gdb and 1 out of 20 times that I will execute the program, the main thread wont wait at the cv. At least that is what gdb says. Maybe the problem is not there. Thx anyway
Comment: Regardless of whether the main thread waits or not it should never get past the while loop until jobs is 0 (or less). You should log the value of jobs before and after you change it anywhere and see what the output is. Maybe that will help narrow down the issue.
Comment: Do you have more working threads than jobs?
Comment: That was EXACTLY what I was thinking right now. I do not know the exact jobs size because I have multiple "job data flows" so I thought that if there are 3 jobs and i wake up 4 threads then the 3rd one will signal the cv and the 4th one will cause the segmentation maybe.
Comment: 3 jobs, 4 threads. I wake them all up. The 3 worker threads pop from the queue. The 4th pops what?? Then the third one decreases the jobs counter to 0, then signals the main thread and the 4th causes the sigsegv.
Answer: There is an obvious concurrency issue in your code. For the sake of simplicity, imagine that we have 2 workers, and 2 jobs left.  The following scenario could happen:

Job count is set to 2
The parent is stuck waiting for the condition variable
Both workers start a processing iteration in their endless loop
The first worker finishes its task faster for whatever reason, and executes the locked job count update. The job count is decremented to 1, so the parent is left waiting and the lock is released.  The worker then starts its next iteration. Unfortunately in reality there is no job left (1 was done, and 1 is currently performed by the second worker). But it starts an hypothetical job (perhaps addressing an array element that was in the meantime deleted or out of range ?)
The second worker finishes its job, and the job count being decremented to 0, causing the parent to be awakened, holding the lock, exit the loop and unlock.
You noted that while parent thinks it is over, the first worker still processing an hypothetical task, maybe trying to access myArray whereas it gets cleaned up, or whatever else could go wrong.

So you have two opportunity for races and/or segfalts: in the parent and in the still active worker, busy with a task that doesn't exist.
I think the worker loop would be much safer if you'd start the loop getting the lock, checking if there are still jobs remaining, and decrement upfront the job count, so that peer worker will know what's really left:
void *run() {

    for (int i = 0;; i++) {
        pthread_mutex_lock(&counter_mutex); // lock counter
            if (jobs == 0){
                pthread_cond_signal(&counter_cv);
                cout << "All jobs are done" << endl;
            }
            else jobs--;
        pthread_mutex_unlock(&counter_mutex);   // unlock counter

        // do some calculations here

    }
}

The advantage is that worker only work if there is really work left.  The only trouble then, is that the parent is awaken by the first worker out of job. Other workers could however still be running.
If this is a problem you could for example maintain also a counter of still active jobs, and make the parent loop on  (jobs>0 || active_jobs>0)
Comment: There is a part of the code that I didnt add on the post where the workers when poppin from the queue/array check the size of it. So it is not possible to pop if the queue is empty. So when the first thread pops the size is now 1 but the jobs is not yet decremented. The 2nd thread pops and now the size is 0. 1st thread ends faster but decrements the jobs to 1 but it wont pop again cause the size of the queue is 0. I think my problem was caused by "spurious wake up" on a cond var in another part of my program. I edited it and it seems like it works. Thx
Comment: @pirox22 is the popping from the queue and size checking protected by the mutex ?
Answer: The only trouble I see with your bits of code is that each worker will run like crazy (possibly at 100%CPU if your processing does not involve timers/IOs), and won't terminate once there are no more jobs.
In a real case your worker should get a job request from a queue or something similar, and block (waiting for an available job or some termination command) when there are no available jobs to process.
I see no race condition here. The only bug I see is the systematic decrementation of the job count. It will soon become negative and that could cause all sorts of troubles if you assume it cannot. To avoid that, simply test the counter to zero before decrementing it.


Is there a way to access memory map as vector?

Question: TL;DR I have a large list of terms in a file which I have memory mapped. Can I access these terms as if they were a vector in memory. E.g., std::string term = terms[1];
The File
I have a large binary file (which I create) containing millions of terms arranged like so:
applebarnhouseskytrain...

The header is a UInt64 int specifying the number of terms, followed by the same number of UInt64 ints, each specifying a byte offset for each term (starting at zero). The header for the previous example would look something like this (although in binary):
5 0 5 9 14 17

So the binary file altogether would look like this. I am open to modifying the way this data is stored, if that will help.
The Memory Map
I am using the mio library to memory map this file. I can iterate over the entire file, or access any individual byte. E.g.,
mio::mmap_source in;
in.map("index", error);
for (auto& b : in) {
    std::cout<<b;
}
std::cout << in[3];

I read the offsets in the header into a vector in memory:
std::vector<int64_t> offsets;

Accessing a Term
Currently, if I want to access, say, the second term, I would first get the offset of the term offset[1] and the offset of the subsequent term offset[1+1]. Then I would create a std::string term; by looping through in[] using the offsets I just obtained.
Questions

Is there some datatype I can define so that I can access each term as if it were in an vector/set/map without loading actual terms into memory? E.g., std::string term = terms[1]; //assigns "barn" to term
Am I going about this all wrong? Is there a faster, more efficient way of getting a term from a file? I have been thinking about the above for so long, I feel I might be missing other obvious options.
Comment: There's nothing built-in that does this. You can definitely define a `word_view` class or something that does this though.
Comment: You can use `std::string_view` to describe each term - you'll still need to write your own top-level container to look up the offsets and construct that view, though.
Answer:
Is there some datatype I can define so that I can access each term as if it were in an vector/set/map without loading actual terms into memory?

No.
With a custom allocator you could make a standard container use mapped memory for storage, but you couldn't use the data that was in the memory before the container took ownership.
You have to create a custom class for this purpose.
Comment: Thanks, I've been working on making a custom allocator, which if I can get working, I'll benchmark against just letting my OS put things in the swap file. A follow-up question, though, what do you mean by "you couldn't use the data that was in the memory before the container took ownership"?
Comment: @B.Wooster The container initialises objects into the memory. Whatever is in the memory will be overwritten. So it is not useful for reading a mapped file.


Getting a true 95th percentile in DataDog

Question: I have an application that publishes a metric to DataDog with multiple tags, and my DataDog agent has a line that looks like
histogram_percentiles: 90, 95, 99

So my metric (lets call it ResponseTime) has a metric in the DataDog viewer for each of those (i.e. ResponseTime.90perentile).
However if you look at this metric carefully it appears to be calculating these percentiles on a short range (not sure what) and for each tuple of the tags that exist.
Ideally what I'd like to get is a 95th percentile of the ResponseTime metric over all the tags (maybe I filter it down by 1 or 2 and have a couple of different graphs) but over the last week or so.  Is there an easy way to do this?
Answer:
However if you look at this metric carefully it appears to be
  calculating these percentiles on a short range (not sure what) and for
  each tuple of the tags that exist.

The short range that you have noticed is actually the flush interval which defaults to 10 seconds. As per this article on histogram metric by datadog,

It aggregates the values that are sent during the flush interval
  (usually defaults to 10 seconds). So if you send 20 values for a
  metric during the flush interval, it'll give you the aggregation of
  those values for the flush interval

For your query -

Ideally what I'd like to get is a 95th percentile of the ResponseTime
  metric over all the tags (maybe I filter it down by 1 or 2 and have a
  couple of different graphs) but over the last week or so. Is there an
  easy way to do this?

as per my reading of the datadog docs, there isn't a way to get this done at the moment. It might be a good idea to check with datadog support regarding this.
More details here.
Answer: Datadog now supports the distribution metric type which aggregates data on the Datadog server-side instead of in flash intervals on the agent side (like histograms). This allows for true p95s, etc. to be calculated over any time period.
After starting to emit a distribution metric, you'll need to enable the percentiles for the metric on the Metric Summary page:

Then you should be able to see the percentiles on the graphing editor:

You'll also need to use a new name for the metric if you change it from a histogram to a distribution metric.


Can a baserunner "steal" first base?

Question: How was this play able to happen?
Answer: From MLB stats:

A stolen base occurs when a baserunner advances by taking a base to which he isn't entitled. This generally occurs when a pitcher is throwing a pitch, but it can also occur while the pitcher still has the ball or is attempting a pickoff, or as the catcher is throwing the ball back to the pitcher.

Note that steals happen (or are at least initiated) when the ball is not yet put in play by the batter.
In the video, Baez puts the ball in play and is therefore entitled to first base.  It's poor defense (both execution and decisions), but it's just the batter becoming a runner and advancing safely to first base (and then advancing to second on the bad throw).  No steal.

Can a baserunner "steal" first base?


Pegar somente elementos iguais em uma matriz

Question: Tenho uma matriz por exemplo:
1 4 7
1 3 4 6 7
1 2 3 4 5 6

Tenho que ver os números que se repetem em TODAS as linhas, no exemplo, deveria ser resultado os números: 1 e 4. Pois são eles que repetem em todas as linhas.
Vector<Integer> validos = new Vector<Integer>();
int cont = 0;
for (int linha = 0; linha < resultado.length; linha++)
{

    for (int coluna = 0; coluna < resultado.length - 1; coluna++)
    {
        for (int linha2 = (linha + 1); linha2 < resultado.length; linha2++)
        {
            for (int coluna2 = 0; coluna2 < resultado.length - 1; coluna2++)
            {
                if(resultado[linha][coluna] == resultado[linha2][coluna2])
                {
                    cont ++;
                }
            }
        }
    }
}

Onde a matriz é a variável resultado, que é um int[][], tentei implementar um cont que se houvesse o número em todas as linhas ele então iria valer o número de linhas da matriz, exemplo, 3.
Só que o código não está funcionando.
O objetivo é pegar esses números repetidos e colocar no Vector validos
Comment: O três não aparece na primeira linha. :p
Comment: Erro de digitação kk é 1 e 4 no caso
Comment: Certeza renan? Onde ta isso? Porque é estranho, mesmo usando Java8, o Eclipse não mostra ela como deprecated, como alguns métodos do Time, etc.
Comment: @lvcs Não tenho como postar links aqui, estou pelo celular. Mas no Google vc encontra muito sobre o assunto.
Answer: Não sei bem qual linguagem você escreveu. De todo modo escrevi em C#, você vai saber fazer as adaptações com os nomes de classe List, Vector, etc.
No final do código, na variável int[] validos terá apenas 1, 2, 3. Como você quer, mas tive que adicionar o 3 no primeiro nível da matriz porque não existia.
        int[][] vetor = new int[][] {
            new int[] { 1, 3, 4, 7 },
            new int[] { 1, 3, 4, 6, 7 },
            new int[] { 1, 2, 3, 4, 5, 6 }
        };

        List<int> validosList = new List<int>();

        for (int i = 0; i < vetor.Length; i++)
        {
            for (int j = 0; j < vetor[i].Length; j++)
            {
                int numero = vetor[i][j];

                if (validosList.Contains(numero))
                {
                    continue; //Número já validado. Pula essa verificação.
                }

                int contagem = 0;

                for (int k = 0; k < vetor.Length; k++)
                {
                    for (int l = 0; l < vetor[k].Length; l++)
                    {
                        int numeroEmVerificacao = vetor[k][l];

                        if (numero == numeroEmVerificacao)
                        {
                            contagem++;
                            break; //Vai pra próxima linha do array
                        }
                    }
                }

                if (contagem == vetor.Length)
                {
                    validosList.Add(numero);
                }

            }
        }

        int[] validos = validosList.ToArray();

Comment: A linguagem estava na tag, é java.
Comment: Adaptei para Java, funcionou perfeitamente!


BigCommerce Stencil - GraphQL query using front matter not returning anything

Question: I'm not sure if it's a bug, but I'm not able to make GraphQL work in the Cornerstone template. I'm expecting an error or something getting returned at least, but nothing is being rendered at all from gql.
I am on the pages/product.html template, and I even tried this example from the docs:
---
product:
   videos:
       limit: {{theme_settings.productpage_videos_count}}
   reviews:
       limit: {{theme_settings.productpage_reviews_count}}
   related_products:
       limit: {{theme_settings.productpage_related_products_count}}
   similar_by_views:
       limit: {{theme_settings.productpage_similar_by_views_count}}
gql: "query productById($productId: Int!) {
 site {
   product(entityId: $productId) {
     variants(first: 25) {
       edges {
         node {
           sku
           defaultImage {
             url(width: 1000)
       }
      }
     }
    }
   }
  }
 }
 "

My goal is to have access to the paths/URL on each of the product's category because product.category is just an array of category names. Here's the query I am able to make work on the GraphQL playground (86 to be replaced by $productId in the front matter GraphQL query, I think?):
query getProductCategories {
  site {
    product(entityId: 86) {
      categories {
        edges {
          node {
            name
            path
          }
        }
      }
    }
  }
}

If there's no way around this, maybe I'll just try to do the fetching in the client side.
Comment: There's a bug at the moment where the GQL results do not work when viewing the theme locally in stencil-cli, but they work when you upload the theme. The CLI bug is being fixed soon.
Answer: This now works correctly, as of 20-Sep-2021.
There was a bug, tracked as an issue here: https://github.com/bigcommerce/stencil-cli/issues/732 which has been resolved and closed.


How can I calculate the relative probability of each move resulting in a win, given historic win/loss data for the possible moves?

Question: We are at a position in a game where there is a decision to be made.  As there are relatively few game positions and possible moves, we have collected statistics from previous occasions the game was played and we found ourselves with the identical game position.
For each move, we have recorded if selection eventually led to a win or a loss.
Move     Win    Loss
 A        3      5
 B        2      4
 C        0      0
 D        0      4

My goal is to randomly select a move while proportionally favouring those moves more likely to win.  So for each of the possible moves, I would like to calculate the probability that that move is the most-winning one.  Let's call these PA, PB, PC and PD.  These should sum to 1.
Give the example data above, my intuition says:-

Move A should be the most favoured, and so PA is the largest of the values.
Move C has no data.  It could be anything between always a loss and always a win.  Might need to be treated as a special case as there is nothing from which to calculate?
Move D appears poor so far.  Perhaps this is just sampling luck, and over time we would fnid ourselves with 12 wins and 4 losses.  There is still some chance it is really the most winning move.

Now I get a stuck.  It's seems as though I should calculate the win ratio for each of the moves and apply some certainty-factor based on the total number of times that option has been selected to come up with distribution representing the likely win ratio for the underlying population.  How would I combine these overlapping distributions and reduce them down to probabilities for each move?  I'm not sure how I would combine all of these numbers to reach PA, PB, PC and PD.  This seems a reminicent of an ANOVA?
In the actual problem there can be between 1 and 7 moves available.  I'm assuming any answer can be generalised up to more possible moves.  If it makes any difference, the game is not circular; having made a move we can never return to the same position within the game.  It could only be reached again in a new game.  There are, however, multiple ways to reach the same game position from the start of the game.
Comment: Sounds like a [simple version](http://en.wikipedia.org/wiki/Multi-armed_bandit#Variations) of a [multi-armed bandit](http://en.wikipedia.org/wiki/Multi-armed_bandit) problem  - note that this is also a [tag](http://stats.stackexchange.com/questions/tagged/multiarmed-bandit) here, so you could search on that, though not everyone has used the tag, as [a search on 'bandit' finds more](http://stats.stackexchange.com/search?q=bandit)
Comment: Is there anyway I can make this question clearer or more answerable?
Answer: A big problem is that the probability of making a move and the probability of winning after making the move may not be independent. There may actually be only a tiny difference in the quality of the moves, but good players tend to make move A or slightly less often B, while bad players tend to make move D - the players who made move A or B might have done quite well doing move D as well, and the players who made move D might have lost after making move A or B as well.
With move C, it's not as if you have no information. You have the information that in 18 games, nobody made move C. That shows that out of 18 players, nobody believed that C was the best move. Which seems to indicate that it is indeed not a very good move. For the other moves, the more people picked that move, the more successful it was.
Answer: The first step is to get the probability of winning conditional on each move.  This is simply the proportion win / (win + loss) for each choice.  Regarding move C, you can use the overall probability of winning:  The game was played 18 times and won 5 times, so you could set its probability of winning to 5/18.
Then the probabilities of winning conditional on move are: A = 3/8, B = 1/3, C = 5/18 and D = 0.
You might want to collect more data on Move D to discover if there is a nonzero probability.  Otherwise in your random draws it will never be selected.
Next you want to make the random draws.  To construct the distribution for this you simply normalize the above probabilities such that they sum to one.  Define x = 3/8 + 1/3 + 5/18 + 0.  Then the probability of selecting A = (3/8) / x. The probability of selecting B = (1/3) / x. Etc.
Comment: Is there no need to take into account the number of times a Move was made and therefore how accurate the winning probability is when comparing options?  Consider 999 wins:1 loss vs. 1 win:0 losses.
Comment: To make random draws there is no role for quantifying accuracy (i.e. no need for standard errors).  It may be interesting to examine standard errors but they play no role in establishing the estimated probabilities.  If for some other reason you need to quantify the precision just look up standard errors for proportions, for example [here](http://www.stat.wmich.edu/s216/book/node70.html).
Comment: OK, in that case I was making it more complicated than it needed to be. Link was interesting anyway because that is what I thought I needed to know.
Answer: Normally this type of problem can be solved by regularizing the probability distribution. In brief, you can add a constant (usually 1 or 1/2) to the observed outcome frequencies for every condition.
So the probability of winning after move A would be calculated as p(w|A)= (3+1) / (3+1 + 5+1), p(w|D)= (0+1) / (0+1 + 4+1), p(w|C)= (0+1) / (0+1 + 0+1).
This solution in general works fairly well. I am not sure that applies to your case, but it is possible that there are situations where the the probability of winning is very high/low regardless of the move (e.g., an already won-lost game), or the observed frequency of a move is related with the probability of winning if the movie was played (e.g., no chess player would start a game with e.g. A4).
If these or similar considerations are relevant, you might want to adjust the constants added to the observed frequencies. In the first case you could weight the constant added to the observed probability of winning / loosing by the expected outcome in the specific position (e.g., if the probability of wining is 9 out of 10, you might add 0.9 to the observed frequency of winning and 0.1 to the frequency of loosing). In the latter, you might decide to penalize the moves that have never been chosen (e.g., 0.1 to the probability of winning, 0.9 to the probability of loosing).


Tenhho uma função Javascript fuciona o no Chrome mas não no Firefox

Question: Tenho uma função javascript que não deixa escrever números, somente letras, no chrome ela funciona exatamente como é proposto, agora no mozilla não funciona, abaixo minha função:
function soletra(event) {
    var value = String.fromCharCode(event.which);
    var pattern = new RegExp(/^[a-záàâãéèêíïóôõöúçñ ]+$/i);
    return pattern.test(value);
}

eu fiz uma adaptação para funciona, porém não ficou bom que deixa algumas teclas inativa se o campo estiver em foco:
function soletra(event) {
    if(event.keyCode == 9 || event.keyCode == 8 || event.keyCode == 46
        || event.keyCode == 39 ||event.keyCode == 37 || event.keyCode == 38 || event.keyCode == 40)
        return;
    var value = String.fromCharCode(event.which);
    var pattern = new RegExp(/^[a-záàâãéèêíïóôõöúçñ ]+$/i);
    return pattern.test(value);
}

e o chamado da função está assim:
$("#id").bind("keypress paste drop", soletra);

Answer: Eu já tive esse problema com Firefox, depois de pesquisar a solução que achei foi usar isso:
var key = event.keyCode || event.which;

No caso do Firefox, se o keyCode não funcionar, usa o which, e também funciona para os outros navegadores, então você usa varável key. Li que em alguns eventos de teclado diferentes (keypress, keyup, etc), keyCode não retorna um valor, deve ser o caso do seu problema.

$("#i1").bind("keypress", function(event) {
  var key = event.keyCode || event.which;
  $("#i2").val(key);
});

  Digite aqui:
  <input id='i1' type='text'>

  Código da tecla:
  <input id='i2' type='text'>

Veja o exemplo na documentação do mozilla.org, que tem exatamente essa linha de código acima: https://developer.mozilla.org/pt-BR/docs/Web/API/KeyboardEvent
Referência para o SO em inglês: event-keycode-not-working-in-firefox
Comment: Estou tentando implementa essa linha, porém ele continuar a funcionar da mesma forma obrigado pela ajuda em relação ao doc
Comment: @BrunoH aqui você pode ver um exemplo funcional: https://jsfiddle.net/jamfLktj/  e coloquei também um exemplo na resposta


Apply SequenceMatcher to DataFrame

Question: I'm new to pandas and Python in general, so I'm hoping someone can help me with this simple question. I have a large dataframe m with several million rows and seven columns, including an ITEM_NAME_x and ITEM_NAME_y. I want to compare ITEM_NAME_x and ITEM_NAME_y using SequenceMatcher.ratio(), and add a new column to the dataframe with the result.
I've tried to come at this several ways, but keep running into errors:
>>> m.apply(SequenceMatcher(None, str(m.ITEM_NAME_x), str(m.ITEM_NAME_y)).ratio(), axis=1)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python33\lib\site-packages\pandas\core\frame.py", line 4416, in apply
return self._apply_standard(f, axis)
  File "C:\Python33\lib\site-packages\pandas\core\frame.py", line 4491, in _apply_standard
raise e
  File "C:\Python33\lib\site-packages\pandas\core\frame.py", line 4480, in _apply_standard
results[i] = func(v)
TypeError: ("'float' object is not callable", 'occurred at index 0')

Could someone help me fix this?
Answer: You have to apply a function, not a float which expression SequenceMatcher(None, str(m.ITEM_NAME_x), str(m.ITEM_NAME_y)).ratio() is.
Working demo (a draft):
import difflib
from functools import partial
import pandas as pd

def apply_sm(s, c1, c2):
    return difflib.SequenceMatcher(None, s[c1], s[c2]).ratio()

df = pd.DataFrame({'A': {1: 'one'}, 'B': {1: 'two'}})
print df.apply(partial(apply_sm, c1='A', c2='B'), axis=1)

output:
1    0.333333
dtype: float64

Comment: @user1272739 assign with `df['sm'] = df.apply(...)`
Comment: Thanks! Is there any way that I could add the SequenceMatcher results as a new column in the existing dataframe? Otherwise I would have to merge the results back in.
Comment: I tried that code but it throws me an error: TypeError: 'float' object is not iterable. What could be wrong?

Apply SequenceMatcher to DataFrame


Dropped new Martin DCAE

Question: So, first day with the new Martin, decided to tune and take some on line lessons. While tuning the G chord, string snapped, scared the hell out of me and dropped the flippin guitar! Now on the lower body of the guitar, the top has a slight opening where it meets the side. What do I do?
Answer: If the top has come loose near the point of impact, or if any of the braces have come loose, you should probably take it to a luthier for repair. Unless you know a bit about glues and lutherie, I would encourage you to not attempt your own repair job in such a case.
It may be that only the binding has cracked, in which case you can probably just continue to play it. If you hear new rattles when you are playing, you'll know that the guitar needs some work. In any case, even if the guitar appears OK, it might be best to let a luthier take a look at it when you get a chance; they can repair the binding by splicing in a new section and tell you if it needs more work done.
This is kind of a bummer at the beginning of your journey, but in the future remember that strings break. They rarely pose any danger or cause any harm; get used to that happening (although I can't remember the last time I broke a string), and maybe wear a strap to prevent accidental droppings.
Comment: Agreed. luthier. Bummer. Lesson learned. Strap.
Comment: I wouldn't wait to see if it rattles first, I'd just take it for a professional opinion. If he says it's only cosmetic, fine & dandy. If not, you could save a far more expensive repair later.
Comment: @Tetsujin - agree that it needs a look at now, but can't see how it would get worse later - unless it gets dropped again.
Comment: @Tim - we can only see the drop-point; the obvious external damage. If you stretch the point a long way & liken it to a head injury, then if the object [head] is moving at the time of impact, as opposed to being stationary & hit by a moving object, then the damage is actually done on the opposite side of the skull - it's called [contre coup](https://en.wikipedia.org/wiki/Coup_contrecoup_injury). By extrapolation, internal strut & support damage could be on the opposite side. Basically, the main damage could simply be invisible without proper inspection.

Dropped new Martin DCAE


How can I clear my local database using azure mobile services?

Question: I'm using Azure Mobile Services and I want to clear local database, how can I do that?
I have a problem with my local database. When I logout in app and login with other user, the data of the previous user is loaded for current user and I don't have idea why this occurs. I use debug on server side and the server return correct data, then I believe that the problem is the local Database.
Answer:
I'm using Azure Mobile Services and I want to clear local database, how can I do that?

For deleting your SQLite file, you could follow Deleting the backing store. Also, you could leverage the capability provided by IMobileServiceSyncTable to purge records under your offline cache, details you could follow Purging Records from the Offline Cache.

When I logout in app and login with other user, the data of the previous user is loaded for current user and I don't have idea why this occurs. I use debug on server side and the server return correct data

Since you did not provide details about your implementations (e.g. user log in/log out, user data management,etc), I would recommend you check whether your server/client side both enable per-user data store. You could use fiddler to capture the network traces when other user logging in, and make sure that the correctly user identifier (e.g. UserId) is returned, then check the query against your local database. Moreover, I would recommend you follow adrian hall's book about Data Projection and Queries.
Comment: Updated links: [Deleting the backing store](https://adrianhall.gitlab.io/develop-mobile-apps-with-csharp-and-azure/chapter3/client/#deleting-the-backing-store) and [Purging Records from the Offline Cache](https://adrianhall.gitlab.io/develop-mobile-apps-with-csharp-and-azure/chapter3/client/#purging-records-from-the-offline-cache)
Answer:
You can delete all of the local DB files by doing the following.
var dbFiles = Directory.GetFiles(MobileServiceClient.DefaultDatabasePath, "*.db");
foreach (var db in dbFiles)
{
    File.Delete(db);
}

However, this would delete all data each time and cause performance issues, as every time after you did this, you'd be getting a fresh copy of the data from your Azure DB, rather than using the cached copy in the device's SQLite DB.
We typically only use this for debugging, and the reason it's in a foreach is to capture all databases created (refer to my last suggestion)
There are a few other things you could try to get around your core issue of data cross-over.

There's another reason you might be seeing this behaviour. With your PullAsync, are you passing it a query ID? Your PullAsync line should look similar to this.
GetAllFoo(string userId)
{
    return await fooTable.PullAsync("allFoo"+userId,fooTable.Where(f=>f.userId == userId));
}

Note that the query ID will be unique each time (or at least, for each user). This is used primarilly by the offline sync portion of Azure, but in combination with the Where statement (be sure to import System.Linq), this should ensure only the correct data is brought back.
You can find more information about this here.

Also, some things you may want to consider, store a separate database for each userId. We're doing this for our app (With a company ID) - so that each database is separate. If you do this, and use the correct database on logging in, there's no chance of any data cross over.


Jetpack Compose theming at runtime from network api call

Question: I have an API call for theming that returns all colors for my application. I want to apply these colors all of jetpack components. can anyone please suggest a proper way to do this.
I have two files theme. kt and color. kt .
I need to change values in these classes dynamically.
Comment: App sample from Google with dynamic theming: https://github.com/android/compose-samples/tree/main/Jetcaster
Comment: if your themes are static and they never gonna change you can keep them in your app get the current theme from api.
Comment: @MohammadBaqer can you please suggest proper way ?
Answer: you should receive your colors from API then store that in dataStore then observe your data store in the main theme compose function whenever colors changed then create a new color pallet then set that in your MaterialTheme. automatically everywhere used MaterialTheme colors are updated.
Comment: could you please share sample project?


Drupal7: Private/Password Protected Section in a Drupal Website

Question: What would be the best way to do this In a Drupal7 Website:
I want to add to an already existing Public Drupal Website an 'Internal Section' that can be viewed only by Login (Password Protected).
This 'private/internal' Website will have 60+ info. pages, its own menu hierarchy, and any content that relates to it will also be private (private directory for images and files)
Thank you.
Boutros.
Answer: Drupal has only a very course access control structure for viewing content.  For the any user role, you can set permission to 'access content' (Drupal 6) or 'view published content' (Drupal 7), but that's about it.
Most inportant, out of the box, Drupal does not have any setting that let you set the permission "access content/view published content" for a particular content type.  It is all or nothing. I.e. if you untick "access content/view published content" for the anonymous user, no content on your site will be available to those that are not logged in.
Assuming that you're looking for more fine-grained access control, (where only a section of your website is private) you need to install a node access control module.  Many such modules exist. There is a page at the Drupal.org wiki giving an overview of node access modules.  You may start by reading that and try to find a suitable module.
To make a recommendation, I'll say that the one named Content Access seems to match your requirements.  It lets you control access per role and per content type.  I use this module to create premium or private content for logged in users on my sites.
To use it, install and enable it.  Create a new content type for the private pages - let's say you call it "Private page". Then (assuming Drupal 7) navigate to Administration » Structure » Content types and click "edit" for the "Private page". There is now a tab "Access control" that lets you control all types of access (Including "View") for this content type only.
As for menus, Drupal will usually manage this for you.  Any menu items that leads to content that is inaccessible, will not be visible for the roles that are not allowed to see it.


What's the full syntax of CKEDITOR.addTemplates()?

Question: I wasn't able to find any documentation on this one.
Can I somehow create multiple sets of templates in my template plugin (f.e first you choose a template, then you choose color scheme for the chosen template)?
Right now in /templates/templates/default.js I have something lile^
CKEDITOR.addTemplates("default",{imagesPath:CKEDITOR.getUrl(CKEDITOR.plugins.getPath("templates")+"templates/images/"),templates:[ /* list of my custom templates */ ]});

What does the first "default" mean?
Answer: It seems there's no documentation for the "addTemplates"-method for CKEditor 4, but there is for CKEditor 3. I'm not sure which version of CKEditor you're currently in?
Here's a snippet from the CKEditor 3 documentation:
// Register a template definition set named "default".
CKEDITOR.addTemplates( 'default',
{
    // The name of the subfolder that contains the preview images of the templates.
    imagesPath : CKEDITOR.getUrl( CKEDITOR.plugins.getPath( 'templates' ) + 'templates/images/' ),

    // Template definitions.
    templates :
        [
            {
                title: 'My Template 1',
                image: 'template1.gif',
                description: 'Description of My Template 1.',
                html:
                    '
Template 1
' +
                    '
Type your text here.
'
            },
            {
                title: 'My Template 2',
                html:
                    '
Template 2
' +
                    '
Type your text here.
'
            }
        ]
});

Full documentation:
https://docs.cksource.com/CKEditor_3.x/Developers_Guide/Templates
Comment: I currently use CKEditor 4.4.1. Thank you for the snippet.
It seems "default" is a set's name and I can add multiple sets in plugin.js
Reference: http://docs.cksource.com/ckeditor_api/symbols/CKEDITOR.config.html#.templates
Comment: @E.Sereda - very well, then I'm afraid I can't be of much help, as you're right about there not being any documentation on the addTemplates-method in CKEditor 4.X, but the method might be similar, so you should try out the snippet :-D
Comment: Hope so. I'll be right back after attempting to do this.


Are you able to define a named function within sorted()?

Question: I am trying to define a function in the 'key' parameter of a sorted() function.
I can get this to work by using a lambda function, however i was wondering if it was possible to split up my code, for readability (so i can spread it across multiple lines), to add a defined function in the key parameter.

sorted_contacts = sorted(contacts_array, key= def d(x):
        #...do something here
);

SyntaxError: invalid syntax
Sorted_contacts = sorted(contacts_array, key= def d(x):
                                                    ^
Answer: This is not possible in Python. Named functions are statements, and function arguments must be expressions.
The closest you'll get is a lambda function, but it can only have one line of code.
To solve your problem, you can define the function outside the call, and then reference it:
def d(x):
    ...
    return ...

sorted_contacts = sorted(contacts_array, key=d)

This is due to the fact that in Python, functions are first-class objects, which means that they are useable in an expression like any other object (integer, string, class instance, etc.)


PHP cannot assign object parameter values intermittently

Question: For some reason PHP cannot assign values to properties of my object intermittently! If I keep refreshing the page it works about 4 times out 5 but on the failed one all the values are not assigned I worked out that the object values are missing.
I have a function to collect a database row and assign it to an object. The query is not the issue the data is being pulled everytime. The issue is to do with this line
$obj->$key = str_replace("''","'",$row[$value[0]]);

I do a print out to make sure the value is there and then a print out of the object parameter directly after and I can see that all the property values are blank. Not that it is nothing to do with the value itself if I just set the value as a hard coded string the same thing happens
$obj->$key = 'a'

I don't have this problem with standard variables, the values are always there it just seems that object parameters values cannot be assigned 1 time out 5.
It doesn't seem to be a code related problem since it works 4 times out 5.
function getRow($class,$where = false,$order = false, $limit = false) {

    $obj = new $class();

    $sql = "select * from ".$obj->table." where 1 ";

    if ($where) {
        foreach ($where as $key => $value) {
            $opvalue = $this->getOperator($value);
            $op = $opvalue[0];
            $value = $opvalue[1];
            if ($obj->cols[$key][1] == 1)
            $sql .= "and ".$obj->cols[$key][0]." $op $value ";
            else if ($obj->cols[$key][1] == 2)
            $sql .= "and ".$obj->cols[$key][0]." $op '$value' ";
        }
    }
    if ($order)
    $sql .= "order by $order ";
    if ($limit)
    $sql .= "limit $limit ";

    $res = $this->runQuery($sql);

    //map values
    if ($row = mysql_fetch_assoc($res)) {

//            print_r($row);

        foreach ($obj->cols as $key => $value) {

            if ($obj->cols[$key][1] == 3) {

                $obj->$key = poststringToArray($row[$value[0]]);

            }
            else {

                $obj->$key = str_replace("''","'",$row[$value[0]]);

//                    echo "obj->$key  = {$row[$value[0]]}
";
//
//                    echo "obj->$key = ".$obj->$key.'
';

            }

        }
//            echo '
';
//            print_r($obj);
//            echo '
';

    }
    else {

        return false;

    }

    return $obj;
}

Forget about the complex code above here's a simple example where the property value will be blank missing intermittently, but the variable retains the value. If I keep refreshing this script the value of 'id' will be blank sometimes!
Class SalesOrderService {

var $id;

}
$test = 'a string';

echo 'the value of $test is \''.$test."'";

echo '

';

$so = new SalesOrderService();

$so->id = 'aaa';

print_r($so);

Comment: @Daedalus: He's setting the key dynamically. $object->$key = 'A'; would set the value of $object->id if $key = 'id', for example.
Comment: ok I found this in the logs "Attempt to assign property of non-object" meaning the object doesn't exist! I have PHP 5.3.3 and starting to find a few similar posts now
Comment: @ParhamDoustdar Didn't see it documented anywhere; but it does work, so, deleting my comment.
Comment: I might add that the same problem occurs for hardcoded keys not just dynamic ones so for example $obj->key would not keep the value sometimes also
Comment: Are you using StdClass?
Comment: nope I'm not, see the last example
Answer: The solution is a bug in 5.3 I upgraded to 5.4 and it works fine now


Content Type template override - Dash or Underscore?

Question: I am looking for some clarification on whether to use and underscore ("_") or a dash ("-") when creating a page template override for a content type. Some tutorials and articles say the format should use underscores such as ...
page--content_type_name.tpl.php
while others specify the format as dashes like ...
page--content-type-name.tpl.php
If my content type has a multi-word name how should the template override filename be formatted?
Comment: Both seem to work fine for me, have you tried them?
Comment: I have tried both and only the underscore works for me.
Comment: Hmm must be the theme I'm using or something but yeah, now you mention it the method with underscores has always worked
Answer: If you look at theme_get_suggestions(), there is a comment:
// When we discover templates in @see drupal_find_theme_templates,
// hyphens (-) are converted to underscores (_) before the theme hook
// is registered. We do this because the hyphens used for delimiters
// in hook suggestions cannot be used in the function names of the
// associated preprocess functions. Any page templates designed to be used
// on paths that contain a hyphen are also registered with these hyphens
// converted to underscores so here we must convert any hyphens in path
// arguments to underscores here before fetching theme hook suggestions
// to ensure the templates are appropriately recognized.

If you look at a preprocess, like template_preprocess_node(), you will see things like:
$variables['theme_hook_suggestions'][] = 'node__' . $node->type;

This is using the machine name of the content type directly, which is underscored.
-- is a delimiter between things, underscores are used within a thing.


Flipkart Affiliate - Can I track the status of an order made through my website?

Question: I am being an affiliate to flipkart and I am advertising flipkart products on my website.
My requirement is this:
If a user redirects to flipkart from my website and purchases a product, can we get the order/purchase id of that particular purchase and can we track it's status and show it in my website.
Comment: Is Flipkart exposing any API to you?
Comment: I didn't see any API's from the flipkart. But I wanted to be sure that this is not possible.
Comment: I'm voting to close this question as off-topic because it is asking for information on a product rather than looking for assistance with code.
Comment: @RajKumar See anything is possible and there WILL be a way. It would be better if you find some possible ways, try something, if you face issues then ask here.
Answer: If you are using flipkart product api you can track orders with 2 extra parameters.
if you are not using flipkart product api, just using their advertisement banners and links on your website then you can only see product details in the report section such as price and category of product.
Note: Flipkart doesn't return any order id.
Comment: "If you are using flipkart product api you can track orders with 2 extra parameters." Yes I am using flipkart product API, but I didn't see any parameters related to tracking.


Immaculate Reception Legality

Question: Was the touchdown at the end of the game between the Steelers and the Raiders in the 1972 AFC Divisional playoff game legally a touchdown by the NFL double touch rules or was the pass first touched by the Steelers' receiver?
Comment: @edmastermind29 the steelers, like the patriots were the darlings of their respective decades.
Comment: Well, the Steelers won, so yes in practice.  I've never seen a replay that was convincing one way or the other.  One of those deep mysteries of sports...
Comment: NO! I say no! It was, and still is, and always will be raiders against the world.
Comment: @skullpatrol like how the "tuck rule" wasn't a fumble?
Answer: Oh I know this one!  My first cousin, Fred Swearingen #21 was the referee.  The best answer is by the off-field official, Art McNally IN USA TODAY below.  It wasn't my cousin's best call.  But it was LEGAL!  Fred had called a conference to debate the "double-touch" rule. Steelers' QB Bradshaw threw a pass which caromed off Pittsburgh running back Frenchie Fuqua, then hit Oakland Raiders' defensive back Jack Tatum, then scooped up by Pittsburgh's Franco Harris.  LEGAL TOUCHDOWN!
http://www.usatoday.com/story/sports/nfl/2012/12/18/art-mcnally-recalls-immaculate-reception-40-years-later-denies-looking-at-the-replay/1778279/
Fred had ANOTHER controversial call also involving the Steelers and Lynn Swann in Super Bowl XIII.
Answer: The call on the field was that the ball rebounded from the Oakland defender, and was caught before it hit the ground by Franco Harris, making it a legal reception and a touchdown.
Those facts, and whether the entire city of Pittsburgh ought to be burned to the ground have been debated by Raiders fans ever since.

Immaculate Reception Legality


How to compare two ranking algorithms?

Question: I want to compare two ranking algorithms. In these algorithms, client specifies some conditions in his/her search. According to the client`s requirements, these algorithm should assign a score for each items in data base and retrieve items with highest scores.
I have read different topics related to my question in this site and searched the net. According to my searches, the most relevant article which explains about some metrics for comparing ranking algorithms, was this: Brian McFee and Gert R. G. Lanckriet, Metric Learning to Rank, ICML 2010 (https://bmcfee.github.io/papers/mlr.pdf).
I think prec@k, MAP, MRR, and NDCG, are good metrics to use, but I have a problem:
My algorithm sort results, so the first item in my result list is the best one with highest score, the second result have the second top score, and so on. I limit my search algorithm to for example find 5 best results.The results are the most top 5 items. So,precision will be 1. When I limit my search to find best result,It finds the best one. Again, precision will be 1.But the problem is that,it is unacceptable for people who see this result.
What can I do? How I can I compare these algorithms and show one is better than the other?
Answer: Discounted Cumulative Gain (DCG) is one of the most popular metrics used for evaluation of ranking by any search engine. It is a measure of ranking quality. In information retrieval, it is often used to measure the effectiveness of web search engine.
It is based on the following assumptions:

Highly relevant documents are more useful if appearing earlier in a search result.
Highly relevant documents are more useful than marginally relevant documents which are better than non-relevant documents.

The formula for DCG goes as follows:
$$DCG_p = \sum_{i=1}^p \frac {rel_i} {log_2 (i+1)} = rel_1 + \sum_{i=2}^p \frac {rel_i} {log_2 (i+1)} \tag{1}$$
Where:

i is the returned position of a document in the search result.
$rel_i$ is the graded relevance of the document
summation over p (number of results returned) hence, accumulated cumulative gain gives the performance metrics of the returned result.

DCG is derived from CG (Cumulative Gain), given by:
$$CG_p = \sum_{i=1}^p rel_i \tag{2}$$
From (2) it can be seen that $CG_p$ does not change for a change in the order of results. Thus to overcome this issue DCG was introduced. There is a different form of DCG, which is popular for placing a very high emphasis on retrieval of the documents. This version of DCG is given by:
$$DCG_p = \sum_{i=1}^p \frac {2^{rel_i} - 1} {log_2 (i+1)} \tag{3}$$
One obvious drawback of the DCG equation presented in (1) and (3) is that algorithms returning a different number of results cannot be compared effectively. This is because the higher the value of $p$ the higher the value of $DCG_p$ will be scaled to.
To overcome this issue, normalized DCG (nDCG) is proposed. It is given by,
$$nDCG_p = \frac {DCG_p} {IDCG_p}$$
where $IDCG_p$ is the Ideal $DCG_p$, given by,
$$IDCG_p = \sum_{i=1}^{|REL|} \frac {2^{rel_i} - 1} {log_2 (i+1)}$$
Where |REL| is the list of documents ordered by relevance in the corpus up to position p.
For a perfect ranking algorithm,
$$DCG_p = IDCG_p$$
Since the values of nDCG are scaled within the range [0,1], the cross-query comparison is possible using these metrics.
Drawbacks:
1. nDCG does not penalize the retrieval of bad documents in the result. This is fixable by adjusting the values of relevance attributed to documents.
2. nDCG does not penalize missing documents. This can be fixed by fixing the retrieval size and using the minimum score for the missing documents.
Refer this for seeing example calculations of nDCG.
Reference
Answer: Useful Resources:

http://www.cs.utexas.edu/~mooney/ir-course/slides/Evaluation.ppt
http://www.nii.ac.jp/TechReports/05-014E.pdf
http://www.stanford.edu/class/cs276/handouts/EvaluationNew-handout-6-per.pdf
http://hal.archives-ouvertes.fr/docs/00/72/67/60/PDF/07-busa-fekete.pdf
Learning to Rank for Information Retrieval (Tie-Yan Liu)

How to compare two ranking algorithms?


How do I set the RUST_TEST_TASKS environment variable?

Question: I am attempting to write tests for my Rust program. Normally, these tests are run in parallel but I want to run them sequentially. I looked around and I can set this environment variable RUST_TEST_TASKS=1, but I am not sure where to do that.
Answer: The environment variable is actually RUST_TEST_THREADS
Answer: I think what they mean is setting the environment variables in the shell the test runner is running in, such as:
RUST_TEST_TASKS=1 ./my-test-runner

or exporting it:
export RUST_TEST_TASKS=1
./my-test-runner

Comment: I don't believe this is the solution. I tried it and I can still see the output as a result of running the tests in parallel. I think it might have to do with https://doc.rust-lang.org/std/env/fn.set_var.html but I am not sure where to  call it. I am also using `cargo test` to run my tests.
Comment: I don't think that is actually different from setting it in the shell... Can you compose an example as a gist, maybe?


Trying to use ExecuteScalar , and get " Specified cast is not valid " error

Question: I'm trying to get product price by using product name.  Below is the function I am using.
public int GetProductPrice(string ProductName)
{
    cnn.Open();
    SqlCommand cmd = new SqlCommand("SELECT ProductPrice FROM Products WHERE ProductName ='" + ProductName + "'", cnn);
    int price = (int)cmd.ExecuteScalar();
    return price;
}

Now I keep getting this error Specified cast is not valid, and I don't know why. Can someone help me ?
Comment: http://en.wikipedia.org/wiki/SQL_injection. Read. Learn.
Comment: dont use ExecuteScalar() in this situation - what happens if you have several products with the same name ...
Comment: Yea i know it but every product have another name (complitly diffrent)
Answer: Firstly, you should use parameterized SQL instead of putting the parameter directly into the SQL. Also, you should use a using statement to close the command - and connection - when you're done. Oh, and create a new SqlConnection for each operation. So something like:
public int GetProductPrice(string productName)
{
    // Quite possibly extract the connection creation into a separate method
    // to call here.
    using (var conn = new SqlConnection(...))
    {
        conn.Open();
        using (var command = new SqlCommand(
            "SELECT ProductPrice FROM Products WHERE ProductName = @ProductName",
            conn))
        {
            command.AddParameter("@ProductName", SqlDbType.VarChar)
                   .Value = productName;
            object price = command.ExecuteScalar();
            // And you'd do the casting here
        }
    }
}

Next, we don't know the type of the ProductPrice field. It could be that you're getting a long returned, or perhaps it's decimal. The simplest way to find out is just to use:
object tmp = cmd.ExecuteScalar();

... and then look in the debugger. Also look at the type of the field in the database - that should really tell you what to expect. Have a look at the SqlDbType enumeration for mappings between the two.
Comment: @user2120874: You might get better marks if you show your teacher that you preventing hackers taking over the database in your project. If I were a teacher, I'd mark you down for having SQLi in your code, because when you graduate, SQLi is a HUGE business risk for whomever you end up working for.
Comment: @user2120874: But please take the rest of the answer into consideration too - otherwise you're open to SQL injection attacks and all kinds of problems.
Comment: @user2120874 - "this is a school project so they dont try to SQLI me" You'd be surprised. I would. :)
Comment: well i cant belive it , the field ProdcutPrice was Nvarchar  , change it to int and everything works great, thanks
Comment: well i will take it for my other projects , but this is a school project so they dont try to SQLI me .
Thanks for the help
Comment: @НадеждаТарашкевич: It absolutely does - but it's not a method call as per your comment. It's a property assignment, assigning the `Value` property in the just-added parameter.
Answer: The cast of the ExecuteScalar works if your SQL query is retrieving a count of rows and not the column data. The returned count from SQL is then being cast to an int in C#.
Comment: As it's currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).


I was adding a background image in my react js . But it is showing an error

Question: I have seen on the internet they were telling to add image in the public folder . So i added my image  into the public folder by creating a sub folder name img in which i place my image "mario.png" in it.
As i wanted to add that image as the  background image so i wrote this code in my index.css

body{
    margin:0;
    padding: 0;
    font-family: sans-serif;
    background: url(/img/mario.png);
    background-size: 100%;
    background-position: bottom;
    background-color: #95e8f3;
    min-height: 100%;
}

and still it was not working it was showing the error :
**
./src/index.css (./node_modules/css-loader/dist/cjs.js??ref--5-oneOf-4-1!./node_modules/postcss-loader/src??postcss!./src/index.css)
Error: Can't resolve '/img/mario.png' in 'D:\Users\PIRATES\Desktop\cool\src'
**

But when i created a asset folder in src then it was working but i want to know why it is not working when i am placing my image in the public folder.
Comment: url value should be in quotes url('/img/mario.png')
Comment: Check this answer: https://stackoverflow.com/questions/64852112/create-react-app-4-0-cannot-resolve-image-path-in-public-folder
Comment: @PunithMithra i have added the double quotes even it is not working.
Comment: Have you tried this https://stackoverflow.com/questions/57162841/use-images-in-css-files-with-reactjs?
Comment: @fast-reflexes Yes this is working . But everyone was telling that you should add the image in the public folder and the react automatically detects it is in public folder so this was not working  and i just want to know why ?
Comment: For me, it works to put it in the public folder. If you use it like `` does it work then?
Comment: @fast-reflexes yes it works for me.  If i am adding the img tag.  Thanks for the link of the above post.
Comment: So then it seems to be something special with this css processing (the postcss-loader). As I said it works for me even when used as a background in css (but I don't know if I use this postcss-loader) but I don't know why it doesn't work for you. Now at least you know that you can use the public folder for other purposes.
Answer: You are not using url function properly. docs
body{
    margin:0;
    padding: 0;
    font-family: sans-serif;
    background: url("/img/mario.png");
    background-size: 100%;
    background-position: bottom;
    background-color: #95e8f3;
    min-height: 100%;
}

Answer: You have to add the relative path inside the url function, not the absolute path from the root of the project.
Considering your screenshot, you have to write something like this:
background: url('../public/img/mario.png');


Exchange 2003 OWA Authentication Problem

Question: I've got a problem with OWA, and I'm pretty much at the end of my rope with this. OWA authenticates perfectly on the internal network. No problems at all. On the outside, you have to log in two or three times before it "sticks" at which point it won't ask you to authenticate again for hours.
The problem cropped up after an admin left and we were forced to change a bunch of passwords, so it's almost certainly a password/directory ownership issue somewhere. However, I've gone over the configuration and I can't find anything that's not running with a password unique to the local machine.
The setup is Exchange 2003, running behind an Apache proxy. Since the problem is external only, I've gone over the proxy extensively, and I can't find any problems. The whole setup has been running fine for ~4 years, so again, it's probably tied to the password change (which shouldn't have effected the proxy in any way).
I'm sure it's some stupid configuration setting that I'm missing, but I can't find it for the life of me. Anyone have any ideas?
@PQD
Here's mine:
SetOutputFilter proxy-html
RequestHeader unset Accept-Encoding
ProxyHTMLURLMap https://myserver.mydomain.com/exchange /exchange

ProxyPass /exchange https://myserver.mydomain.com/exchange
ProxyPassReverse /exchange https://myserver.mydomain.com/exchange

ProxyPass /exchweb https://myserver.mydomain.com/exchweb
ProxyPassReverse /exchweb https://myserver.mydomain.com/exchweb

ProxyPass /public https://myserver.mydomain.com/public
ProxyPassReverse /public https://myserver.mydomain.com/public

ProxyPass /OMA https://myserver.mydomain.com/OMA
ProxyPassReverse /OMA https://myserver.mydomain.com/OMA

ProxyPass /Microsoft-Server-ActiveSync https://myserver.mydomain.com/Microsoft-Server-ActiveSync
ProxyPassReverse /Microsoft-Server-ActiveSync https://myserver.mydomain.com/Microsoft-Server-ActiveSync

I'm pretty sure this isn't the problem. Nothing has changed on the proxy, and this configuration has worked for years.
Comment: Can you provide more information about the configuration?  Is this a single server configuration that is hosted in your "DMZ" behind the proxy server or is it a split configuration where the OWA server is in the DMZ and the Exchange server is in the internal LAN?   -Thanks
Comment: It's the second one. Proxy in the DMZ, Exchange on the internal LAN.
Comment: how about the hosts? give it a try. assuming your apache is accessed from the outside by https://some.name.com/ - change all myserver.mydomain.com to some.name.com and add some.name.com to /etc/hosts so it points internal ip...
Comment: External DNS is managed by corporate, and getting the change passed through would be a total nightmare. People CAN authenticate, it just takes 2 attempts for some reason.
Comment: well - you do not have to touch dns managed by the corp. i assume it points to your the machine with apache rev proxy; only think you need to change is to use the same hostname in your apache config and put mapping that hostname<->internal ip in /etc/hosts
Answer: setting apache2 reverse proxy for exchange 2003 owa was [ censored ]. but at the end it works fine for me.
the trick was to fool apache into thinking that in-house exchange server has same host-name as one under which people access the apache from the outside of company.
so in dns i have: owa.company.com pointing to public ip of apache and in /etc/hosts of machine with apache reverse proxy i have owa.company.com pointing to the internal ip of exchange:
10.11.12.13     owa.company.com

my apache also does ssl encryption - internet uses connect over https, while apache uses http to talk with exchange. in vhost configuration i have:
# you can skip it if internet clients use http
RequestHeader set Front-End-Https "On"

ProxyPass /exchange http://owa.company.com/exchange/
ProxyPassReverse /exchange http://owa.company.com/exchange/
ProxyPass /exchweb http://owa.company.com/exchweb/
ProxyPass /oma http://owa.company.com/oma
ProxyPassReverse /oma http://owa.company.com/oma
ProxyPassReverse /exchweb http://owa.company.com/exchweb/
ProxyPass /public http://owa.company.com/public/
ProxyPassReverse /public http://owa.company.com/public/
ProxyPass /iisadmpwd http://owa.company.com/iisadmpwd/
ProxyPassReverse /iisadmpwd http://owa.company.com/iisadmpwd/

obiously mod_proxy, mod_proxy_http are loaded.
and yes - i'm aware that this sounds like voodoo; it does to me, but i found it described somewhere, applied - and it worked. what can i say...
Answer: After about 10 days the issue just went away. Seems like something weird was propagating through the AD. I bounced it off the corporate exchange people, and they had nothing. I ran through the whole configuration and it was all normal.
Unfortunately I have no idea what eventually fixed it: it was certainly dependent on some screwy behaviour attached to the admin who left, but his credentials weren't directly attached to the exchange server or IIS, and I fixed so many peripheral issues in that time period that it could have been any number of things.
Highly unsatisfying non-solution. If has any AD experience that would explain a server that forced repeated re-authentication, I'd be interested in hearing about it, and I'll switch the accepted solution to anything convincing.


How does average treatment effect relate to averages of individual treatment effects?

Question: I have a balanced panel with two groups, A and B, and run a standard fixed effects regression. I want to derive how the $\beta$ coefficients of these two regressions are related:
\begin{eqnarray}
y_{it} &=& \beta x_{it} & + \alpha_i + \epsilon_{it} \\
y_{it} &=& \mathbb{I}(i=A) \beta_A x_{it} + \mathbb{I}(i=B) \beta_B x_{it} & +\alpha_i + \epsilon_{it}
\end{eqnarray}
In my application, the case arises that the estimate of the average treatment effect is very far from the unweighted averages of the interacted treatment effect estimates, i.e.
$\hat{\beta} \neq \frac{\hat{\beta}_A + \hat{\beta}_B}{2}$.
So I want to understand how OLS does the "weighting" of the two groups when computing $\hat{\beta}$
What I would ideally like to find is a formula that relates $\beta$, $\beta_i$ and the covariances of my $x_{it}$s.
Comment: Well, for a start, there is no $\beta$, only $\beta_A$ and $\beta_B$.  So what is this $\hat{\beta}$ supposed to be estimating?
Comment: Possible duplicate of [Intuition for recursive least squares](https://stats.stackexchange.com/questions/66950/intuition-for-recursive-least-squares)
Comment: @kjetilbhalvorsen thanks for the pointer, I would say my question is a special case of RLS. What would be my aim for this question is to answer this special case using just the elementary methods I am familiar with (i.e. basic variance/covariance operations) to get the insight what is going on
Comment: @Ben, thanks for the comment & edit. I clarified my point: in my original specification, $\hat{beta}$ was not supposed to depend on the group but I have reason to suspect this may not be true in the data I have.
Comment: see https://stats.stackexchange.com/questions/88464/recommendation-for-sequential-least-squares-programming-book
Comment: Possible duplicate of [Recommendation for sequential least squares programming book](https://stats.stackexchange.com/questions/88464/recommendation-for-sequential-least-squares-programming-book)
Answer: Let $\tilde{y}_{it}$ and $\tilde{x}_{it}$ denote the group-demeaned variables. Now we run three bivariate regressions:
\begin{eqnarray}
\tilde{y}_{it} &=& \beta \tilde{x}_{it} & + \epsilon_{it} \\
\tilde{y}_{it} &=& {\beta}_A \tilde{x}_{it} & + \epsilon_{it} & \text{   if   } i=A \\
\tilde{y}_{it} &=& {\beta}_B \tilde{x}_{it} & + \epsilon_{it} & \text{   if   } i=B \\
\end{eqnarray}
Now write the estimate of each coefficient as a function of expected values:
\begin{eqnarray}
\hat{\beta}_{A} = \frac{Cov(\tilde{x}_{At},\tilde{y}_{At})}{Var(\tilde{x}_{At})} =
\frac{E(\tilde{x}_{At} \times \tilde{y}_{At}) - E(\tilde{x}_{At})E(\tilde{y}_{At})}{ E(\tilde{x}_{At}^2) - E(\tilde{x}_{At})^2} =
\frac{E(\tilde{x}_{At} \times \tilde{y}_{At})}{ E(\tilde{x}_{At}^2)}
\end{eqnarray}
where the last equality follows by definition because the tilde variables have been de-meaned by group. Proceeding analogously for group and then $\beta$, I can write my overall coefficient as:
\begin{eqnarray}
\hat{\beta} =& \left(\frac{E(\tilde{x}_{At}^2)}{E(\tilde{x}_{At}^2) + E(\tilde{x}_{Bt}^2)}\right)\hat{\beta}_A + \left(\frac{E(\tilde{x}_{Bt}^2)}{E(\tilde{x}_{At}^2) + E(\tilde{x}_{Bt}^2)}\right)\hat{\beta}_B  \\
  = & \left(\frac{Var(\tilde{x}_{At})}{Var(\tilde{x}_{At}) + Var(\tilde{x}_{Bt})}\right)\hat{\beta}_A + \left(\frac{Var(\tilde{x}_{Bt})}{Var(\tilde{x}_{At}) + Var(\tilde{x}_{Bt})}\right)\hat{\beta}_B
\end{eqnarray}
Thus, the "average treatment effect" is a weighted average of the individual treatment effects, where the weight is proportional to the variance of the explanatory variable within the group.
Numerical Example
<code>require(dplyr); n = 10^6; set.seed(1)
sim.l <- data.frame(t = 1:n) %>% mutate(
  e.common = rnorm(n),
  x.A = 5 + e.common + rnorm(n),
  x.B = (e.common*4 + rnorm(n))/15,
  y.A = x.A + rnorm(n),
  y.B = x.A + rnorm(n)
)
# most of the variation in x.B is driven by common shock, but var x.B << var x.A
round(cov(sim.l[,-c(1:2)]), digits = 2)

sim.p <- sim.l %>% melt(id.var = 't') %>% mutate(
  var = str_extract(variable,'^(.)'),
  group = str_extract(variable,'(.)$')
) %>% dcast(t + group ~  var, value.var = 'value');
lm(y ~ x:group + group, data=sim.p)
lm(y ~ x + group, data=sim.p)
</code>
This reproduces the case where the Average treatment effect (shown in column 2) is almost exactly equal to the treatment effect of group A shown in column 1:
<code>===========================================
             Model 1         Model 2
-------------------------------------------
(Intercept)       -0.00           -0.46 ***
                  (0.00)          (0.00)
groupB             5.00 ***        5.46 ***
                  (0.00)          (0.00)
x:groupA           1.00 ***
                  (0.00)
x:groupB           3.53 ***
                  (0.00)
x                                  1.09 ***
                                  (0.00)
-------------------------------------------
R^2                0.49            0.41
Adj. R^2           0.49            0.41
Num. obs.    2000000         2000000
RMSE               1.24            1.33
===========================================
*** p < 0.001, ** p < 0.01, * p < 0.05
</code>
Based on the formula derived above, we can exactly reproduce these coefficients:
<code>      x.A    x.B    y.A    y.B
x.A 2.0032 0.2669 2.0034 2.0028
x.B 0.2669 0.0756 0.2670 0.2667
y.A 2.0034 0.2670 3.0050 2.0038
y.B 2.0028 0.2667 2.0038 3.0045

beta_A = 2.0028/2.0032
beta_B = 0.2667/0.0756
beta   = (0.0756*beta_b   + 2.0028*beta_a)/(2.0028 + 0.0756)
</code>


There is a phone number with $10$ digits. What's the probability that each digit occurs exactly once?

Question:
There is a phone number with $10$ digits. What's the probability that
  each digit occurs exactly once?

This is a task of an old exam and I'm curious if my solution is correct.
So we have a phone number made up of $10$ digits. Any number between $0$ and $9$ can occur as a digit. Thus in total we have $10^{10}$ combinations. But we want that each digit occurs exactly once. Since we have length $10$ with $10$ different numbers, there are $10!$ different possibilities of arranging those numbers (where each number occurs exactly once).
What we need to do is $10^{10}-10!$
Thus the probability that each digit occurs exactly once in a phone number of length $10$ is $$\frac{1}{10^{10}-10!}$$
?

By the way, it took me about $15$ minutes to get to that solution and in the exam I got like $5$ minutes for one task.. :p
I hope it's correct at least?
Answer: What you need is the possibilities for the combination of interest (here: all different digits), divided by total possibilities. You calculated the components correctly but not how to put them into a probability. So
$$\frac{10!}{10^{10}} = \frac{9!}{10^{9}} \approx 0.000363$$
which is significantly larger than your result.


Storing useful data from SQL Server stored procedure

Question: I'm very new to c# and trying to muddle my way around things. I'm used to coding on an iSeries (RPG) so c# is a very different concept to me.
I have a stored procedure which returns a row (or rows) of data depending on the matching rows.
Imagine my table users holds the following columns:
UserId, CompanyId, FirstName, LastName, CompanyAdmin, AccountStatus, UserName.

I have a stored procedure which returns all of these columns where there is a matching Company ID (ie.. Select * from users where CompanyId = 1).
I have created a class with matching fields as above and the appropriate get/set methods (ie. public int UserId {get; set;} etc).
I've used a SqlDataReader and am attempting to load each field from my class with data from the reader, but I'm getting an error

'reader["UserName"]' threw an exception of type
'System.InvalidOperationException' object
{System.InvalidOperationException}

What am I missing? Ultimately I would like an object that has all of my fields defined that will match my result set from my SQL Server stored procedure.
My code throwing the error looks like this..
    public partial class Admin : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            DataSet ds = new DataSet("ot_Users");
            using (SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["ApplicationServices"].ConnectionString))
            using (SqlCommand command = conn.CreateCommand())
            {
                command.CommandText = "dbo.ot_User_GetByUserName";
                command.CommandType = CommandType.StoredProcedure;
                command.Parameters.AddWithValue("@UserName", Environment.UserName);

                conn.Open();

                SqlDataReader reader = command.ExecuteReader();

                ot_Users user = new ot_Users();

                while(reader.Read())
                {
                     user.CompanyId = (int) reader["CompanyId"];
                     user.UserId = (int) reader["UserId"];
                     user.FirstName = (String) reader["FirstName"];
                     user.LastName = (String) reader["LastName"];
                     user.CompanyAdmin = (bool) reader["CompanyAdmin"];
                     user.AccountStatus = (int) reader["AccountStatus"];
                     user.UserName = (String) reader["UserName"];
                }

                reader.NextResult();

                CompanyName.Text = user.FirstName.ToString();
            }
        }
    }

Comment: Can we see the selected coulumns from your stored procedure? Also always try to minimize posibilities of exception! e.g. try `reader["FirstName"] != DBNull.Value ? reader["FirstName"].ToString() : string.Empty;`
Comment: firstname, lastName and Username. are they (char) isnt that should be string?
Comment: I've changed them now, but still get the same problem
Comment: try string with lower case 's'.
Comment: i was about to write that.. good catch @humpty dumpty
Comment: ok I've changed from "String" to "string" and get a different error now..

I'm getting a nullreference exception on line

CompanyName.Text = user.FirstName.ToString();

huMpty duMpty - I've tried the line you suggested but my editor doesn't like it (only assignment, call, increment, decrement, and new object expressions can be used as a statement).
Comment: As for my stored procedure (you guys will hate this I'm sure)

    SELECT *
    FROM dbo.OTUser
    WHERE CompanyId = @CompanyId;
Comment: Have a debug point on that line and check `?reader["UserName"].GetType()` on immediate window. You will see what type is coming and you need to cast it to the same type and also `user.UserName` should be of that type as well.
Comment: `while(reader.Read())` will read until EOF then after that you are trying to advance to the next record `reader.NextResult();` are you sure this isnt causing a problem? You are also missing braces for the first `using`
Comment: I've commented out the reader.NextResult() and still get the same issue, it looks like my SP is not returning any rows, although I've checked and it certainly should do, let me do some testing on the stored proc and see if that is my culprit.

thanks everyone for your responses so far, much appreciated.
Comment: Just need to check that there are records in the DB that match the username of the user logged into the machine
Comment: Found it!

My parameter on the SP was defined incorrectly, I'd defined it as varchar and not varchar(50) so I presume this defaulted to 1 character.

I'm still getting errors now my SP is returning data, but this is down to casting problems which I can sort.

Thanks everyone for your tips and also the help in tidying up my code.
Answer: I think the problem could be with casting UserName as a char. A char represents a single character only, a string represents multiple characters, so you might want to use that one instead!
That or perhaps the UserName isn't cased properly?
Comment: ``char`` is probably wrong but the cast can't be the source of the error he reported because it happens after the reader fetches the data.
Comment: Ok - I've changed that and I'm still getting the same error, I've double checked my ot_User.cs and the fields in there are also defined as String (they are now anyway).
Comment: Could you post the actual schema of your data? What DB/Server are you using?
Answer: try this.
Please note : that if your sproc returns more than one row it will take the last one only. because you are looping through an object only. if you are expecting more result than you should use List or somekind a collection object.
 public partial class Admin : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            DataSet ds = new DataSet("ot_Users");
            using (SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["ApplicationServices"].ConnectionString))
            {
                using (SqlCommand command = conn.CreateCommand())
                {
                command.CommandText = "dbo.ot_User_GetByUserName";
                command.CommandType = CommandType.StoredProcedure;
                command.Parameters.AddWithValue("@UserName", Environment.UserName);

                conn.Open();

                SqlDataReader reader = command.ExecuteReader();

                ot_Users user = new ot_Users();

                while(reader.Read())
                {
                     user.CompanyId = reader["CompanyId"] != DBNull.Value ? (int)reader["CompanyId"] : 0 ;
                     user.UserId = reader["UserId"] != DBNull.Value ? (int)reader["UserId"] : 0 ;
                     user.FirstName = reader["FirstName"] != DBNull.Value ? reader["FirstName"].ToString() : string.Empty;
                     user.LastName = reader["LastName"] != DBNull.Value ? reader["LastName"].ToString() : string.Empty;
                     user.CompanyAdmin = reader["CompanyAdmin"] != DBNull.Value ? (bool)reader["CompanyAdmin"] : false ;
                     user.AccountStatus = reader["AccountStatus"] != DBNull.Value ? (int)reader["AccountStatus"] : 0 ;
                     user.UserName = reader["UserName"] != DBNull.Value ? reader["UserName"].ToString() : string.Empty;
                }

                if (user != null)
                {
                    CompanyName.Text = user.FirstName.ToString();
                }
            }
        }
    }
}


Testing equality of coefficients from regression models with long format dataframes

Question: How can I test the equality of coefficients from multiple regression models?
I want to compare multiple measurement methods to see if they capture the dynamic processes of the same construct in similar ways; the hypothesis I intend to test is that the regression coefficients across models are equal because all measurement methods are equally sensitive to changes in the construct studied.
First, the linear increase of the three methods as the number of interaction increases was modelled. I used mixed effect regression models for each method with a long format data frame, as each method was assessed multiple times per participant.
However, the methods weren't assessed the same amount of time, so the data frames used for each regression model aren't of the same length. TS1 and TS2 were assessed before each interaction (12 in total), TS3 was only assessed 2 times: before the first and last interaction with the system
To then compare the coefficients, I thought about using seemingly unrelated regression (SUR) to fit the regression models together. This would allow for a comparison of the coefficients. However, I am unsure if this method is the most appropriate for the data used. To use seemingly unrelated regression, the regression models have to be based on the same data set. So the three long format datasets used for each regression model need to be combined in one data set.
Is there a more appropriate method / statistical model to test the equality of coefficient? Or is seemingly unrelated regression the best method to compare the equality of multiple regression coefficients?
If anyone has an idea, I would be truly grateful!
These are examples of the mixed effect models I used. The long-format data frames are all based on the same original dataset:
lmer(ts1 ~ interaction_num + (1|subject_id), dfts1_long)
lmer(ts2 ~ interaction_num + (1|subject_id), dfts2_long)
lmer(ts3 ~ interaction_num + (1|subject_id), dfts3_long)
Comment: Is time a real time measurement, and if so, are the three methods measured on the same absolute time scale? E.g. is, fo0r a given person, ts1 measured at t=0, t=2, t=5; ts2 measured after ts1, namely at t=6, t=10, t=11 and t=20? And ts3 after ts2? Or do they all three start at t=0? Here I assume that t=6 means "measured at day 6" or in some other time unit. I was thinking about a extended lmer model, using data  from all three methods in one single model, but need more info about the time variable.
Comment: So, for a given person, ts1 is measured a t=0, t=1, t=2, t=3, t=4 ... t=11. The same applied for ts2 as they were measured at the same time. However, for ts3 the measurement of "time" were t=0 and t=11. So 12 measures for ts1 and ts2, but only 2 for ts3. The variable "time" is generated by the transformation of the dataset from wide to long. In the original dataset, the variable names were ts1_0 for the measure of ts1 at t=0, ts1_1 for t=1 ... If that helps clarify the structure
Comment: So, you measured ts1 etc. at real time points. What was the unit in which time was measured? Hours, days, months? And you want to model a linear increase (or decrease) in ts1 as time proceeds from, say, day 0 to 11? Do I interpret this correct?
Comment: The time variable describes before which interaction with a system the construct was measured. So ts1 at t=0 is the assessment before the first interaction; t=1 is before the second interaction, ... It is not a real measure of time in a unit like seconds or hours. Sorry for the confusion. And yes exactly, I would like to model the linear increase in ts1 /ts2/ts3 as time (number of interactions) increases.
Comment: I do not believe this question to be about programming, debugging. Also, it certainly is not about a routine operation to be performed. It deals with finding the right statistical model for relatively complex data. With a multilevel model using e.g. lmer in R, it must be possible to formulate a suitable model. I would li8ke to help the OP with doing this. So please could you open the question again?
Comment: To have your question reopened, you should reformulate it. First, better explain how you coded "time", in the way you explained it in your final comment above (maybe call it "repetition number" instead?) Second, emphasise that you are searching for a statistical model to answer the question about differences in the effect of "time" (or repetition) for the three measurement methods. The impression may have been created (reading back your and mine comments) that your question deals with how to best code time.
Comment: Thank you very much @BenP for your interest in answering this question!
Comment: I'm a partly retired teacher in statistics and consider it as a hobby. @kleinemaus, are you going to reformulate your question so that it might be reopened? Think that would be nice, I could then post a formal answer and explain the model I have in mind. If it will not be reopened, maybe we could plan a teams-session, but I would prefer explaining things here so that others may benefit too.
Answer: You could combine the three datasets, to make one (very) long dataset. For a given  subject_id you could then have e.g. 26 records or observations, 12 for method 1, 12 for method 2, and 2 for method 3. You should make a factor variable, "method", valued 1, 2 or 3, depending on the method to which each record (row) in your data frame belongs. So for this subject the factor "method" would be 12 times 1, 12 times 2, and 2 times 3. Your dependent variable is "ts" and contains the 26 values for the 3 methods. Finally, variable "interaction_num" takes values 1 to 12. Instead of the three models you proposed in your post, you could use one single model now:
model1 <- lmer(ts ~ method*interaction_num + (0+method||subject_id)
This would give you (almost) the same results as running the separate models you proposed. In the fixed effects results you will see the regression coefficient for interaction_num, which holds for method 1 (the reference of the factor method). The term method2:interaction_num shows you how much for method 2 the effect of interaction_num deviates from the effect for method 1. This is probably what you are most interested in. Same for method3:interaction_num. The effect of method2, shows you the ts difference between method2 vs. method1 for interaction_num being 0. This is not meaningful, but if you would code interaction_num from 0-11 instead of 1-12 then it would have a meaningful interpretation: difference between method 2 and 1 for the first interaction.
There will be three different random intercept variances, one for each method. The "||" in (0+method||subject_id) means that these three random intercepts are uncorrelated. This is only done above to let the results be more similar to the ones you would have if you would run your separate models. However, you probably will get a better model fit (deviance) if you would use (0+method|subject_id) so that correlations between the three random intercepts are allowed and estimated.
This model1 will estimate only one single residual variance! This is different from the three models you proposed, because these will give you three different residual variances. So, model1 comes with the assumption that the within-subject variance for each method is the same. And hence, the results of model1 will almost (but not exactly) be the same as those of your separate models, if indeed these residual variances are close. You can (afaik) with lme from package nlme specify that you also want to have three different residual variances.
The model1 assumes, for a given method, the same total variance in your data at each interaction number and also the same covariance (or correlation) between a pair of interaction numbers. This is called a compound symmetry covariance structure. This may not be plausible for your data. If you would let the interaction_number have a random effect for each person/method combination, you would have a more flexible covariance structure. That would be possible for method1 and method2, but not for method3, because you only have two observations there.


Do these matrices have any name?

Question: Assume $A$ is a square matrix defined as follow: $$A=\sum_{i} u_{i}u_{i}^T$$ where for each $i$, $u_i$ is a non-negative column vector.
Do the matrices of these forms have any special name?
Comment: If you drop the condition that $u$ is non-negative, the name is "symmetric positive-definite". The extra condition is certainly important. Maybe the answer is "symmetric positive-definite non-negative"? Do you have an example that disproves this guess?
Comment: So $A$ is $U\cdot U^T$ where $U$ is a nonnegative square matrix.
Comment: No it's not. Note that in the definition I have $u_{i}u_{i}^T$ not $u_{i}u_{j}^T$.
Comment: In general, if $a_i$ are the column vectors of $A$ and $b_i$ are the column vectors of $B$, then $A\cdot B^T = \sum a_i \cdot b_i^T$
Comment: Not entirely relevant, but a matrix of the form $u v^T$ is called a dyad.
Comment: I'm pretty sure apt1002 is mostly correct, and $A$ must be positive semi-definite (positive definite is too strong: in addition to the zero matrix, you can construct nonzero, non positive-definite matrices of the type you describe). Your $A$ has all nonnegative entries.  So the question is, does every positive semidefinite matrix with nonnegative entries have the form you describe?
Answer: Yes, these are the completely positive matrices.
Answer: There is a problem with the term 'non-negative column vector'. The notion of non-negative is defined for square matrices.
We can require $u_i$'s such that $M=[u_{ij}]$ is positive (non-negative) that is such that $M=UU^*$ for some square matrix $U$.
There is also a different notion of positivity (again for square matrices) which requires the entries to be non-negative.
One vaguely related idea is the following. If $M$ is a symmetric matrix that is orthogonally diagonalised by
$P=[u_1\space{} u_2\space{}\dots u_n]$
and if $\lambda_i$'s are eigenvectors of $M$ corresponding to the unit eigenvectors $u_i$'s, then
$M=\lambda_1 u_1u_1^T+\dots +\lambda_nu_nu_n^T$
is called the spectral decomposition of $M$.
Comment: I'm assuming the components of the column vectors are nonnegative.  Perhaps the OP should clarify.
Comment: I don't know what do you mean by "$A$, as it is written, denotes a single positive number". I mentioned that each $u_i$ is a column vector so $A$ is a matrix.
Comment: Fixed that. Will look it up.
Answer: Let's assumes there are $n$ such $u_i$'s and there are exactly $k$ linearly independent vectors among them, then matrix $A$ would be a rank-k matrix.
Each of the $u_iu_i^t$ is a rank-1 matrix and they would sum up to a rank-k matrix if exactly k out of them are linearly independent.
 As $x^t(u_iu_i^t)x=(x^tu_i)(x^tu_i)^t \gt0$ , each of the $u_iu_i^t$ is a symmetric positive definite matrix, thus A is also a symmetric positive definite matrix.

Do these matrices have any name?


Prove $P(A \cup B) \leq P(A) + P(B)$

Question: The axioms of probability are:$$\text{(i) }0\leq P(A) \leq1 \text{ for each event $A$}\\ \text{(ii) }P(\Omega)=1\text{ and }P(\varnothing)=0\\
\text{(iii) If $A$ and $B$ are disjoint, then }P(A\cup B)=P(A)+P(B)
\\
\text{(iv) If }A_1, A_2, A_3,... \text{is a sequence of pairwise disjoint events then}\\
P\left( \bigcup_{i=1}^{\infty}A_i\right) = \sum_{i=1}^{\infty} P(A_i)$$
I'm supposed to use the fact that $$A \cup B =(A \cap B^c)+(A\cap B)+(A^c\cap B).$$
Thus, we have$$P(A \cup B) =P(A \cap B^c)+P(A\cap B)+P(A^c\cap B)$$
Not too sure where to go from here.
Comment: How about you add another $P(A \cap B)$ to the right hand side of your last equation
Comment: You can write the set $A$ itself as a disjoint union using the set $B$. $A = ...$
Answer: I would start with noticing that
\begin{align*}
\mathbb{P}(A) = \mathbb{P}(A\cap B) + \mathbb{P}(A\cap B^{c}) \Rightarrow \mathbb{P}(A\cap B^{c}) = \mathbb{P}(A) - \mathbb{P}(A\cap B)
\end{align*}
Similarly, we also have that
\begin{align*}
\mathbb{P}(B) = \mathbb{P}(B\cap A) + \mathbb{P}(B\cap A^{c}) \Rightarrow \mathbb{P}(B\cap A^{c}) = \mathbb{P}(B) - \mathbb{P}(B\cap A)
\end{align*}
Finally, we arrive at the desired inequality:
\begin{align*}
\mathbb{P}(A\cup B) & = \mathbb{P}(A\cap B^{c}) + \mathbb{P}(A\cap B) + \mathbb{P}(A^{c}\cap B)\\\\
& = \mathbb{P}(A) + \mathbb{P}(B) - \mathbb{P}(A\cap B) \leq \mathbb{P}(A) + \mathbb{P}(B)
\end{align*}
and we are done.
Hopefully this helps!
Answer: $A\cup B=A\cup (B\cap A^c)$ and since $A$ and $B\cap A^c$ are disjoint,  $P(A\cup B)=P(A)+P(B\cap A^c)$
Also,
$A\cup B=B\cup (A\cap B^c)$ and since $B$ and $A\cap B^c$ are disjoint, $P(A\cup B)=P(B)+P(A\cap B^c)$
Adding these gives $$2P(A\cup B)=P(A)+P(B)+P(A\cap B^c)+P(B\cap A^c)$$
Since $(A\cap B^c)\subset A$, $$P(A\cap B^c)\le P(A)$$
Likewise, $$P(B\cap A^c)\le P(B)$$
Therefore, $$2P(A\cup B)\le 2P(A)+2P(B)\implies P(A\cup B)\le P(A)+P(B)$$

Prove $P(A \cup B) \leq P(A) + P(B)$


elementary question concerning definition of sifted colimit

Question: I am reading a proof (in Algebraic Theories by Adamek et al, Theorem 2.15) for the fact that sifted categories $\mathcal{D}$ are precisely those for which the diagonal functor $\Delta : \mathcal{D} \to \mathcal{D} \times \mathcal{D}$ is final. By definition, $\mathcal{D}$ is sifted iff for every finite discrete category $I$ and every diagram
$$D : \mathcal{D} \times I \to Set$$
the canonical morphism
$$ colim_\mathcal{D}\big(\prod_ID(d,i)\big) \to \prod_I(colim_\mathcal{D} D(d,i)$$
is an isomorphism. Now according to the proof, in the case that $I$ is the two-element set this

...means that for every pair $D, D':\mathcal{D} \to Set$ of diagrams, the colimit of the diagram
  $$ D \times D' : \mathcal{D} \times \mathcal{D} \to Set, \qquad (d,d') \mapsto Dd \times D'd'$$
  is canonically isomorphic to $colimD \times colimD'$.

[This passage is taken from Remark 2.2.2 of the cited book.]
I interpret the left hand side in the above definition as follows: consider the diagram $D_1 : \mathcal{D} \to [I, Set]$ obtained from $D$ by adjunction, then take the (tip of the) product of the diagram $D_1(d)$ for each $d \in \mathcal{D}$; this gives a functor $\mathcal{D} \to Set $ of which we can now take the colimit. Since products can be formed componentwise in functor categories, in the case $I = 2$ we get the diagram
$$ \mathcal{D} \to Set ,\qquad  d \mapsto Dd \times D'd$$
Am I missing something here or is this really a mistake? The latter seems very improbable to me as the idea of the whole proof would (as far as I can see) not work in that case.
Answer: I think you are connecting two statements that aren't (immediately) connected.  The part in your block quote is simply true in $\mathbf{Set}$ for all (small enough) categories $\mathcal{D}$, not just sifted ones.  It's a simple corollary of the fact that $-\times S$ is a left adjoint (and thus cocontinuous) for any set $S$.
Explicitly, using coend notation for the colimits (though they are just colimits), we have, going right to left from the statement:
$$\left(\int^{c\in\mathcal{C}}F(c)\right)\times\left(\int^{d\in\mathcal{D}}G(d)\right) \cong \int^{c\in\mathcal{C}}\int^{d\in\mathcal{D}}F(c)\times G(d) \cong \int^{(c,d)\in\mathcal{C}\times\mathcal{D}}F(c)\times G(d)$$
Then the question is, when $\mathcal{C}=\mathcal{D}$: "does 'pulling back' along $\Delta : \mathcal{D}\to\mathcal{D}\times\mathcal{D}$ preserve this colimit?"  This is exactly the question: "is $\Delta$ final?"  If $\mathcal{D}$ is sifted, then the answer is "yes" because the far left term is isomorphic to $\int^{d\in\mathcal{D}}F(d)\times G(d)$ by siftedness.  If $\Delta$ is final, we get $\mathcal{D}$ is sifted by recursively applying the above isomorphism and finality of $\Delta$.
Comment: Thanks. Now I understand how the proof is meant. The way they formulate it in the book is not quite correct though, or at least very misleading.
Comment: Why do you bring up coends? You could just as well use the notation $\varinjlim_\mathcal{C} \varinjlim_\mathcal{D}$ and $\varinjlim_{\mathcal{C} \times \mathcal{D}}$.
Comment: @ZhenLin I just used the notation because I prefer it (though it does have some issues.)  (I also prefer the *concept* and calculational fluidity of (co)ends to (conical) (co)limits, but that's a different story, and I agree doesn't make a big impact here.)
Comment: One benefit of coend notation in this case, though, is it makes cocontinuity of $\times$ look like a form of bilinearity which is a useful guide.  The notation is suggestive in other ways (sometimes misleadingly, but not in this case.)