Anotação @Version

Question: Usando a Anotação @Version, mostra este erro ao tentar fazer o update.

18:25:16.650 [http-nio-8080-exec-84] ERROR
  br.com.netsoft.controller.todos.AtualizacaoMonetariaItemController -
  Row was updated or deleted by another transaction (or unsaved-value
  mapping was incorrect) :
  [br.com.netsoft.model.todos.AtualizacaoMonetariaItemEntity#8]

Quando insere está funcionando.
Está mapeado assim
<code>    @Version
    @Column(name = "NR_VERSAO", nullable = false)
    public int getNrVersao() {
        return nrVersao;
    }
</code>
Metodo de alterar
<code>@Transactional
@Repository
public class BaseRepositorio<T> {

    protected Logger logger = LoggerFactory.getLogger(this.getClass());

    @PersistenceContext
    protected EntityManager entityManager;

    public EntityManager getEntityManager() {
        return entityManager;
    }

    public void setEntityManager(EntityManager entityManager) {
        this.entityManager = entityManager;
    }

    public T inserir(T objeto) throws Exception {
        this.entityManager.persist(objeto);
        return objeto;
    }

    public T alterar(T objeto) throws Exception {
        return merge(objeto);
    }

    private T merge(T objeto) {
        this.entityManager.detach(objeto);
        this.entityManager.merge(objeto);
        return objeto;
    }
}
</code>
Comment: O problema parece ser antes disto. Quem chama este método merge? mas se fosse você eu utilizaria o comportamento original do `merge` (sem usar `detach`).
Answer: @guilherme, ajuste o seu método de merge:
<code>private T merge(T objeto) {
    this.entityManager.detach(objeto);
    return this.entityManager.merge(objeto);
}
</code>
O método entityManager.merge(T entity) retorna o objeto resultante do merge e com o version incrementado. No seu método T merge(T objeto) você está retornando o mesmo objeto que recebeu como parâmetro, portanto, com o valor de version inalterado. Ao tentar salvar uma segunda vez, o version está defazado em relação ao banco e, por isso, a Hibernate dispara a exception de "Row was updated or deleted by another transaction".
O comportamento do método entityManager.persist(T entity) já é diferente, pois atualiza o próprio objeto que recebe como parâmetro.
Comment: Obrigado. Imaginei isto. Mas como fazer o correto ? Quebrei a cabeça e não consegui avançar.
Comment: Badaro, não adiantou, mesmo erro.
Comment: @Guilherme, coloque um breakpoint no método alterar para verificar se nrVersao está chegando com o valor correto. Se não estiver. algum ponto da sua rotina está alterando o mesmo ou utilizando uma cópia defazada da entidade.
Comment: Então tenho que verificar o valor antes de alterar. Vou verificar.
Comment: Não funcionou....

Anotação @Version


Computing percentile rank in R

Question: How can I add new variable into data frame which will be percentile rank of one of the variables? I can do this in Excel easily, but I really want to do that in R.
Thanks
Answer: Given a vector of raw data values, a simple function might look like
perc.rank <- function(x, xo)  length(x[x <= xo])/length(x)*100

where x0 is the value for which we want the percentile rank, given the vector x, as suggested on R-bloggers.
However, it might easily be vectorized as
perc.rank <- function(x) trunc(rank(x))/length(x)

which has the advantage of not having to pass each value. So, here is an example of use:
my.df <- data.frame(x=rnorm(200))
my.df <- within(my.df, xr <- perc.rank(x))

Comment: 1. Your function does not mimic Excel's `percentrank`-function, which is good (+1) since the latter gives "strange" results (see my [comparison](https://gist.github.com/1026879)). 2. I wouldn't name the data frame `df`, because `df` is an R function (the density of the F distribution, see `?df`).
Comment: @Bernd Thanks. (1) There are some built-in functions for computing PR in various psychometrics packages. I think I grabbed this one from the `CTT` package a while ago. I didn't check against Excel because I don't have/use it. About (2) I seem to always forget about this! Let's go with `my.*` (Perl way) :-)
Comment: @Tyler Nope. In case there are ties, `rank()` defaults to taking the average of the tied values (cf. `ties.method = c("average",...)`).
Comment: Beware that NA values should be removed! This can be done by adding `x = x[!is.na(x)]`
Comment: @chl why is the `trunc` required?  It seems rank will always return an integer anyway.
Answer: If your original data.frame is called dfr and the variable of interest is called myvar, you can use dfr$myrank<-rank(dfr$myvar) for normal ranks, or dfr$myrank<-rank(dfr$myvar)/length(myvar) for percentile ranks.
Oh well. If you really want it the Excel way (may not be the simplest solution, but I had some fun using new (to me) functions and avoiding loops):
percentilerank<-function(x){
  rx<-rle(sort(x))
  smaller<-cumsum(c(0, rx$lengths))[seq(length(rx$lengths))]
  larger<-rev(cumsum(c(0, rev(rx$lengths))))[-1]
  rxpr<-smaller/(smaller+larger)
  rxpr[match(x, rx$values)]
}

so now you can use dfr$myrank<-percentilerank(dfr$myvar)
HTH.
Comment: 1 - (rank/size) gives you same as excel percentilerank
Comment: I got this from [office.microsoft.com](http://office.microsoft.com/en-us/excel-help/percentrank-HP005209212.aspx)
Comment: An anonymous (attempted) editor tried to add the following comment: "Nice function but sometimes, unfortunately, the RLE may return vector of `length < length(dfr$myvar)`".
Comment: Can you explain or link to the theory of this method?
Answer: A problem with the presented answer is that it will not work properly, when you have NAs.
In this case, another possibility (inspired by the function from chl♦) is:
perc.rank <- function(x) trunc(rank(x,na.last = NA))/sum(!is.na(x))
quant <- function (x, p.ile) {
      x[which.min(x = abs(perc.rank(x-(p.ile/100))))]
}

Here, x is the vector of values, and p.ile is the percentile by rank. 2.5 percentile by rank of (arbitrary) coef.mat may be calculated by:
quant(coef.mat[,3], 2.5)
[1] 0.00025

or as a single function:
quant <- function (x, p.ile) {
   perc.rank <- trunc(rank(x,na.last = NA))/sum(!is.na(x))
   x = na.omit(x)
   x[which.min(x = abs(perc.rank(x-(p.ile/100))))]
}

Computing percentile rank in R


Logistic Regression - Finding out information about a particular $X_i$ when all other Xs and parameters are known

Question: Suppose someone made the following logistic regression:
$Logit(p)$ = $\beta_0 + \beta_1X_1 + \beta_2X_2$
Now, someone else is trying to replicate the model creation, but by mistake the $X_2$ column is replaced by some other values. Hence, the model is coming as:
$Logit(p)$ = $\beta_0^{'} + \beta_1^{'}X_1 + \beta_2^{'}X_2^{'}$
I know all the values of the Xs and the parameters in the two equations except the values of $X_2$
My question is: How can I get back the values of $X_2$? I know that getting the value of $X_2$ for each row may be a challenge. Can I get something at an aggregated level at least (mean, sum, etc.)?
Comment: Do you have any distributional assumptions for $X_2$ that you are willing to make? For example normal, log normal?
Comment: $X_2$ is simply a column in a real life data. Assuming a distribution of it will not be practical I guess
Comment: The fact that $X_2$ is a column in a real life data seems to go against you not knowing $X_2$?
Comment: No no. My issue is the actual column $X_2$ got replaced with something else by mistake. Now I am trying to get some info about the original $X_2$ column
Comment: Do you know what the variable $X_2$ measures ... blood pressure, income etc. ? And do you know the population of the original data set including $X_2$?
Comment: It is actually a proportion variable that has only 3 distinct values (0.31, 0.49 and 0.84) across the entire data. And yes, the population for both the models are the same, hence that info is available.
Answer: I would use model fitting to assess the likelihood of x2 being a specific value within a given range:
Let's say that there are 3 samples (i=3)
Every sample is 0.31, 0.49 or 0.84
Distribution of x = average of all the unique x combinations ([0.31, 0.31, 0.31], [0.31, 0.31, 0.49] etc until max of [0.84, 0.84, 0.84])
For each possible value of x, compare the output of the regression to the original regression function ()  and calculate the loglikelihood (which is basically asking what are the odds that that's the right value of the lost x)
Assumptions:
I'm assuming you have the output of the original () calculation And that the data of x was not replaced by random noise and thus '2 is similar to 2.


Lower guitar tuning

Question: I know that there are devices to quickly lower a guitar's E string into drop tuning. Are there also devices or effects pedals to lower the entire guitar? There's a lot of rock music in E-flat tuning (or lower), and it's inconvenient to retune the whole guitar down a half step or more.
Is there a convenient electronic or mechanical gadget I can use for this instead? Essentially I'm looking for the opposite of a capo. I've heard that octave pedals might be able to accomplish this, but I'm not sure whether they are useful for smaller pitch shifts.
Ideally, I'd like a gadget that could also easily let me retune to A435 or A450 or whatever, and that would work well with an electric guitar or bass.
Comment: I realize that buying a second guitar is an effective way to do this, but I would like a more affordable solution.
Comment: 1/2 step lower than standard tuning is A=415. 1 whole step lower is A=392. You would not want to re-tune to A=435 or A=450.
Comment: I understand the idea behind using a device like this, but in my mind just transposing a song would fix the problem. Just because a song is recorded in a certain key does not mean it has to be preformed in that key. I use to have to put my guitar in D standard tuning for my last band and it really screwed up me being able to name chords and notes that I could easily name in the E standard tuning .
Comment: True, but transposing isn't always an option when working with other musicians or pre-recorded tracks.
Comment: @Wheat Recordings aren't always in A440 or A415. Lots of rock bands (and orchestras) tune to other standards. Sometimes they're just tuned to the piano in the studio.
Answer: Tronical. You can buy one directly from the Tronical company to install on your existing guitar, or you can buy certain models of Epiphone and Gibson guitars with a Tronical tuner already installed. Gibson and Epiphone market them under the trade-name "Min-ETune".
Comment: Wow, that's really cool! Unfortunately at over $300 it costs almost as much as a new Stratocaster, and they don't appear to have bass guitar models, so it's not a practical solution for me. Good to know this is out there though!
Answer: Pitch shift pedals usually do not give a natural detuned sound. There are pedals dedicated for getting a natural detuned sound, like the Digitech Whammy DT or Morpheus DropTune. I haven't used one, so I can't back up their claims.
Comment: @Bradd: My experience of the Digitech whammy is that is quite artificial sounding. That's what the DropTune was supposed to fix. It was earlier versions of the Digitech though, so it's possible they've fixed it. Be sure to try before you buy though!
Comment: The Whammy DT is specifically designed for drop tuning like the Morpheus – that's what the DT stands for in the model name. The current Whammy generation uses different algorithms allowing for more natural polyphonic sound. Of course trying before you buy is important! Thinking of going to check them out at GC this weekend. But anyway, I recommend adding the DT to your answer as it appears to be a superior successor to the Morpheus.
Comment: Unfortunately the local Guitar Center only had the older Whammy 4, so I couldn't test drive the new models. I ended up getting a new piano instead. This is probably the solution I'll go with eventually, though (unless I just end up getting a backup guitar). All of the answers have been **very** helpful though! Thank you everyone!
Comment: Wow, thanks! That's exactly the kind of thing I'm looking for. The DropTune is unavailable but according to its Amazon reviews, there's a whammy pedal from Digitech (Whammy DT, $300) that does the same thing better. These have some disadvantages, but they do avoid problems with tension and intonation that you'd get from manually detuning.
Comment: It looks like the Digitech Whammy 5 can also do drop tuning via its whammy mode. It's not as straightforward to use, but it's $100 cheaper and smaller too.
Comment: @Bradd: Good idea, done.
Answer: I hope this may sort of answer your question. Faced with the same sort of problem, on bass, a 5 string came to the rescue. It had a low B, thus could play a fourth lower than standard. It covered most of the lower notes that would be needed.A guy I work with sometimes uses an 8 string bass - this goes down to a low F# - nearly an octave lower than standard. It also goes up to an F on top, which means there, he's trespassing in guitar territory, but that's not the point for you.
If, however, you're after a bass that will re-tune to particular frequencies automatically, this answer won't help.  It shouldn't take more than half a minute to re-tune a bass manually that is already ,say, at concert, with or without an electronic tuner.
On guitar, the previous answer gives good advice, but it's not cheap.
Comment: Good point, that's another way to do it if I don't mind buying another bass (which is also not cheap). By the way, a B string is only a perfect 4th lower than the E string, not a 5th.
Comment: Oh and you're right that retuning isn't too bad, although I'm not a big fan of the floppy strings you end up with that way, which is another reason I was curious if there was an electronic method.
Comment: Corrected - B is the fifth of E,and it's early here...A fiver will do everything a four string will - and more ! Ebay, less than £!00 last week. A cracking 5 string ! Money back off your old one. What's not to like? Think with the Gibson thing, the strings will go floppy with lowering the tuning.Even if you could get one for bass. In the pipeline, maybe.There could well be a MIDI method, but that'd involve another pick-up...
Comment: Ah, I like my 4-string P-Bass, just wish alternate tunings were less of a hassle. Thanks for the solid advice though!
Comment: One other thought well, two. I'm working on a 4 string bass, tuned to BEAD. With the right strings, it works well. Other - the lower you go, the better the amp. and speakers you'll need.
Answer: There doesn't appear to be a bass version, but Hipshot make a "Trilogy" bridge that allows you to pick one of three tunings for each string individually.
Alternatively, you could just use a capo - down-tune the guitar a whole step then use the capo to bring it back up.
Answer: Have a look at this video:

Apparently he installed 4 drop-tuners, I had never seen it before but it does provide a cool effect and a quick dropped tuning.
It is not as versatile as a second/third/fourth bass, but it is less to carry.
As for price, a drop-tuner can be quite expensive (70-100 USD) but you might find some second-hand ones.
Comment: Solo bassist Michael Manring pioneered the idea of putting Hipshot detuners on every string, on his signature model Zon Hyperbass.
Comment: Huh, that's a clever idea! Looks like another way to get the effect of the Hipshot full bridge on a bass.
Answer: Any solution you choose is going to cost a significant amount of money, such as in buying a new instrument.
There are many models and brands of pitch-shifter effects pedals and rack-mount multi-effects devices that can transpose everything you play down a half-step, whole-step or further, but they all sound artificial, especially if you are playing chords.
If you are talking about bass guitar, you are much better off getting a 5-string bass with a low B, tuning it in standard tuning, and learning to transpose all your fingerings and to avoid playing the open strings. You will get the real natural sound of your instrument, and not something de-tuned or artificially pitch-shifted.
Similarly, you can buy a longer-scale baritone guitar (six-string and tuned to B or C) or 7-string guitar with a low B string, and either use a capo or learn to transpose your fingerings and to avoid open strings. There are many affordable, entry-level baritone and 7-string guitars on the market these days.

Lower guitar tuning


A particle moves along the x-axis find t when acceleration of the particle equals 0

Question: A particle moves along the x-axis, its position at time t is given by
$x(t)= \frac{3t}{6+8t^2}$, $t≥0$,
where t is measured in seconds and x is in meters.
Find time at which acceleration equals 0.
I got the answer 0.5 and 0 but apparently i got the answer wrong. I know I need to use derivative twice. Can someone please help me
Calculation
$velocity = \frac {-3(4t^2-3)}{2(4t^2+3)^2}$
$Acceleration = 36x(16x^4+8x^2-3)$
answer 0.5 and 0 was found through wolfram
Comment: Calculate the acceleration by taking the second derivative of position in respect to time and then solve for a(t) = 0. Show us your work because most likely you made an algebra error.
Comment: Computing the second derivative I get
$$
x''(t) = \frac{12t(4t^2-9)}{(3+4t^2)^3}
$$
so that $x''(t)=0$ has solutions $t=0$ and $t=\pm\frac32$. Since $t\geq 0$ you are left with $t=0$ and $t=\frac32$.
Answer: The acceleration is expressed by the second derivative.
We have $$x(t)=\frac{3t}{6+8t^2}$$ $$x'(t)=\frac{3(6+8t^2)-3t\cdot 16t}{(6+8t^2)^2}=\frac{18+24t^2-48t^2}{(6+8t^2)^2}=\frac{18-24t^2}{(6+8t^2)^2}$$ $$ x''(t)=\frac{-48t(6+8t^2)^2-2(18-24t^2)(6+8t^2)(16t)}{(6+8t^2)^4}=\frac{(6+8t^2) \left [-48t(6+8t^2)-2(18-24t^2)(16t)\right ]}{(6+8t^2)^4}=\frac{ 16\left [-3t(6+8t^2)-2t(18-24t^2)\right ]}{(2(3+4t^2))^3}=\frac{ 16\left [-6t(3+4t^2)-6t(6-8t^2)\right ]}{8(3+4t^2)^3}=\frac{-12t\left [3+4t^2+6-8t^2\right ] }{(3+4t^2)^3}=\frac{-12t\left [9-4t^2\right ] }{(3+4t^2)^3}$$
Therefore, the acceleration of the particle equals $0$ at time $t=0$ seconds and $t=\frac{3}{2}$ seconds.
Answer: First derivative is $\frac{(18+24t^2)-(48t^2)}{(6+8t^2)^2}$ so second derivative becomes $\frac{(6+8t^2)^2.(-48t)-((18-24t^2)(2(6+8t^2)(16t)}{(6+8t^2)^4}=0$ you have an exprssion you will get $t$
Answer: If position is given by $x(t)= \frac{3t}{6+8t^2}$, then aceleration, in the classic sense, is $x^{\prime\prime}(t)=\frac{12t(4t^2-9)}{(4t^2-3)^3}$ for all $t\ge0$.
At which aceleration is 0, is $x^{\prime\prime}(t_0)=0$, which explicitely is $t_0=0$, $t_0=3/2$ and $t_0=-3/2$.
Now, since $-3/2<0$, we forgot this solution and get only with $t_0=0$ and $t_0=3/2$. But $t_0=0$ indeffined us the deffiniont of aceleration. Thus $t_0=3/2$.
Answer: $$x(t)=\frac{3t}{6+8t^2}\to x''(t)=a(t)=\frac{\text{d}^2}{\text{d}t^2}\left[\frac{3t}{6+8t^2}\right]$$
So:
$$\frac{\text{d}^2}{\text{d}t^2}\left[\frac{3t}{6+8t^2}\right]=$$
$$\frac{\text{d}}{\text{d}t}\left[\frac{\text{d}}{\text{d}t}\left[\frac{3t}{6+8t^2}\right]\right]=$$
$$\frac{\text{d}}{\text{d}t}\left[3\frac{\text{d}}{\text{d}t}\left[\frac{t}{6+8t^2}\right]\right]=$$
$$3\left[\frac{\text{d}}{\text{d}t}\left[\frac{6-8t^2}{(6+8t^2)^2}\right]\right]=$$
$$3\left[-\frac{32t(6-8t^2)}{(6+8t^2)^3}-\frac{16t}{(6+8t^2)^2}\right]=$$
$$\frac{12t(4t^2-9)}{(4t^2+3)^3}$$
Now solve:
$$\frac{12t(4t^2-9)}{(4t^2+3)^3}=0\Longleftrightarrow$$
$$12t(4t^2-9)=0\Longleftrightarrow$$
$$t(4t^2-9)=0$$
We got tho equations:

$$t=0$$
$$4t^2-9=0\Longleftrightarrow$$
$$4t^2=9\Longleftrightarrow$$
$$t^2=\frac{9}{4}\Longleftrightarrow$$
$$t=\sqrt{\frac{9}{4}}\Longleftrightarrow$$
$$t=\frac{3}{2}$$


Why are custom User Account fields not shown after account is saved?

Question: Just upgraded to Drupal 7.54 and now when a user account is saved, any other users (other than admin 1 user) are unable to view or update the custom fields associated with the user account. They just see the core fields. The admin 1 user is still able to see all fields.
When the updated user account is viewed or edited by any other user than the admin 1 user (they do have 'administrator' role assigned), they only see the following fields:-

email
password
status
roles
locality

I have disabled modules that act on fields, such as Field Permissions and Conditional Fields, but still getting this behaviour.
Any ideas why this could be happening would be much appreciated!
Comment: This looks like a permissions thing. Do you have custom modules installed? You could try to add some debug code where the field access hook is triggered to see which module might deny access. It is in the field_access function: https://api.drupal.org/api/drupal/modules%21field%21field.module/function/field_access/7.x
Comment: Add a "debug($module);" after "if ($access === false) {".
Comment: Thanks @schlicki that showed up the  og_field_access module name. great tip for quick debugging! (I was waiting on the infrastructure guys to enable x-debug but this got to the bottom of things way quicker :)
Comment: strange how this happens just after saving a user account, but not immediately when the module is enabled.
Comment: glad to help ;)
Answer: Turns out a developer had enabled the "og_field_access" module and this was preventing users from viewing custom fields on the user account pages.
We removed the module dependencies (they weren't needed!) and disabled the og_field_access module and the fields now appear again!


How to cache a view for all users of a role?

Question: I have a view that displays a map. I need to display 7000 entries on the map.
I have installed solar search API to index the content and have created a view using that.
Still, the loading time for that view is around 1 minute for each user. After the first fetch, then it loads fast for that user since it fetches the data from the cache.
I was wondering whether I can cache the entire view for all users of a specific role.
I tried installing the "Views Custom Cache Tags" module.
But I am a bit lost on how to configure it.
Also not sure whether this will solve my problem or not.
Suggestions, please.
Comment: Cache tags won't help here, this is about cache contexts. Default is a context for the hashed user permissions, which is exactly what you want. A cache context for the specific user is only added when there is such a dependency. Then you have to remove this dependency and render it separately.
Comment: Thank You. It helped. How do we know whether there is any such dependency for a specific user or not? I am displaying users list on the map. And there are exposed filters for the view. Is this the reason?
Comment: It doesn't matter whether a filter is exposed, it does, however, matter if you set a default value depending on the current user. Similar the users list, it is not necessarily depending on the current user, but it possibly could.
Comment: I played with this a bit. Created two views - one with exposed filter and the second one without any exposed filter.   The second view was cached as expected. So my conclusion is exposed filter forces each user to load the view completely from scratch. The cache is discarded. Once more observation is when the user logs in again then also the cache is discarded.
Comment: An exposed filter forces each URL to load the view completely from scratch.
Comment: But that is not what I observed. Loading different URL's from scratch is fine. But the same URL (for example the landing page of the view) is loaded from scratch for each user with the same role. In the database as well, I can see an additional raw in the 'cache_data' table, whenever a new user accesses the landing page of the view.
Comment: I'm not sure a view can really be cached per role. What if you have a user with 2 roles assigned which cache should they receive? Views has got to be kept flexible to handle common use cases so that's probably why per role might not work
Answer: To answer the question title and the comment of @Leigh, without going into the details of the complexity of the specific View:

How to cache a view for all users of a role?

Views are cached like any other rendered content. When rendered a View bubbles up the cacheable metadata of Views plugins, like filter or sorting plugins and the rendered content itself.
Additionally there are the three default contexts, as defined in the service container:
core.services.yml:
  required_cache_contexts: ['languages:language_interface', 'theme', 'user.permissions']

The context user.permissions is a hash of all of the user's permissions, which is the result of the roles assigned to a user.
So in the end a View is cached for user roles by default.
Comment: Thanks this was what I was hinting at but didn't have time to explain fully. The cache would need to be a culmination of each user's roles/permissions. Slight tweak to your answer should `is a hash of all user permissions` actually be something like `is a hash of all of the user's permissions`?


Algoritmo de cálculo de melhor rota

Question: Estou fazendo um sistema de lógistica que aborda a questão de rotas/roteamento. Em que devo escolher a melhor rota para viajar.
Inicialmente estou apenas fazendo alguns testes e testando algoritmos avançados, o que estou fazendo parece ser algo ilógico, mas vai atender minha necessidade.
Vamos lá:
Eu tenho um banco de locais cadastrados. Esses locais podem fazer fronteiras com vários outros locais. Para cada localidade é informado a distância em KM.
Com base nos locais cadastrados, o sistema tem uma opção em que o usuário informa dois pontos (origem e destino) e ao final o sistema informa o trajeto para que o veículo chegue ao seu determinado destino considerando o caminho mais curto.
Já tenho alguns cidades, por exemplo, CidadeA, CidadeB, CidadeC, CidadeD, cidadeE.
Por exemplo,
de CidadeA pra cidadeB são 5 km.
de CidadeB para cidadeC são 7 km.
de CidadeC para cidadeE são 22 km.
de CidadeC para cidadeD são 8 km.
e assim como posso ter outras cidades.

E eu escolhi (duas comboboxes) ir da cidadeA pra cidadeB, logo, ele deve me trazer todas as rotas, ou seja, todas as passagens pra chegar até a cidade D do jeito mais "rápido", pela distância, aí ele deve trazer os pontos, assim
CidadeA - > CidadeB
CidadeB - > Cidade C...
e assim sucessivamente.

O usuário informa a origem e o destino, o sistema deve calcular exibindo ponto a ponto por onde deve passar.
Tentei fazer cross joins e analisar, tentei dar for dentro de for até encontrar, mas é algo que parece infinito, mesmo pensando muito fica difícil de montar. Alguém sabe um caminho, algoritmos famosos que façam isso? estou verificando a questão dos algoritmos A*, etc, mas ainda não me ajudaram.
Amigos, eis a estrutura das tabelas:
Tabela Localidade e a tabela DistanciaLocalidades.
    use master
go
CREATE DATABASE Rota
go
USE Rota
GO

CREATE TABLE [dbo].[Localidade](
    [cod_cidade] [int] IDENTITY(1,1) NOT NULL,
    [nome] [varchar](50) NULL,
 CONSTRAINT [PK_Localidade] PRIMARY KEY CLUSTERED
(
    [cod_cidade] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

GO

SET ANSI_PADDING OFF
GO

CREATE TABLE [dbo].[DistanciaCidades](
    [cod_cidade1] [int] NOT NULL,
    [cod_cidade2] [int] NOT NULL,
    [distancia] [int] NULL,
 CONSTRAINT [PK_DistanciaCidades] PRIMARY KEY CLUSTERED
(
    [cod_cidade1] ASC,
    [cod_cidade2] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

GO

Comment: Se você lê inglês, recomendo fortemente o artigo da wikipedia sobre o [*travelling salesman problem*](http://en.wikipedia.org/wiki/Travelling_salesman_problem). [A versão em português](http://pt.wikipedia.org/wiki/Problema_do_caixeiro-viajante) não parece tão confiável.
Comment: Sua questão parece bem lógica sim, só não entendi os seus "kms", você por acaso está usando uma estrutura no banco de dados de n:m? Eu sei que parece irrelevante perguntar isto, mas a estrutura do banco ao meu ponto de vista parece que deve ser analisada também
Answer: Esse é um problema de caminho mínimo, unidirecional acredito (a distância de A para B é a mesma distância de B para A) e sem pesos negativos (a distância é sempre maior ou igual a zero). Considerando que a distância entre as cidades não muda com frequência (exceto pela abertura de novas estradas, etc) e que cidades não são comumente cadastradas/descadastradas, talvez valha a pena utilizar um algoritmo que calcule os caminhos mais curtos entre todos os pares. Então tudo o que você precisa fazer é consultar o resultado, rapidamente, sem ter que recalculá-lo.
Há diversos algoritmos aplicáveis, cada um com características específicas em relação ao nível de complexidade [assintótica], restrições ao domínio, etc, escolha um que atenda às suas necessidades particulares. Na ausência de requisitos específicos, o algoritmo de Dijkstra parece ser o mais simples e eficiente, ao menos para as aplicações mais comuns.
Minha primeira sugestão é implementar (ou buscar por uma implementação já pronta) fora do banco, na camada de aplicação mesmo. Caso você queira fazer tudo no banco, há uma implementação via stored procedure (MS SQL CLR) que pode ser do seu interesse. No restante dessa resposta, vou descrever em linhas gerais como implementar à mão, não necessariamente da forma melhor ou mais eficiente, ok?
Uma implementação simples consistiria no seguinte: Crie uma tabela temporária, com as colunas (cidade, valor, anterior, visitado). Inicialmente, preencha essa tabela com suas cidades, marcando todas como "não visitadas", o anterior como nulo e o valor como "infinito" (ou simplesmente um número muito grande). Então execute os seguintes passos:

Marque o valor da cidade inicial como zero (pois a distância de A a A é zero);
Enquanto seu destino [ou todas as cidades] ainda não foi visitado, faça:

Selecione a cidade X não visitada com o menor valor;
Selecione todas as cidades Y não visitadas que são adjacentes a ela (i.e. alcançáveis a partir dela);
Some a distância entre X e Y com o valor de X, e compare com o valor de Y; se for menor, atualize o valor de Y e o campo anterior de Y;
Marque a cidade X como "visitada".

Ao final, você terá para cada cidade um valor correspondendo à distância mínima entre A e ela, e um anterior dizendo a partir de qual cidade se chega nela percorrendo aquela distância. Para encontrar o caminho exato, parta de B e vá seguindo o anterior até chegar em A.
Comment: Boa tarde amigo. Obrigado. Com a estrutura de tabelas que tenho, será que é possível fazer isso desse jeito igualzinho? obrigado como assim selecionar a cidade c? meu destino é a cidade d, como vou conseguir pegar a cidade c? abraçoo
Comment: @user17245 Sim, sua tabela `DistanciaCidades` possui todas as informações relevantes para montar seu grafo: `(cod_cidade1, cod_cidade2 e distancia)`, correspondendo a `(vértice 1, vértice 2, peso da aresta)`. Mas eu sugiro fazer isso fora do banco, e se possível usar uma implementação já pronta, veja a resposta atualizada. P.S. Mudei os nomes de C e D para X e Y, para não confundir com seu exemplo.
Answer: Pelo que entendi, você tem o problema do caminho mais curto e aí cabe usar o algoritmo de Dijkstra pra encontrar a solução ótima em tempo polinomial. Pra implementar, o ideal seria obter a lista de cidade e suas distancias do banco de dados e montar uma estrutura de dados pra rodar o algoritmo e não tentar rodar isso direto no BD.
Comment: Bom, eu não posso comentar na resposta acima.  Mas, o algoritmo de Thorup só é "eficiente" para uma variante do problema do caminho mínimo (o que possui os pesos inteiros). Se os pesos forem decimal, o de Dijkstra continua sendo mais eficiente. O custo para montar a estrutura de dados usada no algoritmo de Thorup só vale a pena quando se tem muitos vértices - na apresentação só supera o Dijkstra a partir dos 3 mil vértices.
Comment: O caminho mais curto nem sempre é o mais rápido , pense num salva-vidas na areia e um afogando à sua diagonal , nadar em linha reta não seria o mais rápido e sim correr pela areia uma linha reta quase na perpendicular do afogado e aí sim nadar.
Comment: Pesquise sobre algoritmos genéticos e simulacões de formigas (Dorigo).
Comment: No caso da pergunta, as informações que ele possui são cidades e as distâncias (em Km) entre elas. Rapidez para ele é a menor distância percorrida ( ... todas as passagens pra chegar até a cidade D do jeito mais "rápido", pela **distância**, ...). Eu não entendi exemplo do salva-vidas.
Comment: @Motta Ainda é um caso de caminho mínimo. A diferença é que o **peso** de se deslocar na água é muito maior que o peso de se deslocar em terra firme. Colocando-se os pesos certos, um algoritmo de caminho mínimo chegaria à mesma estratégia que você sugeriu, se ela fosse mesmo a melhor é claro (se for necessário um deslocamento imenso em terra para se ganhar uns poucos metros em água, aí já não compensa...).
Comment: @robot_s http://static.nautil.us/2904_b8b9c74ac526fffbeb2d39ab038d1cd7.png
Comment: Só um comentário , os algoritmos citados são os mais conhecidos e implementados.
Comment: @Motta li novamente o que vc havia dito. Vc está considerando que a velocidade na terra (ao andar pela areia) e nadando são diferentes. No caso dele os veículos andam sempre com a mesma velocidade, ou seja, é somente a distância que ele considera para definir o mais "rápido". Se ele quiser considerar outros custos, como pedágio, condição de estrada e tipo de veículo, ai ele vai ter outro problema.
Comment: Mas que pesa, mas foi apenas um exemplo da diferença caminho mais curto x mais rápido. Uma discussão interresante esta que levantou.
Comment: Mais outros um ponto. "faz diferença se a distância é 150.000m ou 150.000,2m?" Quem deve definir a importância disso é autor da pergunta (que ele inclusive, no exemplo, usa valores inteiros em km).
Comment: Novamente, Dijkstra resolve o problema tanto para pesos inteiros quanto para decimais, o Thurop não. Portanto, já que você apontou que o Dijkstra era menos eficiente deveria, pelo menos, indicar quais situações isso ocorre. Baseado nos slides que você mesmo colocou, o experimento que o autor fez mostra a partir de quantos vértices o algoritmo de Thurop supera o Dijkstra's. A respeito da quantidade de arestas do problema do autor, isso pode variar, sobretudo se as cidades forem em uma região metropolitana, por exemplo. Para grafo direcionado basta usar o Dijkstra's com heap de Fibonacci.
Comment: bom dia amigos! muito obrigado pelas sugestões. estou pesquisando e testando todas. com a estrutura de tabelas que botei, como ficaria um jeito de adaptar a esses algoritmos complexos? abraçoo
Comment: @robot_s Ok, removi a referência para Thorup da minha resposta. A propósito, se você possui sugestões melhores, você sempre pode [edit] sua resposta para incluí-las, se for do seu interesse é claro (P.S. vou limpar meus comentários aqui, pois essa discussão já está por demais extensa).

Algoritmo de cálculo de melhor rota


Adding FormData object as value for an object key to be sent as post request

Question: I am working on an app where I need to send an image blob as FormData alongside some other data sent as Json.
For this I'm making an http post request to the api with the following data:
const blob = new Blob(['some-image], {type: 'image/jpeg'});

const formData = new FormData();

formData.append("file", blob);

const data = {
  name: 'John',
  age: '24',
  uploads: formData
};

For http post request I'm using fetch:
async postData() {
    try {
      const res = await fetch(endpoint, {
          method  : 'POST',
          headers : {
            'Content-Type': 'application/json'
          },
          body: JSON.stringify(data),
        });

      return res.status;
    } catch (error) {
      console.log(error)
    }
}

I'm unable to get the file in the backend which just shows an empty object inside uploads['file']. Why is that? Is it not possible to send formData as json with other data?
Comment: Hello, you can't use `JSON` to upload elements to your backend server (don't set the content-type header). Source : https://developer.mozilla.org/en-US/docs/Learn/Forms/Sending_and_retrieving_form_data#the_enctype_attribute
https://muffinman.io/blog/uploading-files-using-fetch-multipart-form-data/
https://stackoverflow.com/a/36082038/12554903
Comment: For me this is not possible.
Comment: @Pioupia so I can either send json or form data and not both?
Answer: You could just send the FormData as the body.
const formData = new FormData();

formData.append("file", blob);

const data = {
  name: 'John',
  age: '24'
};

// add all key-value pairs of object to the FormData
for (const [key, val] of Object.entries(data))
    formData.append(key, val);

// ...
fetch(endpoint, {
    method  : 'POST',
    body: formData
});

Then you can access the blob as the "file" request parameter and each of the object properties as request parameters also (e.g. "age").
Comment: Hi, I know about this but I wanted to send json and one of it's value as formdata. I guess it is not possible.


Autocomplete field - fill the selected id into another form element (eg. hidden field)

Question: this was already asked here
Handling id from autocomplete field
but still without answer: I have an autocomplete field which uses geonames query to search cities. After value is selected I want the city name and admin area to fill into the autocomplete field, and to fill geoname_id of selected value into another (hidden) field. I don't want the geoname_id to be visible - it looks kind of ugly...
I've tried to hack the autocomplete.js and added following line
$("#search-geoname-id").val($(this.selected).data('autocompleteValue'));
into Drupal.jsAC.prototype.hidePopup function
then also tried
$("input[id=search-geoname-id]").val($(this.selected).data('autocompleteValue'));
but both without effect :/
Thanks for your help, Tomas
Comment: You should cut the solution out of your question and post it as an answer, using the option `Answer your own question`. This is not only acceptable, but encouraged.
Answer: I've found solution, so I'll share it with others:
Both options in the question were correct. I just expected that the new value of hidden field will be visible through "see source code" which it's not. When I check the $form_state['values']['geonameid'] it is filled correctly.
The complete solution:
1) In Drupal.jsAC.prototype.found I've filled the 'autocompleteValue' with value instead of key, and added new data 'autocompleteKey' which I've filled with key, surprisingly :)
2) In Drupal.jsAC.prototype.hidePopup just added $('input[id="search-geoname-id"]').val($(this.selected).data('autocompleteKey'));
Hope this helps to someone in future.
P.S. I know modifying core functions is not the cleanest way to do things, but it works.


Can I use same keystore for New Application which will replace exiting application

Question: I have one application in Google Play, Now I had developed one new Application to replace this current android application, Existing application and New Application has different Application Name, Package Name . So I just want to know that Can I use Existing application (Which is currently in market)  keystore for creating new application signed apk file?
My requirement is Existing user will gate alert that new version is available after updating user will get New Application on there device.
I dont want to maintain two seperate apk files (existing application which is in market and New one)
Comment: You need to use the same keystore file and package name .Then you can update the current app in playstore with your new file.ALso you need to increment the version code and name in manifest file .thanks
Comment: When an app installed on device its using its package name.So if you changed it will be a seprate app two app will install on users device.Also it will not override the current one in the playstore
Comment: @ArunAntoney  If package Name is different then I can't use same keystore to update the apk?
Comment: @PPD it will not be considered as an update. Check my answer.
Answer: My requirement is Existing user will gate alert that new version is available
To give users alert that a newer version is available you will have to keep the package name and signing key same and increase the app version number and version code.
Can I use Existing application (Which is currently in market) keystore for creating new application signed apk file
Yes. You can use same keystore to sign any number of applications you want.
But remember signing an app (with different package name) with same keystore does not make it an update to existing app. If package name is different, it will be treated as a brand new application.


View all nodes of a content type using a filter

Question: I have a content type called "Building", which contains a field called "Size". I also have an exposed filter that filters through the size chosen by the user, so the view only displays that size.
What I would like is that the view sorts and displays all the nodes in order of relevancy, i.e. if I choose 5-10 meters, the 5-10 meter buildings show up first, and then the rest of the buildings are also shown.
Thanks, and please feel free to comment critically on this post so I can improve on my questions for the future.
Answer: In this case, if you are providing a contextual filter, the view will render the filtered out result. However, as I understand from your question, you need the entire nodes of that content type with the contextual filter values on the top.
For this you can try using hook_views_post_execute() where the pre_render() function is yet not called and sort manually. Or, you can use hook_views_pre_render() function to shuffle the result. At this point, you have all the nodes and you can sort them manually.
Hope this is helpful. Thanks


Node ID caching on hook_form_alter()

Question: I have a webform with an email field. I am embedding the webform on each node, with a field on the node for the email address.
I implemented hook_form_alter() in my .theme file that loads the node using $node = \Drupal::routeMatch()->getParameter('node');.
I can then load the node and get the proper email address and set it in the webform. The problem is that the node is cached for anonymous users. It doesn't matter what node I'm on, $node->id() always returns the same, cached node ID.
I have tried either these lines, but the node ID is still the same.
\Drupal::service('page_cache_kill_switch')->trigger();

$vars['#cache']['max-age'] = 0;

I understand I have to clear cache tags and cache context, but none of the examples shows exactly how to do this in hook_form_alter().
How can I properly clear the cache?
Answer: When you get route parameters you need this cache context:
$form['#cache']['contexts'][] = 'route';

You might also want to add the node cache data so that the form gets invalidated when the node is saved with a different field value:
$renderer = \Drupal::service('renderer');
$renderer->addCacheableDependency($form, $node);

The cache context is applied to the form and all page elements where the form is placed in. So this works no matter where you embed the form.
If the form is embedded within a node, getting the node from the route match is probably not the best idea. You should use the node or node ID as parameter when calling the form. Then you can try the answer from @sonfd because you don't need separate cache keys for the node and webform.
Comment: This is what did it. Thank you so much!!

 $form['#cache']['contexts'][] = 'route';
Answer: The problem is that the webform gets cached with the $node data from the first node visited, but you need the webform to be unique per node.
You can unset the cache keys for your webform - this will prevent the webform from being cached on its own, but will still allow the webform to be cached with its parent (the node), with all of the webform's cache metadata bubbling up.
In your hook_form_FORM_ID_alter() (the one you're using to set the email address), you can unset the cache keys like:
if (isset($form['#cache']['keys']) {
  unset($form['#cache']['keys']);
}

Node ID caching on hook_form_alter()


Prove that the limit of the sequence $a_n:=\frac{6n+1}{2n+5}$ is 3, using the Epsilon definition.

Question: I want to show that for sequence $(a_n)$, defined as $a_n=\frac{6n+5}{2n+5}$, the following statement is true:
$$\lim_{n \rightarrow \infty} {\frac{6n+1}{2n+5}}=3$$
I started with stating that there exists an arbritary but fixed $ε>0$ and assumed that for an $n$ and $N_ε$, $n > N_ε$. I then used the Epsilon definition to prove that the limit is 3.
$$|a_n-3|<ε=|\frac{6n+1}{2n+5}-3|<ε \iff (\frac{6n+1}{2n+5}-3)=\frac{-7}{n}<ε \iff \frac{-7}{ε}=n$$
Here I set $\frac{-7}{ε}=:N_ε$ and since we know that $n>N_ε$ that means that $$\frac{1}{n}<\frac{1}{ε}=\frac{1}{\frac{-7}{ε}}=\frac{ε}{-7} ≠ ε$$
I don't understand where I went wrong in this proof, I've tried the same thing with other examples and get  stuck in the same place.
Answer: For some reason, your absolute value vanished. It turns ou that$$\left\lvert\frac{6n+1}{2n+5}-3\right\rvert=\frac{14}{5+2n}.$$And$$\frac{14}{5+2n}<\varepsilon\iff n>\frac{14-5\varepsilon}\varepsilon.$$
Answer: You made one algebra error, and a number of other errors in the way your proof is organized and presented.
The algebra error, as José Carlos Santos notes, is that ${6n+1\over2n+5}-3={-14\over2n+5}$, not $-7\over n$.  So that of course affects what comes after. However, let's suppose $a_n-3={-7\over n}$ is correct, and go from there.
The first, minor, problem is either a typo or a misuse of the equal sign: where you write
$$|a_n-3|\lt\epsilon=|{6n+1\over2n+5}-3|\lt\epsilon$$
you really mean is $\iff$ instead of $=$, i.e.,
$$|a_n-3|\lt\epsilon\iff|{6n+1\over2n+5}-3|\lt\epsilon$$
Also, at the other end of the display, where you write ${-7\over\epsilon}=n$, it should be ${-7\over\epsilon}\lt n$.
The second, more serious, problem is that $|a_n-3|\lt\epsilon$ is not equivalent to $(a_n-3)\lt\epsilon$; the absolute value sign is important. So the first set of displayed expressions should end in
$$|a_n-3|=\left|-7\over n\right|\lt\epsilon\iff {7\over\epsilon}\lt n$$
and thus we should set $N_\epsilon={7\over\epsilon}$.
The next problem is merely, I think, a typo: where you say that $n\gt N_\epsilon$ means that ${1\over n}\lt{1\over\epsilon}$, you pretty clearly meant to write ${1\over n}\lt{1\over N_\epsilon}$, since you go on to substitute the expression you derived for $N_\epsilon$ in the continuation.
The final, most significant, problem is one of logical relevance. You note, correctly, that $\epsilon\over-7$ (which, as noted above, shouldn't have the minus sign) is not equal to $\epsilon$. But that doesn't matter. What you really need here is a set of implications along the lines of
$$n\gt N_\epsilon\implies{1\over n}\lt{1\over N_\epsilon}={\epsilon\over7}\implies|a_n-3|={7\over n}\lt7\cdot{\epsilon\over7}=\epsilon$$
Of these, I'd say, the final problem is the most important: It looks like you were trying to derive the inequality ${1\over n}\lt\epsilon$ instead of $|a_n-3|\lt\epsilon$.  In other words, it looks like you lost track of what it is you are supposed to prove. It might help, for future proofs of this type, to keep in mind that, come hell or high water, you need to wrap things up with a logical sentence of the form $n\gt N_\epsilon\implies|a_n-L|\lt\epsilon$.
Answer: Be careful:

When we're given an arbitrary $\varepsilon>0$ we have to find $N_\varepsilon$ such that, for $n>N_\varepsilon$,
  $$
\left|\frac{6n+1}{2n+5}-3\right|<\varepsilon
$$
  We don't need to do "$\iff$" as you seem to want. Now consider that
  $$
\left|\frac{6n+1}{2n+5}-3\right|=
\left|\frac{6n+1-3(2n+5)}{2n+5}\right|=
\left|\frac{-14}{2n+5}\right|=
\frac{14}{2n+5}<\frac{14}{2n}=\frac{7}{n}
$$
  We know have $7/n<\varepsilon$ provided $n>7/\varepsilon$, so we can take $N_\varepsilon$ the largest integer less than or equal to $7/\varepsilon$. For $n>N_\varepsilon$ we thus have
  $$
\varepsilon>\frac{7}{n}\ge\left|\frac{6n+1}{2n+5}-3\right|
$$
  which ends our proof.

Be careful when you remove the absolute value sign and also how you look for upper bounds.


What properties do natural isomorphisms between functors preserve?

Question: If $F,G: C \rightarrow D$ are functors between regular categories and $F \Rightarrow G$ is a natural isomorphism, is it true that if $F$ is faithful {resp. full) then $G$ is full?  I believe this is true but would like some confirmation.
What about if $F$ is essentially surjective?  Is it necessarily true that $G$ is essentially surjective?
Thanks.
Answer: Let $C$ and $D$ be categories (arbitrary), $F,G\colon C\to D$ functors, $\alpha\colon F\to G$ a natural isomorphism. Then:

If $F$ is faithful, then $G$ is faithful.

Proof: Let $c_1,c_2\in C$ be objects and $f_1,f_2:c_1\to c_2$ morphisms in $C$. Then, by naturality:
$$
G(f_i)\circ \alpha(c_1)=\alpha(c_2)\circ F(f_i),\quad i=1,2.
$$
We know that $\alpha$ is an isomorphism, so we have:
$$
G(f_i)=\alpha(c_2)\circ F(f_i)\circ(\alpha(c_1))^{-1},\quad i=1,2.
$$
Therefore, if $G(f_1)=G(f_2)$, then $F(f_1)=F(f_2)$, and finally $f_1=f_2$.

If $F$ is full, then $G$ is full.

Proof: Let $c_1,c_2\in C$ be objects and $g\colon G(c_1)\to G(c_2)$ a morphism in $D$. Let's define the morphism $k\colon F(c_1)\to F(c_2)$ as follows:
$$
k=(\alpha(c_2))^{-1}\circ g\circ\alpha(c_1).
$$
$F$ is full, therefore there exists a morphism $f\colon c_1\to c_2$ in $C$, such that $F(f)=k$. It's easy to check that $G(f)=g$.

If $F$ is essentially surjective, then $G$ is essentially surjective.

Proof: Let $d\in D$ be object, then there exists such $c\in C$, that $d\cong F(c)$. But $F(c)\cong G(c)$ by $\alpha$, thus, by transitivity, $d\cong G(c)$.


What is the difference between Formal Logic and Proofs?

Question: I'm a graduate CS major, and am reading Social Processes and Proofs of Theorems and Programs. But I can't make any sense of it. After searching for resources on YouTube and other sites, I came here in hope that someone could help me.
Asking for explaining the core argument in this paper is too much to ask, so I'll ask questions referring to the introduction:

Many people have argued that computer program- ming should strive to
  become more like mathematics. Maybe so, but not in the way they seem
  to think. The aim of program verification, an attempt to make
  programming more mathematics-like, is to increase dramatically one's
  confidence in the correct functioning of a piece of software, and the
  device that verifiers use to achieve this goal is a long chain of
  formal, deductive logic. In mathematics, the aim is to increase one's
  confidence in the correctness of a theorem, and it's true that one of
  the devices mathematicians could in theory use to achieve this goal is
  a long chain of formal logic. But in fact they don't. What they use is
  a proof, a very different animal. Nor does the proof settle the
  matter; contrary to what its name suggests, a proof is only one step
  in the direction of confidence. We believe that, in the end, it is a
  social process that determines whether mathematicians feel confident
  about a theorem-and we believe that, because no comparable social
  process can take place among program verifiers, program verification
  is bound to fail. We can't see how it's going to be able to affect
  anyone's confidence about programs.

The 2 most frustrating questions I have from this excerpt are:

What is the difference between Formal Logic and Proof that
the author is talking about? A simple example would be appreciated.
What does the author mean when he talks about social process?
Comment: This seems to be more a paper about psychology and feelings than computer science and/or mathematics. Proofs use logic. Formal logic is a basis of proof.
Comment: That paper is forty years old.  Not everybody is prophetic all the time.
Answer: The distinction the author is making is between formal proofs and informal proofs. Most of mathematics is done informally. Informal proofs are just persuasive arguments written in natural language. They are, of course, very stylized, but ultimately it's just an argument. A formal proof is written in a formal language that is in principle (and nowadays in practice) machine-checkable. And for real logics, the algorithm to check such proofs is relatively simple. It is very rare for mathematicians to produce formal proofs, but as tooling for proof assistants improves, largely driven by the needs of software engineering, it becomes more popular.
The social process referenced is presumably processes like peer review and the more amorphous reproving, alternate proofs, alternate explanations of existing proofs that happens as people study a theorem and its proofs. If a bunch of people recheck the work and don't identify any problems, then it's evidence that there aren't any problems. It's clearly not incontrovertible proof that there are no problems.
The following isn't related to your specific question. As Fabio Somenzi points out in a comment, this paper is 40 years old (published in 1979). It's hard for me to imagine someone holding the views given in the quoted introduction nowadays. The reason formal proofs aren't commonly used in software development is that it is perceived (mostly likely, correctly) as not being cost-effective. The main concern that would cause a software developer to not trust some formally verified code is specification error which is not at all helped by using a social rather than mechanical process to verify the proof. It doesn't matter if your theorem is true if your assumptions are wrong. A lot of the impetus for proof assistants (which would have been young or non-existent in 1979) is precisely because informal proofs proved inadequate for verifying software, particularly concurrent software.
Comment: "the algorithm to check such proofs is relatively simple" - citation needed. All proof assistants I know regularly have bugs allowing you to prove False. If the algorithms were trivial, these would not exist.
Comment: @WorldSEnder https://github.com/sorear/metamath-turing-machines gives an indication of the Kolmogorov complexity of a ZF proof checker is surprisingly low. More practically, http://twelf.org/wiki/Zermelo_Frankel is an implementation of FOL (plus the ZFC axioms) in Twelf and https://github.com/logic-tools/sml-handbook/blob/master/code/SML/lcf.sml is an LCF-style implementation. At any rate, "relatively simple" is not the same as "trivial", especially when you do need to care about performance and other non-functional concerns.
Comment: That said, there are several proof assistants that are decades old that have had few if any "proves false" bugs to my knowledge. They certainly haven't "regularly" had them. On the other hand, something like Agda doesn't satisfy the De Bruijn criterion so mistakes anywhere in implementing its rich surface language can lead to inconsistency. (Also, Agda has an unclearly specified and changing language it's implementing). There's also a difference between finding out your logic is inconsistent and incorrectly implementing it, e.g. impredicative Set+LEM+Description in Coq being inconsistent.
Answer:
After reading the paper a bit, I think the autors mean that the proof is more than just the formal logic behind it (they cite Bourbaki):

Indeed every mathematician knows that a proof has not been
  "understood" if one has done nothing more than verify step by
  step the correctness of the deductions of which it is composed and
  has not tried to gain a clear insight into the ideas which have led
  to the construction of this particular chain of deductions in preference to every other one.

I think I agree unwillingly. When I write formal proofs in Mizar (a proof checker where I do have to go into every last detail) and they become longer than a hundred lines, I feel the need to add extra commentary to give the reader an understanding how the proof works. Shorter proofs are usually self-evident. So I'm doing it myself, yes. But I disagree with the authors that using only formal logic doesn't increase the confidence.

If you have a proof for a theorem, but no one reads it, is the theorem really proven for the rest of the world? Mathematicians are, sadly, just humans. When Cantor originally introduced his set theory, he was often frowned upon or simply ignored by the mathematical community. A few decades later David Hilbert said

No one shall expel us from the paradise that Cantor has created.

It is still a problem today. If you prove that, e.g. P=NP, then it will take time for the community to acknowledge the existence of the proof, because from their perspective, there have been so many wrong ones, why should yours, you being an unknown mathematician, be the right one? Fat chance.
The webcomic SMBC has a nice but slightly exaggerated comic about the social process.
Answer: This is a huge topic.
The first point to notice is that for decades a myth, brilliantly fostered by Bill Gates, was widely accepted: that computer software is inherently buggy, because it is so complex. That is at best a half-truth (like most myths). The real reason most software is so buggy is that there are good commercial reasons for it being buggy. Getting wide acceptance for the idea that you have to continually pay for updates/upgrades/"security enhancements" etc has generated billions for the industry. Not least because it can lead to a vast network of "officially approved" consultants who endorse your software and help to make it usable.
There are plenty of examples of cases of bug-free software being produced. For example, the "temporary supervisor" (supervisor was the old term for operating system) written for the first Cambridge University computer by Peter Swinnerton-Dyer. Or software written for JPL for space missions, or much high-speed trading software written today (there are few faster ways of losing millions that writing buggy high-speed trading software).
Of course, one could - and people have - write entire books about the problems of managing large software projects. It is certainly hard. The UK government, for example, has a long string of appalling military software messes to its discredit (torpedo guidance software so bad that submarine commanders wouldn't launch them for fear of sinking themselves, Nimrod etc).
Nonetheless, it is interesting that there has been a move towards using Haskell-type languages for programs (like high-speed trading) where there is a major financial incentive to make the software bug-free.
The second point to notice is the difficulty the math world has had over the last few decades with long proofs. Few people have understood in any detail the proof of the 4-colour theorem. Even fewer have understood the proof of the list of "exceptional" finite simple groups. Then there is the notorious case of the maybe-proof of the ABC-conjecture, which conceivably could be settled next year in the program of seminars and workshops organised by Shinichi Mochizuki and his colleagues. Note also the the messy births for the proofs of the Poincare conjecture and Fermat's last theorem.
Only a tiny number of people could prove the finite simple groups result if stranded on a desert island without comms. Rather more could prove Fermat's last theorem (and would do it rather differently from Wiles/Taylor - I remember sitting next to Gerd Faltings in a seminar at the Newton Centre and bemoaning my failure to get to grips with the proof, whereupon he offered to take me through a proof in 20 minutes). A similar number could probably manage the geometrization conjecture. A much larger number would understand the basic idea behind the 4-colour theorem and could probably scrape together an algorithm to run on a computer.
But in none of these cases can anyone put together a dozen lectures to present the proof to bright teenagers. Indeed none seem yet to have made it into the undergraduate curriculum.
So what is going on? Why are these proofs widely believed?
Part of the reason is that math is a vast structure, full of interconnecting parts. People immediately found Grisha Perelman's proof plausible because of William Thurston's widely known conjecture. Similarly people immediately found the Wiles/Taylor proof plausible because of the Taniyama-Shimura conjecture.
Another reason is that the proofs were produced by mathematicians who already had a high reputation.
But it is still somewhat strange. Why, for example, are most people still unwilling to give a verdict on Mochizuki's claimed proof? He is clearly following the hallowed Grothendieck approach, with all the hallmarks of that style of maths - endless apparently trivial lemmas, most involving hard to grasp concepts, with hard-to-prove results apparently emerging unexpectedly.
You can point to the minor mistakes in Mochizuki's work as undermining confidence. But Andrew Wiles' initial proof had mistakes, some of which were arguably not minor. Or take the case of Louis de Branges and (1) the Bieberbach conjecture, (2) the Riemann zeta hypothesis. Few people believed his proof of (1) until it was debugged and validated by the Leningrad seminar. Yet almost no one pays much attention to his claimed proof of (2). Why not? Clearly, he is prone to carelessness, but the work clearly has a respectable pedigree (eg his book on entire functions).
The last case is interesting. A major problem in maths today is that the system demands regular publication. At the junior end it is completely ridiculous. Who could possibly hope to produce anything non-trivial and novel every year? Most mathematicians produce nothing in that category in a lifetime of research. Of course, Paul Erdos produced a huge number of novel and hard-to-prove results, but he was mainly working in ill-understood areas, doing "bare hands" proofs. A classic example would be the early work on graph theory, which most mathematicians dismissed as trivial until Bela Bollobas' Modern Graph Theory showed it had a respectable body of serious results and associated techniques.
But the consequence of too much teaching, too much admin, and the requirement for too much publication, is that most mathematicians find it hard to get large blocks of uninterrupted thinking time. They are reluctant to use it to attack problems that look too hard. They are even more reluctant to spend time looking at something like Mochizuki's work, which requires ploughing through thousands of pages of material which is way outside the mainstream. What could they gain - apart from the private satisfaction of having nailed down some truth. If they show the proof is hopelessly flawed, they get no credit and it is quickly forgotten. If they show the proof is correct, almost all the credit goes to the person who originally did the work, even if the "verifier" had to make/help make substantial fixes (eg how many people give Richard Taylor much credit for the proof of Fermat's last theorem).
The third point is that it is hard to think of a significant result that has been discovered or proved by a "general purpose" (computer) algorithm. There are several cases of "book proofs" unexpectedly produced by software for theorems/lemmas in Euclidean geometry. But in general nailing down a complete proof from the axioms is a massive task. A fair amount of work goes on in that area, but its fruits have been disappointing. It has usually done no more than confirm results everyone already accepted.
A classic reference on the "social aspects" of (mathematical) proof is William Thurston's 1994 "On proof and progress in mathematics" available here
Answer: Formal logic, and proofs in formal logic, is a game where you encode some symbols and have rules for how you manipulate them.
There is no truth in Formal logic, except how you associate the symbols you write with concepts.  Now, those symbols are often designed to align really well with truth, so the two mix relatively well.
The thing is, most uses of "formalism" is very informal.  When you define a "formal" model of computation, 9999 times out of 10,000 you talk about it informally.  You'll elide or hand-wave some step.  Partly because those steps are boring and annoying to formalize, and partly because humans are fallible.
Today we have formal proof checkers.  These let you play the formal logic game with a computer-mediated umpire, who takes your proof and makes sure you aren't breaking the rules of the game.
We also have computer proof generating assistants, who can take two assertions you believe are connected and generate a formal proof (the annoying, small steps) that one proves the other, with more or less success.
This is a small corner of mathematics.  Most mathematics doesn't use this.  Instead of using formal proof checking programs and assistants, they instead write relatively informal proofs.  Steps are skipped -- elided -- with the understanding that the reader can convince themselves of the validity of the skipped steps.
Proofs then become a matter of writing for the audience; to consume such a proof, you need to have a certain degree of "mathematical maturity" in the areas covered, or the "A follows B" obvious steps won't be obvious to the reader and everything falls apart.
Often such proofs can be turned into formal proofs, but the process is really hard and long.  And sometimes there turns out to be "technical" errors in the proof that need adjusting in order to make the proof formally provable.
In that sense, the formal proofs are stronger than the informal proofs.  But in another they are sometimes not.
With the informal proofs, you are often taking the medium-informal definitions involved and a medium-informal statement of the theorem and proving it in a way that is highly convincing to a subject matter expert.  What more, the proof itself may contain useful insights into why the theorem is true.
In the formal proof, you are instead taking a specific formal set of terms and definitions and proving them.  If those specific formal set of terms and definitions don't align with the medium-informal definitions and statement of theorem, then you can prove something different accidentally.  Reliably connecting the computer-understood formal terms to the medium-informal theory and theorem can be a matter of non-trivial effort; in fact, you will probably have to prove they are the same thing, using an informal proof.
Then, the body of the computer-checked proof could be so technical and dense that the "reasoning" could be opaque to the human reader.  This is a loss, as often the structure of the proof sheds light on related problems and extensions.
I'll then relate this back to computer software.  A formal proof that a program is "correct" then requires that the "correctness" of the program be what the programmer wanted.  Expressing that -- what the programmer wants -- is in a sense the problem of programming in the first place!
Programmers create programs that write programs for them all the time.
Every compiler is a program that takes in a "higher level description" of a problem and outputs a lower level description of the problem.
When programmers attempt to create ridiculously high level languages, like the GUI building toolkits, what is universally found is that you lose strength and performance to a huge degree.
Going from C or C++ or similar languages to a gc+bytecode language costs 2x-3x performance; going to a scripting language costs 5x to 10x.  Power -- what things you can do -- is also lost (this is typically worked around by having the parts of the program that need power be written in a lower-level language, and access it from the higher level one).
Languages like Java came out of research into writing provably correct programs.  Garbage collection and lack of pointer arithmetic and similar make proving properties about the language easier; in comparison, C or C++ has piles of "if the programmer does this, the language makes zero guarantees"; you can write provably correct C or C++, but you end up highly limited in what constructs you can use.
Other languages attempt to give C or C++ levels of abstraction, but add on a proof system to prove that you aren't wandering into the lands of undefined behavior.  You can see this in Rust.
But, fundamentally, the entire problem of computer languages and compilers is attempting to do exactly what the formal proof programs in mathematics are doing.  They are attempting to allow the author to write a higher level description and have the lower level details handled automatically by a computer.
In short, a sufficiently accurate description of a computer program is a computer program already; saying it is "correct" then becomes checking if the description is correct, which is a problem of meaning not mathematical proof.


How to reverse proxy grafana on ingress without using helm?

Question: I wanna reverse proxy grafana. I am not using domain to access ingress, instead i use the node's IP. I am using rewrite target to do the job but, it doesnt work. It is always redirect the app to /login without prefix /grafana (404 error). Expected url is /grafana/login.
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    app.kubernetes.io/component: controller
  name: nginx-example-grafana
  namespace: grafana
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  controller: k8s.io/ingress-nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress-grafana
  namespace: grafana
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx-example-grafana
  rules:
    - http:
        paths:
          - path: /grafana(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: grafana
                port:
                  number: 3000

Comment: As per the official [documentation](https://grafana.com/tutorials/run-grafana-behind-a-proxy/),you can Include the sub path at the end of the "root\_url" in the Grafana configuration file and set "serve\_from\_sub\_path" to "true".
Comment: It works, thank youu
Comment: Thanks for your response. Since the offered suggestion has helped you in overcoming the issue, I have converted the comment as a response to have greater visibility to community members. Could you please provide feedback in it as explained in [https://stackoverflow.com/help/someone-answers](https://stackoverflow.com/help/someone-answers)
Answer: As per the official documentation from Grafana Labs,you can Include the sub path at the end of the root_url in the Grafana configuration file and set serve_from_sub_path to true.
Answer: I'm not sure I understand. It is the expected behavior with the nginx.ingress.kubernetes.io/rewrite-target: /$2 annotation. I will redirect to the second capture group of the regular expression of your path, therefore changing /grafana/login to /login since /login matches the second capture group of your regex.
I would simply remove the redirection, and leave the path to /grafana.
Comment: still doesnt work, i dont know why it still redirect the url without prefix. I found the workaround by insert the required value as suggested by @kiran-kotturi


How can I mirror all traffic to another server?

Question: I have 2 different CentOS servers (A and B) on different networks. How can I mirror/retranslate all traffic from all ports from server A to server B?
Server A has many runned applications that sometimes changes listening port to another random available port. Any clients in internet can access to these applications through server A, and they shouldn't know the real IP of server B. How can I configure server A for this?
For example:

IP of server A is 184.108.40.206
IP of server B is 22.214.171.124

P.S. The apps themselves don't reveal information about the network. I'm only interested in a method of relaying traffic to a specific server
Answer: A port mirror in a switch would be the most common. Are these physical or virtual hosts? Self-hosted or cloud-based?
Comment: Self-hosted physical hosts located on 2 different cities. Unfortunately, the task must be solved at the server level. No changes in switch
Answer: Would I be correct in thinking your application relates to the likes of DDoS evasion?
If your application is HTTP, then it sounds like what you might be after is the concept of a reverse proxy; httpd and nginx are commonly used in this regard, although there are many options these days (particularly if you're into cloud-native). There are also cloud service vendors such as CloudFlare that specialise in this.
You can do something similar with plain TCP too. Haproxy would be one-such example, and at least knowing what the PROXY protocol is can be helpful if you want to maintain visibility of the client IP and have programmer-level control over the service.
If UDP, you can do this with tools such as port mirroring.
If you want to be able to replay (and rewrite) traffic after the fact, then tools such as tcprewrite and tcpreplay (they work with UDP too), or tools like Scapy (a Python framework) can be useful, depending on your use-case.
Is the fact that you have dynamic ports a problem for you, or is that a concious decision, as I could imagine both scenarios.
Comment: No, this is not for DDoS evasion.Just need to proxy all requests through another server so that no one could to know the real IP of the main server


List s3 files within folder lists only first folder files

Question: I have a S3 bucket bucket1 and a folder inside it as test1. There are a bunch of folders in that test1 folder folder1, folder2, folder3. Each folderx contains bunch of files. I want to list all files in all folders. So list folder1's files and then folder2's files and so on.
s3_bucket --> test1
                   ---> folder1
                               --->file1
                               --->file2
                   ---> folder2
                               --->file1
                               --->file2
                   ---> folder3
                               --->file1
                               --->file2

However with my code, it just lists folder1's files. Could someone please verify my code?
s3 = boto3.client('s3')
resp = s3.list_objects_v2(Bucket=s3_bucket, Prefix=s3_path)

for file in resp['Contents']:
     key = file['Key']
     print key

PAGINATION: which doesn't iterate over folder2. folder1 has 1000 files.
    s3 = boto3.client('s3')
    kwargs = {'Bucket': s3_bucket}

    while True:
        resp = s3.list_objects(Bucket=s3_bucket, Prefix=s3_path)
        try:
            contents = resp['Contents']
            print(len(contents))
        except KeyError:
            return
        for file in contents:
            key = file['Key']

        try:
            kwargs['ContinuationToken'] = resp['NextContinuationToken']
        except KeyError:
            break

Comment: Now this is turning into another question on pagination. Please read the docs on [list_objects_v2](https://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects_v2) and the [examples](https://boto3.readthedocs.io/en/latest/reference/services/s3.html#examples). It is very straightforward.
Comment: working perfectly for me. please check if Prefix is proper or not
Comment: @anand can you send me the prefix you are using for my case?
Comment: Maybe you are getting a paginated response, and there are more to retrieve using `ContinuationToken`?
Comment: @Atihska the prefix is `test1/` if you want everything in "test1".  You may also need a paginator.  https://boto3.readthedocs.io/en/latest/guide/paginators.html
Comment: @dmulter You are right. it lists 1000 items. What is ContinuationToken. Is this required just for printing to console or in order to actually process the data beyond 1000?
Comment: @Michael You are right. it lists 1000 items
Comment: So folder1 contains 1000 files itself. I tried pagination but it doesn't continue to the next folder. Please check my edits for code I used.
Comment: @Atihska I didn't use any `prefix`. try without prefix once.
Answer: Thanks all @dmulter @Michael @anand for all your suggestions and guidance.
This worked where s3 path is the key (in unix terms, folder path excluding bucket).
s3 = boto3.resource('s3')
bucket = s3.Bucket(s3_bucket)

for obj in bucket.objects.filter(Prefix=s3_path):
    key = obj.key


What kind of tests should I run for my master's thesis?

Question: I am analyzing the effect of covid rates and covid lockdown levels on murder and suicide rates for the 50 most populous counties in the U.S. I am not sure what to do to analyze the question in aggregate terms- I have transformed the data into panel data: I have 36 months and 50 counties, so 1800 cells of data for suicide and murder. I also have the covid rates in panel data format as well- what kind of variable could I use for an instrumental variable for a fixed effects regression? or what kind of regression would work better for this question? Thanks!
Comment: Do you at least have a working theory in mind? Id start there rather than chasing trends in your data.
Comment: @ShawnHemelstrand my current model is:

Murder Rate_ijt = β_0 + β_1 COVID Rate_ijt + β_2 Lockdown Level_ijt + β_3 Unemployment Rate_ijt + β_4 Civilian Labor Force_ijt + X'_ijtγ + County_i + Time_t + ε_ijt

where Unemployment Rate_ijt and Civilian Labor Force_ijt are the unemployment rate and civilian labor force in county i in month j and time t, respectively.

The same is said for the suicide model
Comment: Perhaps you can edit that into your question.
Comment: What is your hypothesis?
Answer: To use two-stage least squares (2SLS) regression for your analysis, you will need an instrumental variable that is strongly correlated with your explanatory variable (i.e., covid rates and covid lockdown levels), but uncorrelated with the error term of your outcome variable (i.e., murder and suicide rates).
One potential instrumental variable for your analysis could be the date that the county implemented its covid lockdown measures. This variable is likely to be strongly correlated with covid rates and lockdown levels, but it may not be directly related to murder and suicide rates.
In terms of methodology, you can just use 2SLS.


Does the .clone() method always make a deep copy?

Question: While reading through a few of the Rust docs, I came across the following;

... use .clone() to explicitly copy the data.

The choice of "copy" here kind of puzzled me, specifically whether I can always replace "copy" with "deep copy" and for the statement to always hold true.
To my understanding, the Clone trait indicates the possibility of either a shallow or deep copy of the data type implementing it, but the explicit use of .clone() method will create a deep copy.

Is my understanding correct?
If not, what are some examples the method does not create a "deep" copy of an object?
Comment: How `clone()` will be implemented will depend on the type. The compiler only requires it to return a `T` given an `&T`. The contract of the trait is somewhat vague, it speaks of "duplicating" and "returning a copy of" a value. Intuitively,  the requirement is that the copy behaves and has the same meaning as the original. This applies to both cloning a `Vec>` and cloning an `Arc>` - although one is deep copy and the other shallow (or no copy at all, depending on what you count as copy), in both cases you get a value that is equivalent to the original.
Comment: The whole question is kinda...off-base? In managed languages the question makes sense but in Rust unless you are explicitly dealing with a pointer type like `Rc` or `Arc` how could it possibly be anything other than deep? Calling `.clone` returns an *owned* type, meaning it owns all the memory allocated to hold the thing. The original owner can't still own the memory holding the original thing (or even part of the thing for aggregates), so it implies a deep copy. Similarly for a shared pointer it makes sense that calling `clone` gives you a copy of the *pointer*, not the thing pointed at.
Comment: ...also Rust does not have objects. I realize that Rust structs kinda sorta look like objects if you squint and they meet many criteria of objects from OO languages (ditto for traits and interfaces), but it's not a perfect fit and I suspect you are being lead astray by the superficial similarity when in this case the difference matters.
Comment: @JaredSmith I agree with your sentiment, but I've got one small nit: `.clone()` only returns an owned type if `T` happens to be an owned type. If `T` is, say, a shared reference, then `.clone()` will return a copy of the reference. It sometimes confuses people that `"Hello".clone()` doesn't really do anything useful.
Comment: @SvenMarnach good point, I guess it never occurred to me to clone references :P
Answer:
Is my understanding correct?

No. It is true that Clone can be either shallow (e.g. Rc) or deep (e.g. Box) copy, but it is always triggered by calling .clone(), so .clone() can make a shallow copy too (depending on the type).


Static/Class Variables and GUI

Question: I recently was pulled back into Java programming for academic purposes and I ran into something interesting during my first project.
I had to design a GUI for a storefront with buttons that did various tasks, and I noticed something weird was happening with my variables. Any CLASS VARIABLES that my buttons changed through their Actions did not need to be declared static, while any CLASS VARIABLES my main method touched had to be declared static.
Why is this? My Main Method and ActionListener Methods are all on the same level of execution. (they're all in the same class, none of them are embedded within each other)
I hope this question is not too obvious/dumb, I am trying to relearn as much about Java as possible.
Comment: The main method is itself static. Action listener methods are not.
Comment: @SilvioMayolo So is this to say in Java that static methods can only access static (or local) variables?
Comment: A side note, but the term "class variable" is used to refer to static fields. There is no such thing as a non-static class variable. I think you're looking for the word "field" or "attribute" or "property."
Comment: @Zar Good to know. I emphasized the word class because I didn't want it to get confused with a local (method level) variable.
Answer: Your main method is static, which means it isn't run on an instance of an object, the method is called on the class without having an object of the class instantiated. This makes a certain amount of sense for a main method because it's the first thing your program does so there's no way to instantiate an object.
Static methods and variables exist on the class itself, while instance members are (non-static) variables that are part of each object created from a class. Each object has its own copy of the instance variables declared for that class, giving each object its own state. Since the static variables are on the class, and there is only one copy of the class (it's loaded only once by the classloader), there is only one of each static variable.
I use "class variable" to refer to static variables, and "instance variable" to refer to non-static variables.
In Swing GUI programming there is very little reason to make use of static variables, my advice is to avoid them if at all possible, because having global mutable state is ugly and hard to debug. In Swing all the UI components and listeners are objects, once your main method initializes and displays a main jframe no other static methods or variables are usually needed.
Comment: Hmm. So I guess as a rule of thumb I should avoid loading important variables in the main method, that way I won't have to declare them as static?
Comment: @user3303680: in swing programs as i've written them typically the main method only creates a window and displays it. hard to give more advice without seeing code.
Comment: Without going into too much detail I put the file I/O code in the main method, before the window is created. This was to load my fictional store's inventory into array from a textfile.
Comment: @user3303680: that makes sense. you could alternatively pass the inventory that you read in into the jframe subclass object, or have the jframe do the work itself when it is being initialized. but the point is that static means class-level, and non-static members belong to specific instances.

Static/Class Variables and GUI


Mathematical implications of Naive Bayes Classifier on imbalanced data set

Question: Can someone explain what problems occur mathematically when providing a highly imbalanced training data set to a Naive Bayes Classifier?
Whats the mathematical reason for a resulting poor performance due to the imbalanced training data?
Let's say a training data set of 99 samples of negative label and 1 sample of positive label.
Answer: Naive Bayes theorem assumes independence and used Bayes theorem to calculate the probabilities
$$
p(C_k, x_1, x_2, \dots, x_m) = p(C_k) \, \prod_{j=1}^m p(x_j \mid C_k)
$$
where we use the empirical estimates of the probabilities, e.g.
$$
p(C_k) = \frac{\text{the number of samples with class }C_k}{\text{total number of samples}}
$$
So if for some class you have small number of samples, then the estimates for this class would be imprecise. Imagine that your data is the weather (sunny vs cloudy) and you want to predict if Joe is going to eat an ice cream on a given day. The data you have is shown below.
<code>weather | ice cream
--------|----------
sunny   | no
cloudy  | yes
sunny   | yes
sunny   | yes
sunny   | yes
</code>
Now if you were going to predict what is the probability that Joe will eat ice cream on cloudy weather, it would be $p(x|y) \, p(y) = 0/1 \times 4/5 = 0$ just because you had only one sample, so your algorithm assumes anything else then it saw as "impossible". To correct for those, we would usually use Laplace smoothing and replace $0$ with some small, arbitrary value, so instead of zeroes we would end up with some numbers, but they still would not give us a precise result. This is an extreme case, but the general problem is that with small samples you get imprecise estimates.
The problem is not imbalanced data, but small number of samples. With imbalanced data, the algorithm gives bad predictions for the smaller class, but if you had little samples for both classes, it would be equally bad for both. Unbalanced data is generally not a problem, but insufficient data is.
Another problem may be is the proportion of class $C_k$ in your data differs from the proportion in the population (say you have 20% of females in your data, while in the population the fraction is closer to 50%), but then, to fix for that, just replace $p(C_k)$ with the true proportions (i.e. 0.5 rather then 0.2 in this example). This is possible because the conditional probabilities $p(x_j|C_k)$ are in each case calculated within the classes (to calculate $p(x_j|C_k)$ you count how often $x_j$ appeared for samples with $C_k$ class), while the "correction" for the class size is done by multiplying with $p(C_k)$ (check nice, worked example of how Bayes theorem works for more details). The class size does not enter the equation otherwise.
Moreover, you need to remember that naive Bayes gives you imprecise estimates of probability by design, so they would never be precise.
Comment: Hey Tim, many thanks for your comprehensive response but as you said you are describing the problem of "insufficient data". I was looking for reasons why NB could suffer from imbalanced data set. Do you think NB doesn't suffer from just imbalanced class distributions at all? Hence, it would not make sense to apply random over-sampling whereas SMOTE-sampling could help due to additional samples tacking "insufficient data".
Comment: @SamSampleman as said above, the problem might be not enough data, but oversampling does not solve it, because it would only repeat some samples, so you don't gain new information by this. If the problem is that the distributions differ in your data vs population, then you only need to use the population $p(C_k)$ probabilities, instead of those in your data to correct it, as this is the *only* place where the relative sizes of the classes are considered by the algorithm.
Comment: Okay, apparently my question is still not quite clear. I'm totally aware about the "insufficient data" problem you have explained comprehensively. Let me ask you an additional question and please do not refer to "insufficient data" or to different distributions among data and population. Does NB suffer from highly imbalanced class distribution at all (assume same imbalance within the sample and population)? Consider problems of other algorithms when applied on imbalanced training data (e.g. neural networks).
Comment: @SamSampleman the only problems, and solutions, are mentioned above. Also keep in mind that imbalanced data is a [non-issue in many cases](https://stats.stackexchange.com/questions/283170/when-is-unbalanced-data-really-a-problem-in-machine-learning), there's no "imbalanced data problem" in general, just in *some* cases some additional care may be needed.
Comment: @SamSampleman see also my edit.


How well the experiment would do in excluding alternative hypothesis?

Question: The experiment has $N$ observations $N_i$ which are Poisson Distributed random variables (i.e. we have histogram with $N$ bins. The width of bins is not fixed so I can adjust it  ) . I have two hypothesis.The null hypothesis predicts $\lambda_i^{null}$ values of corresponding parameters of Poisson Distribution, alternative  predicts $\lambda_i^{alt}$. The more data we collect, the better confidence level of excluding of alternative hypothesis we get. But how to calculate this confidence level as a function of size of data we collect? In other words I would like to estimate how many data should I collect to exclude alternative hypothesis at specified confidence level.
Firstly, I thought about using chi-squared test and calculate the expected value of chi-square i.e. $\left<\chi^2\right>$. But when I tried different width of bins to obtain the optimal binning, I discovered that the less bins we use the better result we get. So I think it's not the optimal test for this task because we don't fully use information from predictions when we have just a few bins.
I would like to know which tests are more appropriate for such problems.
I would also be grateful for any kind of advice regarding the literature on this subject.
P.S. I've asked the same question at physics.stackexchange, but with some physical details.
Comment: @vitalik: So you have observations $N_1, N_2,\ldots,N_N$ which are independent and Poisson distributed? Does your model predict that you have different $\lambda_i$ for the different channels, i.e. $\lambda_1,\ldots,\lambda_N$? Why do you want to use histograms to describe the theoretical distributions? They are used for continuous data and not for theoretical descriptions of discrete random variables...
Comment: If you only have $N$ possible outcomes then your random variable can't be Poisson distributed. Or are you saying that you have several observations from a Poisson r.v. and that you summarize those observations in a histogram? If so, do you still have the original measurements or do you only have the summarizing histogram?
Comment: @MansT Sorry, probably I used wrong word. I wanted to say that I have N channels. Each channel gives us random variable $N_i$ and we have $N$ independent Poisson Distributed random variables. I can calculate theoretical predictions which are histograms and I can make bins as thin as I want in principle.
Comment: Do you know the values of the $\lambda_i^{null}$ and $\lambda_i^{alt}$?
Comment: @guest Yes, they are theoretical predictions and I can calculate them.
Comment: Calculate them based on this data - or some external source?
Answer: If the $\lambda$ values are smooth and ordered, the chi-square test has very low power for this situation, and as a result the optimal number of bins for the chi-square tends to be surprisingly small; there are numerous papers on this if you want to follow it up.
There are tests with better power.
One approach would be to look at Smooth Tests of Goodness of Fit (see, for example, the book by that title by Rayner and Best); these work by constructing orthogonal polynomials (usually in the log-probability or log-density) with respect to a  probability function or density function as a weight function. (Though it can be done unweighted, as with the original versions of this test, and if there's no suitable set of orthogonal polynomials for the given model, this may be a good choice.)
Incidentally, in the case of a chi-squared test (on all bins), the unweighted smooth test corresponds to partitioning that chi-square into orthogonal components corresponding to the coefficients on the orthogonal polynomials and then taking the lowest-order components of that (yielding a $\chi^2_4$, say, rather than a $\chi^2_{N-1}$).
This can deal with estimation of parameters (e.g. in the case of say the normal, it can encompass estimated mean and variance by omitting the first two orthogonal components).
There's also many many papers on this topic. One reasonable starting point is this:
J. C. W. Rayner and D. J. Best (1990),
"Smooth Tests of Goodness of Fit: An Overview"
International Statistical Review
Vol. 58, No. 1 (Apr.), pp. 9-17


Push A WinForm To The Back?

Question: I have a form that I want to push behind all other windows when a certain operation runs.
Is this possible?
Comment: Why on earth to you want do do that?
Answer: Form.SendToBack() is what you're looking for.

Push A WinForm To The Back?


I want to hide data table excel button on print

Question: I have excel data table button I want to hide in print page.

I using @media Print for hide button but it not working.
Comment: If your question is about buttons with class `buttons-excel`, you should show us those buttons.
Comment: Possible duplicate of [Hide buttons when printing](https://stackoverflow.com/questions/31425557/hide-buttons-when-printing)
Comment: @MrLister `buttons: [
          'excel'
      ] `  this excel button  I want to hide in print page
Answer: Remove or modify the media attribute


Specifying the survey design correctly in the R survey package

Question: I have the results from a survey where a geographic area was stratified and power allocation (using the number of households in each stratum) was used to determine the number of surveys to send out to each stratum.
From each household that returned a survey (which has a unique survey ID), I have the weight results from 1 or more people in the household.  Thus in some instances a single survey from a single household has 5 responses to the weight question.  I convert the weight from each response into one of four weight classes.
I want to obtain a point estimate for the proportion of the population who falls into each weight class along with the associated confidence interval, based on the age group of the respondent (Child versus Adult).  To do so, I have placed the data from the surveys into a data.frame with the following columns:

SurveyID: the ID of the survey that the response came from
Stratum: identifies the Stratum that the survey was returned from
PostStratWeights: the post-stratified weight that accounts for survey non-response
AgeGroup: the age group that the response belongs to (Child/Adult)
WeightClass: the weight class of the respondent (one of four classes)

where each row in the data.frame corresponds to a single response from a survey (so multiple rows will have the same SurveyID if there was more than one weight response from a single survey).
I have specified the survey design as
mydesign = svydesign(ids=~SurveyID,
                        strata=~Stratum,
                        weights=~PostStratWeights,
                        data=survey_response_data)

Do I need to add in fpc for this survey design? I know both the estimated number of households in each stratum and the estimated number of people in each stratum.
Under the assumption that the survey design was correct, I then tried to compute the mean WeightInLbs as follows:
svyby(formula=~WeightClass,
        by=~AgeGroup+Stratum,
        FUN=svymean,
        design=mydesign,
        keep.names=FALSE)

However, running this produces the following error:
Error in `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]]) :
  contrasts can be applied only to factors with 2 or more levels

So I suspect something is wrong with the way I have specified my survey design but don't quite know what.
Comment: Those look correct, so I think we need some information about your data
Comment: What information would you like about the data?  I will edit the information into the original post.
Comment: I have edited the post because I actually break the WeightInLbs into four classes, and so I am using the WeightClass column (a categorical variable) instead of the WeightInLbs column, a continuous variable.  I want to get point estimates for the proportion of the population in each weight class.  Sorry for having the wrong question in there initially.
Answer: Ok. It sounds as if svymean() is converting ~WeightInLbs to a factor variable (maybe it starts as character?), which will happen separately for each combination of the by variables, and that some combination of the by variables has only one weight category present.
If this is what's happening, you can create the factor variable in advance using update
mydesign<-update(mydesign, WeightClassFactor = factor(WeightClass))
svyby(formula=~WeightClassFactor,
        by=~AgeGroup+Stratum,
        FUN=svymean,
        design=mydesign,
        keep.names=FALSE)

Comment: This must have been what was happening because creating the factor variable in advance solved the issue.  I am still a little unsure regarding whether I need to add in the fpc for this design - any thoughts on that?