Interpreting coefficients table in ordinal regression

Question: Can we interpret the coefficients of an ordinal regression in the coefficients table without involving the intercepts?
For example
library(ordinal)

> head(wine)
  response rating temp contact bottle judge
1       36      2 cold      no      1     1
2       48      3 cold      no      2     1
3       47      3 cold     yes      3     1
4       67      4 cold     yes      4     1
5       77      4 warm      no      5     1
6       60      4 warm      no      6     1

fm <- clm(rating ~ temp * contact, data=wine)
summary(fm)

Coefficients:
                    Estimate Std. Error z value Pr(>|z|)
tempwarm              2.3212     0.7009   3.311 0.000928 ***
contactyes            1.3475     0.6604   2.041 0.041300 *
tempwarm:contactyes   0.3595     0.9238   0.389 0.697129
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Threshold coefficients:
    Estimate Std. Error z value
1|2  -1.4113     0.5454  -2.588
2|3   1.1436     0.5097   2.244
3|4   3.3771     0.6382   5.292
4|5   4.9420     0.7509   6.581

What does the contactyes coefficient mean on its own?
EDIT:
Here are the plots for the probabilities per rating level
Comment: @PeterFlom, temp is a factor with two levels, cold and warm (see edit).
Comment: How did you go from "temp" in the model statiemnt to "tempwarm" in the coefficients?
Comment: @PeterFlom Treatment contrasts. `temp` is a factor variable with "warm" as the second level. (I suspect the first level could be "cold".)
Comment: Oh. OK. And the output unhelpfully doesn't put a space between the name of the factor and the level.  That's R for you.  :-)
Answer: The contactyes coefficient is the odds ratio for contact for each level of rating when temp = 0. (Or tempwarm. See my comment, something odd there).
Comment: An introduction to the proportional odds semiparametric ordinal logistic model is [here](https://hbiostat.org/bbr/nonpar#generalization-of-the-wilcoxonkruskal-wallis-test) which shows how to interpret the coefficient in some detail.
Comment: There are many ways to display the results of an ordinal model, including using it to estimate mean of Y | X and nomograms to estimate all the exceedance probabilities as well as the mean and quantiles of Y.  See several examples in [RMS](https://hbiostat.org/rmsc) and [BBR](https://hbiostat.org/bbr).
Comment: @locus replace 'more likely' with '1.34 times higher odds'.
Comment: Thanks for your answer. Can I interpret `contactyes` like this?: `warmer` wines with `contact=yes` are more likely to receive a higher rating than `colder` wines with `contact=no`. Is that correct? I'm struggling because in linear regression we have a single intercept and single equation, but here we have 4, one for `1|2`, `2|3`, `3|4` and `4|5`, so I'm not sure if my interpretation makes sense.
Comment: Or maybe more precisely: `warmer` wines with a `contact=yes` increase the odds of a higher rating relative to `colder` wines with `contact=no` by 1.34. The "higher rating" part is what confuses me, since it seems so unspecific...
Comment: "Higher rating" is between any two levels, because ordinal logistic assumes proportional odds.

"contactyes" by itself refers to the case when temp = cool. When temp = warm, you need to also add in the interaction term. Since that is positive, the effect of temperature is greater when contact = yes.
Comment: Ah, yes, I messed up. The interpretation of `contactyes` should be that colder wines with contact=yes are 1.34 more likely to receive a higher rating than colder wines with contact=no, correct?
Comment: I think it's the "between any two levels" that I don't seem to be able to understand. At first I thought this must mean that the increase in odds ratio of 1.34 for colder wines with contact=yes relative to colder wines with contact=no is the same between Rating 1 and Rating 2, between Rating 2 and Rating 3, and so on. But that cannot be right, as these plots show: `library(ggeffects); plot(ggpredict(fm, terms=c("temp","contact")))`
Comment: I can't see the plots, just the code.  You may be able to add the actual plots to your question.
Comment: @PeterFlom, edited the question with the plots. From these plots, I can see that the difference between contact=yes and contact=no for colder wines is very different among Ratings.
It's still very unclear to me what the coefficient `contactyes` is telling me, and how it relates to those plots.
Comment: I think I may be incorrect, see @FrankHarrell 's link in his comment.
Comment: @FrankHarrell, thanks for your comment. OK, so the interpretation of `contactyes` coefficient is that colder wines with contact=yes have 1.34 times higher odds to receive a higher rating than colder wines with contact=no.
Comment: My problem now is where do the different rating levels fit here? If this were multiple regression, it would be easy for me to visualize a line passing through different values of `y`. But with this ordinal model, I just can't see how the coefficient `contactyes` relates to the different ratings.


Scala Map: Combine keys with the same value?

Question: Suppose I have a Map like
val x = Map(1 -> List("a", "b"), 2 -> List("a"),
            3 -> List("a", "b"), 4 -> List("a"),
            5 -> List("c"))

How would I create from this a new Map where the keys are Lists of keys from x having the same value, e.g., how can I implement
def someFunction(m: Map[Int, List[String]]): Map[List[Int], List[String]] =
  // stuff that would turn x into
  // Map(List(1, 3) -> List("a", "b"), List(2, 4) -> List("a"), List(5) -> List("c"))

?
Answer: You can convert the Map to a List and then use groupBy to aggregate the first element of each tuple:
x.toList.groupBy(_._2).mapValues(_.map(_._1)).map{ case (x, y) => (y, x) }

// res37: scala.collection.immutable.Map[List[Int],List[String]] =
//        Map(List(2, 4) -> List(a), List(1, 3) -> List(a, b), List(5) -> List(c))

Or as @Dylan commented, use _.swap to switch the tuples' elements:
x.toList.groupBy(_._2).mapValues(_.map(_._1)).map(_.swap)

Comment: You beat me by a few seconds with pretty much the exact same solution, so I'll give you the +1 and add that you can use `_.swap` in the final `map` call.
Comment: @Dylan Ah, didn't know the `_.swap` method before. Thank you!


Random timeouts in IIS7

Question: I have a weird problem which I think is caused by my IIS7 installation on Vista 64 bit.
I have a bunch of AJAX calls and JS dynamic file loads (~30) in a local application, and I get random timeouts (or so it seems) in my browser.
In Chrome it looks like the page just stops loading (no HDD activity), in Firefox/Firebug I can see that some of the files are being loaded but they never actually finish. When I reload the page the same occurs, but for (random) other files that must be loaded.
When I try to load one of those JS files concurrently (so during the timeout in FF) in another browser, the file loads there. So I am pretty sure the request can be handled by IIS.
I am thinking about a limit or so on simultaneous requests from the same browser which is not working correctly, but I am pretty clueless on how to solve this.
Does anyone recognize this problem and know a solution?
Thanks!
Answer: I could be wrong but I think IIS in Vista has a limit to the number of concurrent connections. http://www.lockergnome.com/windows/2006/12/04/install-iis-in-vista/
Have you tried hosting your app on Srv2008
Comment: There indeed is, but this should not be a problem because requests should be queued. I know the application works OK on other, non-server OSs (vista, win7 etc)
Comment: Requests aren't always queued, depending on your app, files will be downloaded at the same time.

Hope you get it sorted
Comment: No but IIS should queue the requests (differs from XP in which they would simply give a 'server busy' response)

Random timeouts in IIS7


colorbox does not seem to work anymore after upgrade from 7.x-1.6 to 7.x-2.9

Question: I upgraded colorbox from 7.x-1.6 to 7.x-2.9 and now, I lost functionality of loading the pictures in a box. my page is http://www.quaaoutlodge.com/content/lodge and the centered picture in the middle, used to load nicely in a colorbox frame before the update but this does not work anymore now. The link looks like:

The link is being "auto-created" as follows:
In my Content type, I have added a Field myImage that allows me to upload and insert the image into the content. The Format of the field, I set to Colorbox. I setup my image styles as follows:
Content image style: Hide
Colorbox image style: Original image
Colorbox gallery type: Per page gallery
Colorbox caption: None

And don't understand how this isn't working anymore after the upgrade...
Whaty's wrong with this I'm wondering? Should I rollback to 7.x-1.6 or can I fix this?
EDIT A
The colorbox module is installed & enabled and it says A light-weight, customizable lightbox plugin for jQuery 1.4.3+. Looks like I'm using jQuery 1.4.4:
EDIT B
I just copied modules/colorbox/ to libraries/colorbox/ but that didn't change much. You say the "JS lib" is not included on the page? I have

and

in my source code, the second line refers to jQuery 1.8
Comment: I have a feeling that the module itself is not available at the moment, can you check that? also make sure you are using jquery version 1.8 at least!
Comment: please see **EDIT A** above
Answer: i think you need to update the colorbox library too !
right now - the JS lib is not included in the page ...
Comment: no do not copy the module to libraries ! revert that pls. take a fresh copy from http://www.jacklmoore.com/colorbox/ and extract it to the libraries folder, actually /admin/reports/status will give you hints here.
Comment: Hi remy, please sese **EDIT B** above
Comment: yep, that was it! I downloaded the `colorbox-master.zip` extracted the archive to `/sites/all/libraries`, renamed it to `colorbox` and cleared the caches. Thanks!


calculate the two dimensional integral $\int_B x^2-y^2$

Question: I'm trying to calculate the two dimensional integral \begin{equation*}\int_B x^2-y^2\end{equation*} Where $B$ is the area bounded by \begin{equation*}xy=1,y=0,x=y,x^2-y^2=1\end{equation*}
I have been suggested to try and substitute $u=x^2-y^2,v=xy$ but I'm having a hard time finding the set on which I should parametrise.
Can anyone clear this up for me?
Answer: $u = x^2-y^2, v = xy$. The conditions $xy = 1 \to v = 1, y = 0 \to xy = 0 \to v = 0, x = y \to x^2 = y^2 \to x^2-y^2 = 0 \to u = 0$. Thus you integrate over $B': u = 0, u = 1, v = 0, v = 1 \to I =\displaystyle \int_{B'} u\dfrac{\partial(x,y)}{\partial(u,v)}dudv$. Don't forget to calculuate the "Jacobian"...
Comment: Is $dxdy$ equal to $dudv$?


Non-unital commutative semigroups $S$ such that for all $x \in S$, the function $S \rightarrow S$ given by $y \mapsto x+y$ is a bijection?

Question: Does there exist a commutative semigroup $S$ with the following (additively denoted) properties?

For all $x \in S$, the function $S \rightarrow S$ given by $y \mapsto x+y$ is a bijection.
$S$ has no identity element; that is, there is no $0 \in S$ such that for all $x \in S$, $x+0=x.$
$S$ is non-empty. (This is to rule out the trivial example of $S=\emptyset$, thanks to Najib Idrissi for pointing this out.)

Of course, if 2 is replaced by its negation (namely, that there is an identity element), then examples are easy to find; in fact, these are precisely the Abelian groups.
Comment: I'm pretty sure you meant $x+0=x$ in condition 2, too.
Answer: Assume $S$ is nonempty, containing the element $x$, and is such that addition of any element gives a bijection. Then since addition of $x$ is a bijection, there exists a $y$ such that $x+y=x$. We claim that $y$ is an identity element. Let $z\in S$. Then, since addition of $x$ is a bijection, there exists a $w$ such that $z=w+x$. But then $z+y=(w+x)+y=w+(x+y)=w+x=z$. Thus, $S$ does indeed have a $0$, and is therefore an abelian group (inverses exist by the bijectivity of addition).
Comment: Right. So to show that there exists a $0$ element, all we need is a commutative semigroup $S$ together with an $x \in S$ such that $x \mapsto x+y$ is a surjection. Now that's cool.
Comment: Yes, addition does not even have to be injective, though if it is surjective, my proof shows that it makes $S$ into an abelian group and is thus injective.
Comment: I like it. $\;\!$
Answer: No. Let $S \neq \emptyset$ be as in your question. Let $a \in S$ be arbitrary, $T_a : S \to S$ be defined by $T_a(x) = a + x$ and let $0 = T_a^{-1}(a)$. Then for all $x \in S$, $$T_a(x+0) = a + x + 0 = x + (a+0) = x + T_a(T_a^{-1}(a)) = x + a = T_a(x)$$
Since $T_a$ is a bijection, $x+0=x$, and this is for any $x \in S$.


ManyToOne ClassCastException

Question: I am trying to create a "simple" OneToMany and ManyToOne relation.
And I get following Exception:

Caused by: java.lang.ClassCastException:
  org.hibernate.mapping.ManyToOne cannot be cast to
  org.hibernate.mapping.Component   at
  org.hibernate.mapping.PersistentClass.getRecursiveProperty(PersistentClass.java:464)
  ~[hibernate-core-5.0.9.Final.jar:5.0.9.Final]     at
  org.hibernate.mapping.PersistentClass.getRecursiveProperty(PersistentClass.java:420)
  ~[hibernate-core-5.0.9.Final.jar:5.0.9.Final]     at
  org.hibernate.cfg.annotations.CollectionBinder.bindStarToManySecondPass(CollectionBinder.java:758)
  ~[hibernate-core-5.0.9.Final.jar:5.0.9.Final]     at
  org.hibernate.cfg.annotations.CollectionBinder$1.secondPass(CollectionBinder.java:719)
  ~[hibernate-core-5.0.9.Final.jar:5.0.9.Final]     at
  org.hibernate.cfg.CollectionSecondPass.doSecondPass(CollectionSecondPass.java:54)
  ~[hibernate-core-5.0.9.Final.jar:5.0.9.Final]     at
  org.hibernate.boot.internal.InFlightMetadataCollectorImpl.processSecondPasses(InFlightMetadataCollectorImpl.java:1655)
  ~[hibernate-core-5.0.9.Final.jar:5.0.9.Final]     at
  org.hibernate.boot.internal.InFlightMetadataCollectorImpl.processSecondPasses(InFlightMetadataCollectorImpl.java:1623)
  ~[hibernate-core-5.0.9.Final.jar:5.0.9.Final]     at
  org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:278)
  ~[hibernate-core-5.0.9.Final.jar:5.0.9.Final]     at
  org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:847)
  ~[hibernate-entitymanager-5.0.9.Final.jar:5.0.9.Final]    at
  org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:874)
  ~[hibernate-entitymanager-5.0.9.Final.jar:5.0.9.Final]    at
  org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:60)
  ~[spring-orm-4.3.2.RELEASE.jar:4.3.2.RELEASE]     at
  org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:338)
  ~[spring-orm-4.3.2.RELEASE.jar:4.3.2.RELEASE]     at
  org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:373)
  ~[spring-orm-4.3.2.RELEASE.jar:4.3.2.RELEASE]     at
  org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:362)
  ~[spring-orm-4.3.2.RELEASE.jar:4.3.2.RELEASE]     at
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637)
  ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]   at
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
  ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]   ... 39 common frames
  omitted

Is someone able to tell me what I am doing wrong?
Below you can find my two entities.
Thank you in advanced.
  import javax.persistence.OneToMany;
@Entity
@Table(uniqueConstraints={@UniqueConstraint(columnNames={"questionGroupID","questionID"})})
public class QuestionGroupEntity implements Serializable{

    private static final long serialVersionUID = 1796640204447018439L;

    @Id
    @NotEmpty
    @Column(name="question_group_id")
    private String questionGroupID;

    @NotEmpty
    private String label;

    @NotEmpty
    private String questionID;

    @OneToMany(fetch=FetchType.EAGER,targetEntity=QuestionGroupMappingEntity.class,mappedBy="questionGroup")
    List<QuestionGroupMappingEntity> questionGroupMappings;

AND
import javax.persistence.ManyToOne;
    @Entity
    @Table(uniqueConstraints={@UniqueConstraint(columnNames={"questionGroupID","questionID","answerID"})})
    public class QuestionGroupMappingEntity implements Serializable {

        private static final long serialVersionUID = 8437546139229082305L;

        @Id
        @SequenceGenerator(name="groupMappingIDGenerator")
        @GeneratedValue(generator="groupMappingIDGenerator",strategy=GenerationType.AUTO)
        private String groupMappingID;

        @NotEmpty
        private String questionID;

        @NotEmpty
        private String questionGroupID;

        @NotEmpty
        private String answerID;

        @ManyToOne(fetch=FetchType.EAGER)
        @JoinColumn(name="question_group_id")
        private QuestionGroupEntity questionGroup;

Comment: have you defined the getters and setters? On what call are you getting the error?
Answer: Thank you for your answers.
I solved this issue by adding those properties
spring.jpa.hibernate.ddl-auto=update
spring.jpa.generate-ddl=true
The problem was that spring-data was not able to update the schema.
Answer: Please check your QuestionGroupEntity entity, the config the private String questionGroupID; with the column name "question_group_id", but you still add a @UniqueConstraint(columnNames={"questionGroupID","questionID"})} with the "questionGroupID" column, which will occue the exception

Caused by: org.hibernate.AnnotationException: Unable to create unique key constraint (questionGroupID, questionID) on table question_group_entity: database column 'questionGroupID' not found. Make sure that you use the correct column name which depends on the naming strategy in use (it may not be the same as the property name in the entity, especially for relational types)
      at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.buildUniqueKeyFromColumnNames(InFlightMetadataCollectorImpl.java:2090)
      at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.buildUniqueKeyFromColumnNames(InFlightMetadataCollectorImpl.java:1965)
      at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.processUniqueConstraintHolders(InFlightMetadataCollectorImpl.java:1953)
      at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.processSecondPasses(InFlightMetadataCollectorImpl.java:1629)
      at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:278)
      at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:847)
      at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:874)

After fix this issue, it ok with spring boot 1.4
Comment: Event if I remove all unique constraints I still got this error
Comment: By the way this was not the exception I got.

ManyToOne ClassCastException


Simulated annealing - why when encountering a lower energy state we move to it with probability 1?

Question: I'm doing simulated annealing, following a recipe I have -

Compute the energy of current state and a neighbor state.
If the energy of the neighbor is lower - move to neighbor.
If not - move to neighbor with probability $e^{-\Delta/\tau}$ where $\Delta$ is the difference between new energy and old energy.

This seems a bit arbitrary to me. Why not compute the exponent for both the current state and the neighbor, and then just move/stay with these probabilities? This means using a sigmoid, which is derived by dividing the exponent of one of the options with the sum exponents of the two: So the probability to move is $e^{-E1/\tau} / e^{-E1/\tau} + e^{-E2/\tau}$ which can also be written $1 / e^{-\Delta/\tau}$
Comment: You do not describe [simulated annealing](http://mathworld.wolfram.com/SimulatedAnnealing.html). The correct algorithm moves to *higher* energy states with certain small probabilities (which get smaller as the "temperature" decreases). Your description is merely a slowed-down version of a randomized gradient descent.
Comment: In my description - it does move to higher energy states with certain small probabilities, the issue here is whether or not it can stay even if the neighbor energy is lower.
Comment: Please, then, change your title, which directly contradicts what you just said.
Comment: It does not contradict anything. When encountering a lower energy state, the recipe states that the system will move to it. But I can see the confusion so I'll rephrase.
Answer: This is not a complete explanation but probably better than none:
In short - The original SA recipe is like so because of the proof of convergence.
In more details - the SA process is a markov process. This process has to be a reversible one for the proof that shows the algorithm converges (this has a clear definition and is equivalent to satisfying something called the detailed balance equations http://en.wikipedia.org/wiki/Detailed_balance#Reversible_Markov_chains).
In order to satisfy this condition the probability to move between states was chosen as in the recipe above.
Some more intuition as to why the alternative I'm offering, the sigmoid, is not right here: The partition function (the normalization element in the equation) is not the sum over the two neighboring states, but instead the sum of all states and their energy values - which is usually incalculable).


Epic Online Services errors.com.epicgames.common.policy_missing_action

Question: been experimenting with UE5 and EOS
I havent filled in the branding section completely in the dev portal for EOS (in draft) as I dont have a policy or game logo
I have the game logging in fine (using accountportal) and that returns success and I get the overlay, so no issues, but when I try and create a session it fails with the below Error
[2022.05.07-10.50.43:574][724]LogEOSSDK: Warning: LogEOS: Error response received from backend. ServiceName=[Sessions], OperationName=[CreateSession], Url=[], HttpStatus=[403], ErrorCode=[errors.com.epicgames.common.policy_missing_action], NumericErrorCode=[UNSET], ErrorMessage=[Client  missing matchmaking:createSession], CorrId=[EOS-Cq8EcZDt1UCJ_3H0vBAlMw-P4g0uuJPKUeOfcjLzhEBIQ-p7QMqQjV-EyAynCi_1iBjw]
If I set the "isLan" option it works fine no issues, have I missed something in the setup to stop it working online or is it because the branding is in draft and not verified that it only allows Lan?
Answer: This was potentially due to the online portal client policy, was pretty sure I had it so the client to create/host sessions but i went through it all again and now it seems to work


How to make a ROC curve for multiple parameters/thresholds

Question: I have a prediction model that performs a binary classification. The model takes 3 independent predictor parameters variables that can only be integers. I have calculated true positive rates (TPR) and false positive rates for each combination of three predictors. Each dot in the figure below represents TPR (y-axis) vs FPR (x-axis) of each combination.

Since there are more than 1 parameter, I can not apply a conventional ROC curve, however is there a way to represent success of a model with multiple parameters? Or would it still be reasonable to construct ROC for best and worst case scenarios, such as the figure below?
Comment: I think you need to edit this to clarify. What do you mean by an integer parameter? Do you mean predictor variables which can only take on integer values? What do the dots represent? Why are there different numbers of dots in each column, or is that just overprinting?
Comment: It is not overprinting, they are just happened as a consequence of multiple predictor parameters. I edited the rest as you suggested, I hope it is clearer.
Comment: Im also confused as to how you drew your curve. An roc curve is parameterized by varying the classification threshold in your model. Is this what you did? Why does the x axis stop short of 1?
Comment: I only drew lines over best tpr vs fpr scores. :D I drew it in paint. it stop short of 1 cuz I didnt bother drawing anymore.
Answer: You are showing individual ROC curves for each predictor, but I assume you have a multivariate model (e.g., logistic regression). What you should be showing is the ROC curve from the final model, using the predicted probability values obtained after applying the model to validation data.
Answer: The main problem with using multiple variables in the way you suggest is that it's undefined: let's say you have two variables, so your ROC would use 2 thresholds T_A and T_B. If at some threshold combination (e.g. T_A = 1 and T_B = 1) an observation is positive according to T_A but negative according to T_B, what is the prediction? Combine using logical AND? Logical OR? Round the average?
Let's say you decide to combine the variables using the logical AND method. You can now sample 2D threshold space and create a scatterplot-like ROC. But there are issues with this, read on.
A normal 1-variable ROC samples 1D theshold space. A consequence of this is that a positive at threshold T1 can not be a negative at a threshold T2, where T2 < T. I.e. the ROC is monotonously increasing.
The issue with sampling 2+D threshold space is the curve can go up, down, left or right from a given state, and you can't unambiguously connect the dots with a line. This ROC can have weird shapes that can't really be described by an auc value, and may have multiple optimal threshold combinations.
This is why it is more common to fit the variables with a (logistic?) model, that basically boils everything down into a single 'prediction probability' that you can treat like a 1D ROC variable.


Why is it saying this is not a json object?

Question: i am new to json parsing and i'm trying to figure out why it's returning null.
Here is my java code (if you aren't familiar with the spigot api https://hub.spigotmc.org/javadocs/spigot/overview-summary.html)
Can you tell me what i am doing wrong? i'll give the gson part of the code and then i'll give the rest. think of it as just outputting the json in a console if you don't feel like reading the api.
try {
                    URL hypixel = new URL("https://api.hypixel.net/player?key=apikey&name=" + username);
                    URLConnection urlConn = hypixel.openConnection();
                    urlConn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:18.104.22.168) Gecko/20100316 Firefox/3.6.2");
                    urlConn.getDoOutput();

                    try(final BufferedReader reader = new BufferedReader(new InputStreamReader(urlConn.getInputStream()))) {
                        final JsonParser parser = new JsonParser();

                        parser.parse(reader.readLine());

                        final JsonObject object = parser.parse("").getAsJsonObject();

                        String userId = object.getAsJsonObject("player").get("_id").getAsString();

                        p.sendMessage(ChatColor.GREEN + "UID: " + userId);

                }

                } catch (IOException e) {
                    p.sendMessage(ChatColor.RED + "Something went wrong!");
                }

(p.sendmessage would be the thing going in console)
Here is all of the code:
@Override
    public boolean onCommand(CommandSender sender, Command command, String label, String[] args) {

        Player p = (Player) sender;

        if(command.getName().equalsIgnoreCase("hypixel")) {

            if(args.length == 2) {
            String username = args[0];

                try {
                    URL hypixel = new URL("https://api.hypixel.net/player?key=apikey&name=" + username);
                    URLConnection urlConn = hypixel.openConnection();
                    urlConn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:184.108.40.206) Gecko/20100316 Firefox/3.6.2");
                    urlConn.getDoOutput();

                    try(final BufferedReader reader = new BufferedReader(new InputStreamReader(urlConn.getInputStream()))) {
                        final JsonParser parser = new JsonParser();

                        parser.parse(reader.readLine());

                        final JsonObject object = parser.parse("").getAsJsonObject();

                        String userId = object.getAsJsonObject("player").get("_id").getAsString();

                        p.sendMessage(ChatColor.GREEN + "UID: " + userId);

                }

                } catch (IOException e) {
                    p.sendMessage(ChatColor.RED + "Something went wrong!");
                }

                }

        }

        return false;
    }

Any help is appreciated thank you!
(Oh and here is the part of the response from the api that i want to parse)
{"success":true,"player":{"_id":"5442f08f48b8f1e1e64a0400"}}

Answer: parser.parse("").getAsJsonObject()

is expecting "" to be JSON, which it is not.


Calling default done function in jquery file upload

Question: I am using the  Jquery file upload plugin for uploading the file.
<code>    jQuery('#UploadForm-form').fileupload({
'maxFileSize':20000000,
'acceptFileTypes':/(\.|\/)(csv)$/i,
'done':function(e,data)
       {
        // do something here
       // call defaultDone here
       },
'url':'/manage/upload?parent_id=1',
'autoUpload':false,
'maxNumberOfFiles':1});
</code>
The problem is that in the done function above, I want to do something and then call the default done function provided by the plugin.
How can I do that ?
Comment: What is the exact requirement to do it?
Comment: @ManishJangirBlogaddition.com The EXACT requirement is that I have to set the file name in a hidden field and then show the downloadTemplate as it shows by default. I guess you are the one who has downvoted. May I know why ?
Answer: You have to memoize your default function. Hopefully you will get some idea from following code.
<code>// Initialize your file upload object
jQuery('#UploadForm-form').fileupload({
    'maxFileSize' : 20000000,
    'acceptFileTypes' : /(\.|\/)(csv)$/i,
    // Don't override default done function here
    // You have to memoize default function from widget instance later.
    // Hence, I comment following line
    // 'done':function(e,data)
    // {
    // // do something here
    // // call defaultDone here
    // },
    'url' : '/manage/upload?parent_id=1',
    'autoUpload' : false,
    'maxNumberOfFiles' : 1
});
// Memoize and call the default done as follows.
var widgetInstance = $('#UploadForm-form').data('blueimp-fileupload') || $('#UploadForm-form').data('fileupload');
widgetInstance.options.defDoneFn = widgetInstance.options.done
widgetInstance.options.done = function myCustomFn() {
    // do something here
    widgetInstance.options.defDoneFn .apply(this, arguments);
};
</code>
Note: I have tested the code against http://blueimp.github.io/jQuery-File-Upload/basic.html. There could be some changes like data key<code>(.data(<data key>)</code> if you are using different version of fileupload.


using QueryDSL in osgi

Question: I have been trying to use querydsl in a project which is an osgi bundle.
my pom.xml has the following dependencies:
<code>    <dependency>
        <groupId>com.mysema.querydsl</groupId>
        <artifactId>querydsl-apt</artifactId>
        <version>2.5.0</version>

    </dependency>

    <dependency>
        <groupId>com.mysema.querydsl</groupId>
        <artifactId>querydsl-jpa</artifactId>
        <version>2.5.0</version>
    </dependency>
</code>
As well as the plugin
<code>        <plugin>
            <groupId>com.mysema.maven</groupId>
            <artifactId>maven-apt-plugin</artifactId>
            <version>0.3.2</version>
            <executions>
                <execution>
                    <goals>
                        <goal>process</goal>
                    </goals>
                    <configuration>
                        <outputDirectory>target/generated-sources/java</outputDirectory>
                        <processor>com.mysema.query.apt.jpa.JPAAnnotationProcessor</processor>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    <plugin>
            <groupId>org.ops4j</groupId>
            <artifactId>maven-pax-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.apache.felix</groupId>
            <artifactId>maven-bundle-plugin</artifactId>
            <extensions>true</extensions>
            <!-- | the following instructions build a simple set of public/private
                classes into an OSGi bundle -->
            <configuration>
                <instructions>
                    <Import-Package>com.mysema.query.jpa,*</Import-Package>
                    <Export-Package>com.mypackage.package.*;version="${project.version}"</Export-Package>
                </instructions>
            </configuration>
        </plugin>
</code>
Still when I try to start the bundle I get:
<code> Error executing command: Unresolved constraint in bundle com.mypackage.package [163]: Unable to resolve 163.0: missing requirement [163.0] package; (&(package=com.mysema.query.jpa)(version>=2.5.0)(!(version>=3.0.0)))
</code>
I was using an older version of querydsl but apparently they fixed some stuff about osgi recently so I upgraded. The problem persists.
What I am missing for querydsl to work inside osgi?
Comment: The querydsl-jpa bundle has a lot of dependencies (direct and transient), I'm assuming you've loaded all of those and that the querydsl-jpa bundle is STARTED?
Comment: no. you are right. I am new to osgi and I assumed that doing osgi:instal mvn..... in the karaf console I would get all dependencies resolved. apparently not
Answer: Installing each dependency by hand will be a pain, but AFAIK there's nothing that will take a maven artifact and chain back of all dependencies - this would fail as where would it stop?
You could end up with every version of every logging framework (even if you had pax-logging installed), or the wrong implementation.
Alas in maven's case there's currently no way of applying semantic versioning or higher level requirement and capability.  (Though BND (maven-bundle-plugin, bndtools) makes some sensible assumptions at a code level)
Karaf features (see the PDF manual in distribution's ${KARAF_HOME}) can do a lot to alleviate this but it can take some work to setup.  There's a(t least) couple of ways to generate features files;

Use the features-maven-plugin
Use the maven-build-helper plugin to publish an XML file that you handcraft (laborious but you can maintain versions using resource filtering).
Comment: the features-maven-plugin  generate-features-xml was very helpful. It did not fully solve my problem, but it took me at least half the way. thank you
Comment: I think most people are in the same position, luckily the OSGi Alliance are working on resolving this. Glad it helped =)

using QueryDSL in osgi


Since the beta distribution is similar in form to the binomial, why do we need the beta distribution?

Question: It appears that the binomial distribution is very similar in form to the beta distribution and that I can re-parametrize constants on either pdf to make them look the same. So, why do we need the beta distribution? Is it for a specific purpose? Thanks!
Comment: "I can re-parametrize constants on either pdf to make them look the same" — did you try it?  You can't.  The binomial distribution doesn't even have a pdf; it has a pmf.
Comment: As everyone else has pointed out, the beta and binomial are not in the same family of distributions (i.e. one is not a generalization of the other). There are however several other distributions which are generalizations of others such as the exponential(\beta) is just a gamma(\alpha=1, \beta). Sometimes it is convenient to work with and have results based on a specific form of a distribution rather than always having to use the complex generalized forms.
Comment: To better understand the beta distribution, it may help you to read this CV thread: [What is the intuition behind beta distribution?](http://stats.stackexchange.com/questions/47771/)
Comment: Note that the binomial doesn't have a pdf; being discrete it has a probability function.
Answer: They're related, but not actually so similar in form.
In the beta, the variable (and its complement) is raised to some power, but in the binomial the variable is the power (and it also appears in a binomial coefficient).
While the functional forms do look somewhat alike (there are terms in one that correspond to terms in the other), the variables that represent the parameters and the random variable in each are different. That's rather important; it's why they're actually not the same thing at all.
The binomial distribution is usually used for counts, or in scaled form, for count-based proportions (though you could use it for other bounded discrete random variables on a purely pragmatic basis). It's discrete.
The beta distribution is continuous, and so is not normally used for counts.

By way of example, compare these two functions:
$y = b^x,\, x=0,1,2,3,...$ and $y = x^a,\, 0<x<1$.
Both these functions are defined by expressions of the same form (something of the form $c^d$), but the roles of variable and constant are interchanged and the domain is different. The relationship between the beta and the binomial is like the relationship between those two functions.
- In summary: different form, and different domain
Here's a simple example of a beta distribution, the $\text{beta}(1,1)$. Which binomial distribution does the same job?

Or consider a $\text{beta}(2,1)$; it's hard to find a binomial which looks similar. Here's one attempt:

The entire beta pdf sits between the first two green spikes in the binomial pf, though they can't really be shown on the same plot because the y-axes measure different things.
While the shapes are vaguely similar in the sense that they're both left skew, they're really quite different, and used for different things.
--
Here's a challenge:
For $X_1\sim\text{beta}(1,1)$ and a $X_2\sim\text{beta(3,2)}$, find binomial distributions (presumably scaled) that can simultaneously reasonably accurately (say to within $c=(0.95, 1.05)$  times the correct probability, give or take) which have the same mean and variance or mean and range (you pick), but also approximately reproduce the probability of being in these three subintervals: (a) $(1/\pi,1/e)$, (b) $(\exp(-\frac{1}{2}),2/\pi)$, and (c) $(\exp(-3),1/\pi^2)$

The beta is used to do many things, including model continuous proportions, act as a prior on the $p$ parameter of a binomial, it is the distribution of uniform order statistics (and can be used in the derivation of the distribution of order statistics for other continuous distributions, used as a mixing distribution for the binomial $p$ (producing the beta-binomial distribution), to model task completion times in project management, and many other things.
Comment: The number of successes in zero trials is always zero, so the probability function is a spike at zero, and the cdf is a step function that jumps from 0 to 1 at x=0. So ... nothing like a uniform on (0,1).
Comment: for beta(1,1), i understand it is a uniform distribution on [0,1]. But for the binomial, is it the case where we dont have any trials at ALL?


Get Referer URL and FOLDER name in PHP

Question: I have several folders on my domain, within each folder contains an index.php file that checks to see if the database connection passes or fails, if it fails, the page is redirected to a top level file (outside of all folders) called offline.php. This part works great. The basic format I'm using to redirect if the db is offline is:
if ( !$dbconnect ) {
header("Location: https://www.test.com/offline.php");
}

Then, within the offline.php page, I need to check to see which folder brought the user to the offline.php page, and display a unique message to the user - based on the folder that brought them to the offline.php page.
For example:
test.com/test1/index.php redirects to offline.php, the message would say 'test1 brought you to this page'
test.com/test2/index.php redirects to offline.php, the message would say 'test2 brought you to this page'.
In multiple browsers I've tried the following code, which always results in 'unknown uri':
$url = 'https://' . $_SERVER['HTTP_REFERER'] ;
if ( strpos($url,'test') !== false ) {
echo 'test';
} elseif ( strpos($url,'test1') !== false ) {
echo 'test1';
} elseif ( strpos($url,'test2') !== false ) {
echo 'test2';
} else {
echo 'unknown uri';
}

Suggestions?
EDIT
Due to the unreliable nature of HTTP_REFERER I've decided to put all of the conditions within the index.php page and forget about the offline.php page. A HUGE thank you to everyone who offered suggestions!
Comment: I see you've added some stuff since I answered earlier.  What happens when you just echo $_SERVER['HTTP_REFERER'], does anything show?
Comment: nothing shows, just a blank page.
Comment: $_SERVER['HTTP_REFERER'] is unreliable, use with extreme caution
Comment: I would just start session in that error-generating page, put script name in some variable and redirect. Then this redirected page could start session and see if this variable exists by `if(isset($_SESSION['somevariable']))`
Comment: I like Kitet's better than mine. I didn't think of storing it in a session variable, but it would be much cleaner IMO.
Answer: Why would you use redirects at all? They are heavy on the server, slow and just plain old unnecessary. Use a switch statement and have 1 controlling page instead of multiple folders and pages.
Answer: If you use the following code on your offline.php page, you can see all of the $_SERVER variables available (referring URL is in there)
echo '
',print_r($_SERVER),'
';

From there, you can take $_SERVER['HTTP_REFERER'] use a select case, or if then statement and accomplish your goal.
Based on some of your questions in the comments and people pointing out the use of $_SERVER['HTTP_REFERER'] being unreliable, you could do something like this instead.
On your index.php page with the dbconnect check, you could modify it to be something like this. header("Location: https://www.test.com/offline.php?org=".urlencode($_SERVER['REQUEST_URI']));
Then, on the offline.php,
$page = urldecode($_GET['org']);
$org = explode('/',$page);

echo $org[1] to get the first value after the slash, $org[2] would get the next value etc..
Comment: Just tried this but the referring url was not included in the list, should it be listed?
Comment: It depends on the browser. In some browsers, the referer is the page that performed the redirect, in others it's the page that linked to the page with the redirect.
Comment: See http://stackoverflow.com/questions/2158283/will-a-302-redirect-maintain-the-referer-string
Comment: Ohh yeah, good call Baramar.  user1172854, is it possible to adjust your script that redirects them to include the originating page in the url? That way it would be definitive where it's coming from and is cross-browser dependent.
Comment: How best to include the originating page in the url?
Comment: I modified my answer to show one way to include the originating url.

Get Referer URL and FOLDER name in PHP


Como definir nome personalizado no índice de array em Javascript?

Question: No PHP podemos utilizar:
$validation = [
  200 => ['icon-check', 'The key match with the message!'],
  400 => ['icon-close', 'The key doesn\'t match with the message!'],
  403 => ['icon-close', 'Impossible to verify, the key or text can be wrong!']
];

Dessa forma podemos obter o resultado utilizando $validation[200][0] para retornar icon-check.

No entanto, não consigo fazer o mesmo no Javascript:
var validation = [
   200 => ['icon-check', 'The key match with the message!'],
   400 : ['icon-close', 'The key doesn\'t match with the message!'],
   403 = ['icon-close', 'Impossible to verify, the key or text can be wrong!']
];

Todos tipos de tentativas falham, tanto com =>, com : e com =.

Minha única alternativa foi definir o índice na variável:
var validation = [];

validation['200'] = ['icon-check', 'The key match with the message!'];
validation['400'] = ['icon-close', 'The key doesn\'t match with the message!'];
validation['403'] = ['icon-close', 'Impossible to verify, the key or text can be wrong!'];

Entretanto, existe alguma alternativa sem ter que repetir o validation[XXX] = [...]?
Comment: Acredito que possa fazer algo semelhante com objeto: `var validation = {200: [...]}`, mas não sei se é a melhor abordagem.
Comment: http://pt.stackoverflow.com/questions/41140
Comment: http://pt.stackoverflow.com/questions/173916
Answer: A maneira correta(e a única que conheço) de ter uma estrutura de dados com índice string é criando um objeto. Ex.:
var validation = {
    200: ['icon-check', 'The key match with the message!'],
    400: ['icon-close', 'The key doesn\'t match with the message!'].
    403: ['icon-close', 'Impossible to verify, the key or text can be wrong!']
}

O problema se deve à confusão de conceito de arrays que o PHP proporciona.
Veja bem, o PHP chama de arrays o que estruturalmente é um ordered map, então é fácil pensar que $array = [ '400' => 'xxxxxx' ]; é um array quando na verdade é um Map<string, string>.
O javascript oferece uma estrututa de dados que também é um Map, são os Object, que podem ser criados na forma literal:
var obj = {
    400: "xxxxxxxx",
    "403": "yyyyyy"
}

Ou com construtores (não é muito utilizado):
var obj = new Object();
obj['400'] = "xxxx";
obj.propriedade = "yyyyy"

Enfim, toda a linguagem tem uma estrutura de dados indexado com strings pois é uma estrutura bem comum e útil pois a complexidade de busca por índice é geralmente O(1). No python temos dicionários, no PHP os arrays associativos, no java temos HashMaps, etc.
Answer: Para atribuir uma key e valor utilize SEMPRE o formato json ao invés de array:
var obj = {
    '200': ['icon-check', 'The key match with the message!'],
    '400': ['icon-close', 'The key doesn\'t match with the message!']
}

Mas porque fazer isso? Você me pergunta...

Bem, você consegue até especificar um índice numérico para um array no javascript, porém isso não é recomendado, pois ele irá interpretar o array como se tivesse valores até aquele determinado índice, exemplo:
var arr = [];
arr[200] = ['icon-check', 'The key match with the message!'];
console.log(arr.length);
// Result: 201

Dessa forma para se ter mais flexibilidade na hora de especificar uma key e não correr o risco de introduzir um bug no seu código, é extremamente recomendado usar o formato json.


Finding the integer solution that makes $\lvert x-y \rvert$ the greatest?

Question: I was attempting to solve this problem.
Let $x,y$ be non-negative integers which satisfy the equation.
$$2^{x} +2^{y} = x^{2} +y^{2}$$
Find the maximum possible value for $\lvert x-y \rvert$?
At first I tried constructing a function of two variables and finding critical points but the critical points ended up having non-integer values. Then I tried manipulating the equation to get $x^2 +y^2$ to look like $\lvert x-y \rvert$ but that didn't help make it easier to find integer solutions. I ended up trying to guess solutions to the equation and the best I could come up with was $(3,0)$ which gives $\lvert x-y \rvert$ a value of $3$. Only thing is that I'm not sure if this is the highest that you can get the value of $\lvert x-y \rvert$ to be.
I was wondering is there any way to solve this problem without guessing and prove that the solution you get is what maximizes $\lvert x-y \rvert$?
Answer: Consider the function $f(x)=2^x-x^2$:

Note that for $x\ge5$, $f(x)\ge7$. Thus, there is no value $y$ so that $f(x)+f(y)=0$ since $f(y)\gt-2$.
Looking at the data, we get solutions of
$$
\{(0,3),(1,3),(2,2),(2,4),(3,0),(3,1),(4,2),(4,4)\}
$$
Thus, the solutions with the greatest $\left|x-y\right|$ are $\{(0,3),(3,0)\}$. Therefore,
$$
\max\left|x-y\right|=3
$$

$\boldsymbol{f(x)}$ is convex for $\boldsymbol{x\ge3}$ and increasing for $\boldsymbol{x\ge4}$
Since $f''(x)=\log(2)^22^x-2$, we have $f''(x)\ge0$ when $x\ge1-2\log_2(\log(2))$. And since $2^2\gt e\implies\log(2)\gt\frac12\implies\log_2(\log(2))\gt-1$, we get that $3\gt1-2\log_2(\log(2))$.
Therefore, $f''(x)\ge0$ when $x\ge3$.
Furthermore, since $\frac{f(4)-f(3)}{4-3}=1$, the Mean Value Theorem says that $f'(x)=1$ for some $x\in(3,4)$. Since $f(x)$ is convex for $x\ge3$, we have that $f'(x)\ge1$ for $x\ge4$.
Comment: @user262291: we don't really need a computer to plot this function. The fact that $f(x)$ is increasing for $x\ge4$ is pretty easy to show.
Comment: That is a nice solution, but I was wondering whether there is a way to do this problem without the use of computers. I actually got this problem from a GMAT practice test and when writing the GMAT you aren't given any access to computers or calculators.
Answer: You can check that the only integer solutions happen in $x,y \in [0,4]^2$. They are $(3,0), (3,1), (4,2), (2,2)$ and $(4,4)$, plus their symmetric entrees. You can show that with $x,y \ge 5$ the difference
$$
2^x+2^y-x^2-y^2
$$
is increasing in both $x$ and $y$, so the optimal value of $|x-y|$ is indeed 3.
Comment: How do you know that there aren't any integer solutions when x>4 and y<4 or vice versa.
Comment: @user262291 because when $x,y \ge 5$ the difference between the sides of the constraint is positive and increasing in both variables.
Comment: Do you mind proving that all the possible integer solutions must come from the interval [0,4]x[0,4]


How to traverse an ordered "network"?

Question: I have an interesting python "network" challenge. Not sure what data construct to use and am hoping to learn something new from the experts!
The following is a snippet from an ordered table (see SORT column) with a FROM NODE and TO NODE. This table represents a hydrologic network (watershed sequencing) from upstream to downstream. TO NODE = 0 indicates the bottom of the network.
Given there can be many branches, how does one traverse a network from any given FROM NODE to TO NODE = 0 and store the list of nodes that are sequenced over?
For example, starting at:
HYBAS_ID (this is same as FROM NODE)= 2120523430

see <---- about 2/3rd way down, the sequence of Nodes would include the following:
[2120523430, 2120523020, 2120523290, 2120523280,2120523270,2120020790, 0]

Table:
 SORT   FROM NODE   TO NODE
30534   2121173490  2120522070
30533   2120521930  2120521630
30532   2120522070  2120521630
30531   2120521620  2120522930
30530   2120521630  2120522930
30529   2121172200  2121173080
30528   2120522930  2120523360
30527   2120522940  2120523360
30526   2120519380  2120523170
30525   2120519520  2120523170
30524   2120523440  2120523350
30523   2120523360  2120523350
30522   2120525750  2120523430
30521   2120525490  2120523430
30520   2121173080  2120522820
30519   2120523430  2120523020 <------- if you start network here
30518   2120523350  2120523020
30517   2120522820  2120523290
30516   2120523020  2120523290 <------ the second node is here
30515   2120523170  2120523280
30514   2120523290  2120523280 <------ third node here
30513   2120523160  2120523270
30512   2120523280  2120523270 <------ fourth
30511   2120523150  2120020790
30510   2120523270  2120020790 <------ fifth
30509   2120020790  0          <------ End.

Would a dictionary and some kind of graph structure work to traverse this network? The code may be fed a list of thousands or millions of FROM NODES to calculate the routing so efficiency is important.
Comment: Store them all in a dict with the from value as key and the to value as dict, and the problem is easy. If that takes too much memory, you can store them in a disk-backed key-value store, and it's just as easy.
Comment: For future clarity, "HYBAS_ID" is identical to "FROM NODE" (mistakenly overlooked standardizing the naming convention but they're equivalent).
Answer: The standard way to do this is with an unordered dictionary representing a directed graph, and then to iterate through it. First you have to populate the dictionary (which let's say is a csv for the sake of argument), and then iterate through the values. so
my_graph = {}
with open("myfile.csv") as f:
    for l in f:
        split_lines = l.split(",")
        my_graph[int(split_lines[1])] = int(split_lines[2])

HYBAS_ID = 2120523430
current_hybas = my_graph[HYBAS_ID]
return_value = [HYBAS_ID]
while current_hybas != 0:
    return_value.append(current_hybas)
    current_hybas = my_graph[current_hybas]

print return_value

Should approximately get you what you want.

How to traverse an ordered "network"?


Como declarar um vetor com 2 posições?

Question: Tenho um exercício com o seguinte enunciado:

Escreva uma classe cujos objetos representam alunos matriculados em uma disciplina. Cada objeto dessa classe deve guardar os seguintes dados do aluno: matrícula, nome, 2 notas de prova e 1 nota de trabalho. Escreva os seguintes métodos para esta classe:
media: calcula a média final do aluno (cada prova tem peso 2,5 e o trabalho tem peso 2)
final: calcula quanto o aluno precisa para a prova final (retorna zero se ele não for para a final)

Meu código está assim:
public class Aluno {
    private String matricula;
    private String nome;
    private double notaProva;
    private double notaTrabalho;

    public void media(double nota1, double nota2) {
        this.notaProva = nota1;
        this.notaTrabalho = nota2;

        double mediaPonderada;
        mediaPonderada = ((this.notaProva*2.5+this.notaTrabalho*2)/(2+2.5));
        System.out.println("Média final: "+ mediaPonderada);
    }
    public void resultadoFinal() {

    }
    public void detalhes() {
        System.out.println("Nome: "+this.nome);
        System.out.println("Matricula: "+ this.matricula);
        System.out.println("Nota da prova: "+ this.notaProva);
        System.out.println("Nota do trabalho: "+ this.notaTrabalho);

    }
    public String getMatricula() {
        return matricula;
    }
    public void setMatricula(String matricula) {
        this.matricula = matricula;
    }
    public String getNome() {
        return nome;
    }
    public void setNome(String nome) {
        this.nome = nome;
    }
}

public class PrincipalMain {

    public static void main(String[] args) {
        Aluno aluno = new Aluno();
        aluno.setNome("Magno");
        aluno.setMatricula("20194456458");
        aluno.media(9, 5);
        aluno.detalhes();
    }
}

Como eu faço para declarar um vetor no atributo notaProva com 2 posições?
Answer: Para definir que o tipo de uma variável é um array é necessário usar colchetes, por exemplo:
private double[] notasProvas;

Em Java não é possível declarar um array e já definir a quantidade de posições. O tamanho sempre precisa ser definido na inicialização.
No seu caso, imagino que a melhor saída é definir um construtor para a classe e inicializar o array neste construtor.
public class Aluno {
    // ... outros campos

    private double[] notasProvas;

    public Aluno() {
        notasProvas = new double[2];
    }
}

Claro que, depois disto, você precisará pensar em como vai ficar a estrutura do(s) getters e setters.
Answer: Quando você quiser declarar um vetor, use colchetes. Exemplo: double vetor1 [];
Ao declarar o seu tamanho e conteúdo, pode-se inserir diretamente os dados da seguinte forma: vetor1 = {x1, x2};
Ou você pode criar um vetor vazio, e preenchê-lo durante a execução do programa: vetor1 = new double[2];
Espero ter ajudado

Como declarar um vetor com 2 posições?


What user is my Windows Bash shell running as when it interacts with Windows file system?

Question: I have downloaded and installed Bash for Windows.  When I do a simple ls -al, I get lots of permission denied errors.
I'm not clear quite how the user permissions are being resolved. Let's say that in the bash shell I set up a user called moi and has home drive /home/moi. If I then do ls -al /mnt/c, I get a bunch of permission denied errors, e.g.
ls: cannot access /mnt/c/hiberfil.sys: Permission denied
ls: cannot access /mnt/c/pagefile.sys: Permission denied
ls: cannot access /mnt/c/swapfile.sys: Permission denied total 2388

    drwxrwxrwx 2 root root      0 Sep 26 18:09 .
    drwxr-xr-x 2 root root      0 Jan  1  1970 ..
    drwxrwxrwx 2 root root      0 Oct 22  2015 AutoDiscover
    -r-xr-xr-x 1 root root 398156 Jul 26  2012 bootmgr
    -r-xr-xr-x 1 root root      1 Oct 30  2015 BOOTNXT

I have checked on the "Windows side" and there is no new user called moi.  So, when ls reports permission denied for /mnt/c/swapfile.sys for example, what user is being denied?
A follow on question would be: what do I need to do to get the Bash user and call it my "normal" windows user to have the same permissions?
Comment: The bash shell will be running as whoever you executed with.
Answer: The Windows user is the user who started bash.exe. You can verify this by creating a file from Linux in the /mnt/c directory looking at the owner of the file in Windows.
From the windows command prompt:
C:\test>echo %USERNAME%
tim

C:\test>bash.exe
wsluser@computer:/mnt/c/test$ touch foo.txt
wsluser@computer:/mnt/c/test$ exit
exit

C:\test>dir /q C:\test\foo.txt
 Volume in drive C is OS
 Volume Serial Number is B8BA-E032

 Directory of C:\test

2017-06-26  02:06 PM                 0 tim   foo.txt
               1 File(s)              0 bytes
               0 Dir(s)  363,224,875,008 bytes free

Notice that bash.exe was started by the tim user, and the C:\test\foo.txt file which was created from WSL is also owned by tim.
Answer: You can access the Linux file system using the windows file explorer. It's located in your AppData\Local folder. It's "semi-hidden" but accessible:
C:\Users\[username]\AppData\Local\lxss\rootfs
Comment: Please avoid creating and/or modifying files under ...\lxss\ using Windows apps and tools: There's a reason this is a hidden & system folder!! https://blogs.msdn.microsoft.com/commandline/2016/11/17/do-not-change-linux-files-using-windows-apps-and-tools/


Find $y''$ in terms of $x$ and $y$

Question: Find $y''$ in terms of $x$ and $y$
$$xy=x+y$$
I have calculated $y'$ as
$$\frac{y-1}{1-x}=y'$$
which is correct according to the solutions manual but when I try to calculate $y''$ I get
$$y''= \frac{y'(1-x)+y-1}{(1-x)^2}$$
$$y''= \frac{y'+y-1}{1-x}$$
but the solutions manual get $y''= \frac{2y'}{1-x}$
What am I doing wrong?
Comment: This has been edited a couple of times for formatting. The second expression currently noted for $y''$ does not follow from the first - is this what you intended?
Answer: Do not solve for $y'$ that only makes matters more complicated. Just differentiate twice and solve for $y''$.
$$xy=x+y \implies y+xy'=1+y' \implies y'+y'+xy''=y''$$
Having these equations you can then solve the separatly for the derivatives:
$$y'=\frac{y-1}{1-x}$$
$$y''=\frac{2}{1-x}y'=\frac{2}{1-x}\frac{y-1}{1-x}$$
Comment: Definitely the easiest way to get to $y''$ rather than differentiating the fraction for $y'$
Answer: Note that $$\frac {y'(1-x)+y-1}{(1-x)^2}=\frac {y'}{1-x}+\frac {y-1}{(1-x)^2}=2\frac {y'}{1-x}$$ because of the formula you already have for $y'$. So the two answers are the same.
Answer: Up to
$$
y''= \frac{y'(1-x)+y-1}{(1-x)^2}
$$
it looks good. However, from there you simplify the fraction by $1-x$ without taking the $y-1$ in the numerator into account. So instead of $y'' = \frac{y' +y-1}{1-x}$ that you have, it should've been
$$
y''= \frac{y'+\frac{y-1}{1-x}}{1-x}
$$
and finally, we use that we already know that $y' = \frac{y-1}{1-x}$ to make the numerator into $2y'$.

Find $y''$ in terms of $x$ and $y$


RemoteWebDriver Unable to bind to locking port 7054 within 45000 ms

Question: I use the .NET binding for WebDriver to run tests concurrently through Grid2.  Running tests individually is no issue, however when running multiple tests through the Grid, the following error will occur occasionally, and when it occurs once there are usually multiple test fail for the same reason:

Unable to bind to locking port 7054 within 45000 ms Build info:
  version: '2.29.0', revision: '58258c3', time: '2013-01-17 22:46:35'
  System info: os.name: 'Windows Server 2008 R2', os.arch: 'x86',
  os.version: '6.1', java.version: '1.7.0_11' Driver info:
  driver.version: FirefoxDriver

We have traced the source of the issue to where we create the RemoteWebDriver instance, which looks like:
<code>            var desiredCapabilites = new DesiredCapabilities();

            desiredCapabilites.SetCapability(CapabilityType.BrowserName, "firefox");
            desiredCapabilites.SetCapability(CapabilityType.Platform, new Platform(PlatformType.Windows));
            desiredCapabilites.SetCapability(CapabilityType.Version, "14.0.1");
                try { _driver = new RemoteWebDriver(new Uri("http://qwautomation:4444/wd/hub"), desiredCapabilites, TimeSpan.FromSeconds(180.0)); }
                catch (Exception ex) { Notes.Add("AUTOMATION ERROR: This error originates from MasterSetup.cs (line 47)"); Notes.Add("AUTOMATION ERROR: " + ex.Message); Exception("AUTOMATION ERROR: This error originates from MasterSetup.cs (line 47)"); }
</code>
Where, qwautomation is the machine our Grid Hub is hosted on.  My assumption has been that the Hub is dispatching tests to a node that already contains a FireFox instance on a port, or that the sessions are not being disposed of properly by the JVM (as I can see folders with WebDriver session IDs in the node machine's temp folder).
I have written configuration files for the node and hub, which I will post below.  I didn't know if there would be a way to 'queue' a test if it could not bind to the FireFox port.
Hub
<code>{
        "port": 4444,
        "newSessionWaitTimeout": 30000,
        "nodePolling": 5000,
        "cleanUpCycle": 5000,
        "timeout": 300000,
        "browserTimeout": 0,
        "maxSession": 1
}
</code>
Nodes
<code>{
    "capabilities":
            [
                {
                    "browserName":"firefox",
                    "maxInstances":1
                }

            ],
    "configuration":
            {
                    "cleanUpCycle":2000,
                    "timeout":180000,
                    "maxSession":5,
                    "hubPort":4444,
                    "hubHost":"qwautomation"
            }
}
</code>
EDIT: I have removed the "Version" capability from the DesiredCapabilities object and have not seen the error surface after several test runs.  I don't know if this is an actual solution, but it seems to at least decrease the risk of the Exception.
Comment: kill all firefox sessions and then try.. If it is not working uninstall firefox and again install the same.
Comment: I have reinstalled Firefox with numerous versions and that isn't a fix.  I'm looking for a long term solution, because it would not be practical to have to manually terminate sessions, or reinstall FF when this issue occurs, since we have tests running 24/7.
Comment: Kill all java processes before starting your tests.
Comment: What *exact* version of Firefox is it running?
Comment: We were running 14.0.1 on 6 nodes and 17.0.1 on 3 others.  We have code setup to target nodes, but I have seen this issue arise on all versions of FF I have tried, even earlier than 14.  Switching to Chrome has eliminated this issue, but I would like the ability to run tests on FF still.  I am wondering if this is an issue with the FirefoxDriver, since it appears that it is not releasing the lock of port 7055.
Answer: I've found that rebooting the server usually clears up the problem. If the problem persists, try running fewer consecutive tests through the grid or upping the memory capacity on your Selenium nodes.
Comment: If it's any help, I've been using Ruby bindings for Firefox and I run into the same issues from time to time. It's intermittent and not reproducible to my knowledge.
Comment: I've started to think this is an actual issue with the .NET bindings for Firefox.  We have switched to Chrome without any issues.
Answer: I've encountered this error many times and I discovered that I was actually the cause of it.
I typically run remote desktops from my development computer into my VMs to watch the tests take place on my grids.
When I toggle through windows on my local desktop using Alt-Tab and Shift-Alt-Tab I have done this selecting the VM precisely while Selenium was starting FireFox and inadvertently causing Firefox to start in safe mode because I was holding down keys.
When that happens, Firefox waits for confirmation that you really want to start in safe mode and never fully opens. All subsequent requests to open FireFox will fail until you confirm.
When the errors would occur, I used to stop the test and reboot everything starting with the test generator until one day I started with the VMs first, noticed the dialog and got to digging around for the cause of the dialog. It was then that I realized I was causing FireFox to start this way. Now I'm a bit more careful and don't have the problem any more.


Converter banco MySQL de latin1 para utf8

Question: Tenho um banco de dados MySQL que está atualmente definido com charset latin1. Quando faço um select em uma tabela, por exemplo:
SELECT nome FROM acolhidos

Os caracteres acentuados aparecem dessa maneira:
ALESSANDRO ROGÉRIO MOTA DA SILVA
DANIEL MENDONÇA NOVAES
JOSÉ SILVA ARAUJO

Então, se eu preciso rodar um select no PHP, por exemplo:
$nome = '%JOSÉ%';

$sql = "SELECT nome FROM acolhidos WHERE nome LIKE '%$nome%'";

$query = @mysqli_query($conn->link, $sql);

O José não vai aparecer no resultado da query, porque ficou gravado no banco como JOSÉ
Existe alguma função que converta a minha variável $nome, de forma que os caracteres acentuados fiquem nesse formato: JOSÉ , ROGÉRIO etc, e assim o select acima possa trazer os nomes com caracteres acentuados no resultado da pesquisa?
Mas caso não haja essa função, que resolveria o meu problema de maneira mais simples... como eu poderia converter o conteúdo de todo o meu banco de dados para utf8?
Eu tentei usando o comando:
ALTER DATABASE `bancodedados` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

Porém, após rodar o comando acima, os caracteres acentuados continuam aparecendo dessa forma:"JOSÉ" etc - ou seja - não foram convertidos para "JOSÉ" etc.
Há algum outro comando no MySQL que eu poderia rodar além deste, para fazer a conversão diretamente no banco de dados?
Esta é a configuração atual do banco de dados:
show variables like '%character%'

character_set_client: utf8mb4
character_set_connection: utf8mb4
character_set_database: latin1
character_set_filesystem: binary
character_set_results: utf8mb4
character_set_server: latin1
character_set_system: utf8
character_sets_dir: /usr/share/mysql/charsets/

show variables like 'collation%'

collation_connection: utf8mb4_general_ci
collation_database: latin1_swedish_ci
collation_server: latin1_swedish_ci

Comment: Antes você precisa entender se o problema é a conversão ou se é o ambiente cliente. Só tome cuidado pois são 2 situações: uma é trocar a definição para UTF (sem mexer nos dados), a outra é realmente converter linha a linha. Teria que olhar (por exemplo visualizando em hexadecimal) o valor real dos campos para determinar a melhor ação. Se os dados estiverem corretos e a definição errada, só tem que mudar a definição. Fico pensando, não seria mais simples trabalhar em latin no PHP em vez de converter? Nem todo mundo precisa de UTF. Principalmente línguas ocidentais.
Comment: Depois de conectar ao banco tente usar: **mysqli_query($conexao,"SET NAMES 'utf8'");**
Comment: Para a conversão da base, você pode utilizar o software HeidiSQL, você pode alterar todas as definições nas propriedades da base de dados, ele possui um script próprio para isso. Para os dados, a última versão que eu utilize (dois anos atrás), não tinha suporte. Talvez agora possua.
Answer: Obrigado pelos comentários de todos!
Eu consegui resolver o problema apenas incluindo 'collate latin1_swedish_ci' ao final do select:
$sql = "SELECT nome FROM acolhidos WHERE nome LIKE '%$nome%' collate latin1_swedish_ci";


Computing the quantity $ \frac{{x}\cdot{y}}{{\|x\|\|y\|}}$ in terms of $a$ where $x=(1,0)$ and $y=(a,-2)$

Question:
Let $x=(1,0)$ and $y=(a,-2)$ be  two vectors $ℝ^2$, where $a$ is a real number. Then compute the quantity
$$
\frac{{x}\cdot{y}}{{\|x\|\|y\|}}$$
in terms of $a$.

My work so far:
$$x\cdot y=1\cdot a+0\cdot(-2)=a$$
And so
$$\|x\|=\sqrt{{1^2}+{a^2}}=1+a$$
$$\|y\|=\sqrt{{0^2}+{(-2)^2}}=2$$
Thus
$$\cos(\theta)\frac{a}{\sqrt{{2a+2}}}$$
$$\theta=\cos^{-1}\frac{a}{\sqrt{2a+2}}$$
And I'm stuck here. Where did I go wrong to compute the quantity in terms of a? Would there be any better ways of tackling this?
Comment: Isn't $||x||=\sqrt{1^2+0^2}$ and $||y||=\sqrt{a^2+(-2)^2}$ so it's $\dfrac a{\sqrt{a^2+4}}$?
Comment: Also note that, in general, $\sqrt{p^2+q^2}\neq p+q$.
Comment: @J.W.Tanner Thanks for that! I figured it was due to a miscalculation on my end.
Comment: I think you confuse the $x$ as vector and $x$ as component of the vector.
Answer: Your computation that $\;\mathbf x\cdot\mathbf y=a\;$ was okay, but
$||\mathbf x||=||(1,0)||=\sqrt{1^2+0^2}=1,$ and $||\mathbf y||=||(a,-2)||=\sqrt{a^2+(-2)^2}=\sqrt{a^2+4},$
so  $\dfrac{\mathbf{x}\cdot\mathbf{y}}{\mathbf{||x||||y||}}
= \dfrac a{\sqrt{a^2+4}}.$
Also, as pointed out in the comments, you should be aware that
$\sqrt{1^2+a^2}$ does not generally equal $1+a$.  (To see that, try squaring them both.)
Comment: Thanks again. Your feedback is much appreciated and I can see where I went wrong.


Which experiment design to choose and when?

Question: I went through a "design of experiment" course recently, but there are still things that I haven't sorted out yet regarding which types of experiment to choose when facing a real world problem.
I understand that experiments can be costly, so one may be inclined to use a fractional design over a full factorial design (Taguchi, Box-Benkhen) for instance. But are there general guidelines to which types of design to choose ? Is cost the only issue ?
I come across many theoretical designs through books, but I can't grasp why some types of experimental designs are used instead of others. I am probably lacking something here, so if anyone can help me sort this out it will be deeply appreciated.
Comment: Typically, the penalty for the reduced numbers of observations in fractional factorial designs is the inability to test for certain interactions or reduced power in testing interactions.
Comment: See: https://stats.stackexchange.com/questions/1815/recommended-books-on-experiment-design
Answer: It depends on what you want to do. Without knowing the specifics of what you want to investigate (or if you want compare two specific types of experimental design and the merits of each for a particular process / application), it's difficult to give a precise answer.
At a broad level, Box Hunter and Hunter define three types of experimental investigations:

screening studies. These tell you which of a large number of variables affects the response.
empirical studies. Produces an empirical model showing how the variables affect the response. You don't know the true nature of the response function, f.
mechanistic studies. Produces a model that leads to the correct functional form of the response f, and enables you to explain why the response is the way it is.

The same authors also advise not worrying too much about your first experimental design, as once the first part of the experiment has been completed you will know more about the process, so you can plan a better second experiment, and then plan a better third experiment and so on, in an iterative process.
Box, G.E., Hunter, W.G. and Hunter, J.S., 1978. Statistics for experimenters.
Comment: Thanks for your feedback. I also found this link of interest : [How do you select an experimental design?](https://www.itl.nist.gov/div898/handbook/pri/section3/pri33.htm)
Comment: And here's another one : [Gummi Bear DOE: Choosing a Design](http://blog.minitab.com/blog/statistics-and-quality-improvement/gummi-bear-doe-choosing-a-design)


The class of finite groups (models) and that of countable groups are not elementary classes (a generalized version).

Question: First some definitions: For a set $\Sigma$ of $\mathcal{L}$-sentences, $Mod(\Sigma)$ denotes the class of all models that satisfy $\Sigma$. For a class $\mathcal{M}$ of models, we say it is $EC$ if $\mathcal{M} = Mod(\sigma)$ for some sentence $\sigma$ and $EC_\Delta$ if $\mathcal{M} = Mod(\Sigma)$.
Problem.
Let $T$ be a theory having arbitrary large finite models. (For example, $T$ can be axioms for groups, or fields, or linear orderings.) Let $\mathcal{K}_{inf} = \{ M : M \models T, \mbox{Card}(|M|)\geq \infty \}$ and $\mathcal{K}_{fin} = \{ M : M \models T, \mbox{Card}(|M|)< \infty \}$.
Question.
(a) What does it mean by 'finite models'? Model's universe finite? Or a finite number of models?
(b) $\mathcal{K}_{inf}$ is $EC_\Delta$
(c) $\mathcal{K}_{fin}$ is $EC_\Delta$
(d)  $\mathcal{K}_{inf}$ is $EC$
Obviously I don't want some full detailed solution. It's just I can't even start on the problems.
Give some suggestions and hints. Should I use Compactness Theorem?
Comment: For (a) it means models with a finite universe.
Answer: Ok, here are some hints:
b) Can you express "the domain of M is infinite" as a first order theory? Call this theory $\text{Th}(\infty)$. Then the class of models of $T\cup \text{Th}(\infty)$ is $\mathcal{K}_{\text{inf}}$.
c) Here you can use compactness. Suppose $\mathcal{K}_{\text{fin}}$ is the class of models of $T^*$. Then $T^*\cup \text{Th}(\infty)$ is inconsistent. Do you see a problem here?
d) Use part c): if you had a single sentence picking out $\mathcal{K}_{\text{inf}}$, you could find a theory picking out $\mathcal{K}_{\text{fin}}$.


Como renderizar novamente meu botão no react native?

Question: Tenho esse botão que deve ter um background-color dependendo do estado:
<TouchableOpacity onPress={this._onPressButtonMassa}>
  <View style={[styles.button, this.props.hipertrofia === 'sim' ?
  { backgroundColor: 'black' } : { backgroundColor: 'white' }]}>
     <Icon name="bicycle" size={50} color="#FAFAFA" />
     <Text style={styles.buttonText}>Quero ganhar massa</Text>
  </View>
</TouchableOpacity>

Quando clico nesse botão o estado é alterado:
  _onPressButtonMassa = () => {
    this.setState({ hipertrofia: 'sim' });
    this.setState({ perdapeso: 'nao' });
  };

Porém o botão se inicia com o background-color white, e quando o estado é alterado ele não realiza a renderização do botão novamente, continuando branco independente do valor de this.props.hipertrofia
Como posso fazer pra renderizar novamente meu botão pra que ele altere para a o background-color black quando clicar nele?
Answer: Você está apontando para this.props.hipertrofia, por isso não está alterando...
Estabeleça o valor do state hipertrofia para this.props.hipertrofia no constructor, e no seu button, troque this.props.hipertrofia para this.state.hipertrofia
constructor(props) {
  super(props); //importante chamar super(props)!

  this.state = {
    hipertrofia: this.props.hipertrofia
    perdapeso: this.props.perdapeso
    //outros estados
  }
}

_onPressButtonMassa = () => {
  this.setState({ hipertrofia: 'sim' });
  this.setState({ perdapeso: 'nao' });
};

<TouchableOpacity onPress={this._onPressButtonMassa}>
  <View style={[styles.button, this.state.hipertrofia === 'sim' ?
  { backgroundColor: 'black' } : { backgroundColor: 'white' }]}>
  //...


Interpreting the change in random effect variance over time in a GLMM

Question: I am measuring the change in random effect (random intercept) over time by running models such as
glmer( bull~p1+p2+p3+p4+p5+(1|school),subset=(yr=2011),data=bull, family=binomial(link=logit))
using different subset=(yr=20??) for each year. In the comments to my other question, it was stated that another way to do this was to use dummy variables for each year, and specify them in the model as random slopes, thus:
glmer(bull~p1+p2+p3+p4+p5+(i1+i2+i3+i4+i5+i6-1|school),data=bull, family=binomial(link=logit))
where i1...i6 are the indicator variables of the different years. My question is: how is the output of this model to be interpreted in terms of the longitudinal change in random effects variance ?
Comment: Joe, the two are not exactly equivalent - in your first approach, the fixed effects are _also_ allowed to vary by year. In the second one, they are not. I don't have time to give a detailed answer right this minute but I will later if you haven't already received one. I've also edited your title to something that I think more completely captures the spirit of your question - feel free to change it if you disagree.
Comment: @Macro , thanks, that's fine, and yes, I understood that they are not exactly equivalent. I prefer the 2nd approach (if I can understand it!) because it avoids the problem of the fixed effects changing even though they don't change much (by the overlapping confidence intervals measure), and it seems more parsimonious. Even if it is not "better" I would still like to understand it.
Answer: Your code
glmer(bull~p1+p2+p3+p4+p5+(i1+i2+i3+i4+i5+i6-1|school),data=bull, family=binomial)

fits the binary logistic random effects model:
$$ \log \left( \frac{p_{ij}}{1-p_{ij}} \right) = \beta_0 + \sum_{k=1}^{5} \beta_k x_{ijk} + \sum_{t=1}^{6} \eta_{jt} I_{t} $$
where

$p_{ij}$ is the probability that subject $i$ in cluster $j$ responds with a '$1$', conditional on the covariates and random effects.
$x_{ijk}$ is the value of covariate $k$ for subject $i$ in cluster $j$ (according to your code there are $5$ covariates).
$\beta_0, ..., \beta_5$ are fixed effects regression coefficients
$\eta_{jt}$ is the cluster $j$ random effect for year $t$
$I_t$ is $1$ if the observation was taken in year $t$ and 0 otherwise (according to your code there are $6$ years)

We can see that the distribution of an observation taken in year $t$ is described by the model
$$\log \left( \frac{p_{ij}}{1-p_{ij}} \right)= \beta_0 + \sum_{k=1}^{5} \beta_k x_{ijk} + \eta_{jt} $$
Let $\sigma^{2}_{t} = {\rm var}(\eta_{jt})$ be the random effect variance for year $t$. Since this model allows that random effect variance to change over time, as indicated by the subscript $t$, the level of similarity between responses within the same cluster is also allowed to change over time. As I discussed in this answer, the value
$$ P_{t} = \frac{ \sigma^{2}_{t} }{ \sigma^{2}_{t} + \pi^2/3} $$
is a measure of the how similar individuals sharing a cluster are, in terms of their responses, at year $t$.
Your model produces estimates, $\hat \sigma^{2}_{t}$, of the random effect variances - which you can use to calculate
$$ \hat P_{t} = \frac{ \hat \sigma^{2}_{t} }{ \hat \sigma^{2}_{t} + \pi^2/3} $$
Plotting $\hat P_t$ vs. $t$ will give you an idea whether the (linear) dependencies are  increasing, decreasing, remaining constant, etc. over time, which is gets directly to your question of how to interpret the output of this model.
If you're interested in testing for statistical significance of these changes over time you can compare the model fit above with the model
$$ \log \left( \frac{p_{ij}}{1-p_{ij}} \right) = \beta_0 + \sum_{k=1}^{5} \beta_k x_{ijk} + \eta_j $$
which does not allow the within-cluster dependency to change over time. These can be compared using the Likelihood Ratio Test, where, under the null hypothesis that there is no change over time, twice the difference in the log-likelihoods of the two models will have a $\chi^2$ distribution with $5$ degrees of freedom (since you've deleted $5$ parameters to arrive at the smaller model). As long as $\sigma^{2}_{t}$ doesn't equal 0 for all $t$, and you're fitting this model with glmer (and thus the model is fit by maximum likelihood), this test should be fine to use here.


Por que que os SGBDs utilizam a sua própria paginação se o sistema operacional já possui uma?

Question: Estou estudando banco de dados e acabei chegando ao assunto de substituição de páginas como LRU e MRU. O sistema operacional já faz isso normalmente, por que o DB precisa fazer a sua própria paginação?
Comment: Dê uma olhada no [tour]. Você pode aceitar uma resposta se ela resolveu seu problema. Você pode votar em todos os posts do site também. Alguma lhe ajudou mais? Precisa que algo seja melhorado?
Answer: Porque os objetivos são diferentes. O sistema operacional cria páginas de bytes, o banco de dados cria páginas de dados com propósito específico. Ele sabe melhor o que tem ali dentro, as páginas dele possuem um formato específico, tanto que é comum que nem todas páginas são iguais, cada tipo de página possui uma estrutura de dados específica. Ele precisa ter mais controle de como elas são manipuladas. Em geral é otimizado para a performance.
O OS costuma usar uma lista ligada não muito eficiente para o padrão de acesso do banco de dados.

Claro que a não ser que o DB passe for cima da filesystem (era comum, hoje não é mais), ele também usa a paginação do OS em um nível mais baixo. Uma página do DB pode ocupar várias páginas do OS ou uma página do OS pode conter várias páginas do DB, isto depende do que é melhor para cada situação.
Tem bancos que usam a paginação do OS primariamente. Os que optaram por ter uma paginação própria precisavam de um nível extra de controle do que está memória. Tem um algoritmo de substituição mais apropriado (MRU, LRU, LRU-k, etc.). O algoritmo do banco de dados sabe o que deve estar ativo e pode priorizar o que é mais importante para aquele caso (índices, especialmente primários, precisam estar disponíveis com prioridade). Lembrando que a paginação do OS continua sendo usada para compor as páginas do DB.
Em geral as páginas do disco batem com as páginas da memória no caso do DB. Como elas podem ser de tamanhos diferentes do que é usado no OS (para se adequar ao padrão de acesso), faz todo sentido ter um controle próprio.
Existem bancos que preferem deixar o cache de páginas totalmente por conta do OS, outros usam os dois níveis. Isso pode até dificultar a otimização do consumo de memória, já que o dado está em dois locais diferentes, mas pode dar mais segurança e flexibilidade.

Dependendo da implementação do DB não é possível ficar manipulando direto nas páginas do OS. Nem sempre é necessário manipular os dados que o OS pode jogar em disco. Cada implementação tem sua especificidade e vai manipular do jeito que lhe for mais adequado. O DB precisa decidir por conta própria o que deve ir para o "disco" ou não.

Com um algoritmo mais especializado é possível obter mais performance e escalabilidade, além de mais flexibilidade e confiabilidade.

Com um sistema próprio é possível abstrair as condições dos diversos sistemas operacionais. E pode se adaptar quando uma condição não atende mais as expectativas e necessidades.

Coloquei no GitHub para referência futura.


Difference between basename(__FILE__) and $_SERVER['SCRIPT_NAME']?

Question: Is there any difference between basename(__FILE__); and $_SERVER['SCRIPT_NAME']?
Im just seeing people often use one of the 2, and Im unsure if it matters which one to use. I ofcourse tried them both, and they seem to output the same thing. Just wondering. :)
Answer: Yes, there is a definitely a big difference you need take in consideration.
One is used to get the included script full path (current file) and the other is used to get the script path (relative to web server). So you might use one or the other depending of what you try to accomplish.
So it will return a different result in a different format:

__FILE__ : C:\xampp\htdocs\project\myfileincluded.php
$_SERVER['SCRIPT_NAME'] : /project/index.php

__FILE__ is part of the Magic constants and will return the absolute path of the current included file.
$_SERVER['SCRIPT_NAME'] is a part of the Predefined server variables, and will return the path of the script originally called in the server relative to the server root path.
You might want to use __FILE__ when logging messages or when looking to extract the current project name, branch name, etc.. And you might want to use $_SERVER['SCRIPT_NAME'] when you are looking to refresh the page, identify the origin of the call, and so on.
In your example you are using basename(__FILE__) which will in that case return the name of the included file.
Answer: __FILE__ return the script name where this command is originally situated.
$_SERVER['SCRIPT_NAME'] return the script name where this command is called.
For example, if you have "Script_A.php" that call a function "ThisFile()", where "ThisFile()" is in another "Script_B.php" through include_once, then the result will be different.
__FILE__ => return Script_B
$_SERVER['SCRIPT_NAME'] => return Script A
Answer: __FILE__ returns the filename of the script that is currently being run.
dirname returns the directory of a given filename
basename() returms the filename component of path. To put it better: it strips any "parent" directories off a given string.

$_SERVER['SCRIPT_NAME']     Returns the path of the current script

No. Both are same.They will return same thing But i think $_SERVER['SCRIPT_NAME'] is better as it is small and PHP global variable
Answer: Not so relevant but I think it's worth noticing. Be aware that there is also $_SERVER['SCRIPT_FILENAME'].
One difference is they return different paths in included files:

__FILE__ returns path to included file (where the __FILE__ is called).
$_SERVER['SCRIPT_FILENAME'] returns path to script being executed (top level script).

But there could also be a difference on Linux based systems with Apache  webserver where the DocumentRoot is defined through symlinked path.
For example real path to my site is:

/mnt/data/www/mysite.org/

There is also a directory /www/ where lies this symlink:

mysite.org

pointing to /mnt/data/www/mysite.org/.
Then DocumentRoot (Apache's virtual host definition) is configured with symlinked path /www/mysite.org.
In this setting in some PHP root script index.php:

__FILE__ contains /mnt/data/www/mysite.org/index.php - realpath
$_SERVER['SCRIPT_FILENAME'] contains /www/mysite.org/index.php - symlinked path


Scan duplicate document with md5

Question: for some reasons I can't use MessageDigest.getInstance("MD5"), so I must write the algorithm code in manual way, my project is scan duplicate document (*.doc, *.txt, *.pdf) on Android device. My question is, what must I write before entering the algorithm, to scan the duplicate document on MY ROOT directory on Android device? Without select the directory, when I press button scan, the process begin, the listview show. Is anyone can help me? My project deadline will come. Thank you so much.
public class MD5 {

//What must I write here, so I allow to scan for duplicate document on Android root with MD5 Hash

//MD5 MANUAL ALGORITHM CODE
}

Comment: can you share *why* you think you cannot use `MessageDigest` ?
Comment: You cant use MessageDigest, this implementation, or MD5 ? There are other md, like SHA-1, SHAxxx , or even simpler (with more collisions, but perhaps sufficient to detect duplicate documents ...)
Comment: What do you get when you run `java.security.MessageDigest.getInstance("MD5")`?
Comment: my teacher don't allow me to use that. :(
Comment: If you are not up against a malevolent adversary, there are functions _much_ easier to code and compute with _less_ collisions than MD5 (I'm fond of [Fletcher](https://en.wikipedia.org/wiki/Fletcher%27s_checksum) - use 256 bits if the collision rate of MD5 was barely acceptable).
Comment: @greybeard, Thanks for the useful information, I would learn it after this project.
Answer: WHOLE PROCESS:
your goal is to detect (and perhaps store information about) duplicate files.
1 Then, first, you have to iterate through directories and files,
see this:
list all files from directories and subdirectories in Java
2 and for each file, to load it like a byte array
see this:
Reading a binary input stream into a single byte array in Java
3 then compute your MD5 - your project
4 and store this information
Your can use a Set to dectect duplicates (a Set has unique elements).
Set<String> files_hash; // each String is a string representation of MD5
if (files_hash.contains(my_md5)) // you know you have it already

or a
Map<String,String> file_and_hash; // each is file => hash
// you have to iterate to know if you have it already, or keep also a Set

ANSWER for MD5:
read algorithm:
https://en.wikipedia.org/wiki/MD5
RFC: https://www.ietf.org/rfc/rfc1321.txt
some googling ...
this presentation, step by step
http://infohost.nmt.edu/~sfs/Students/HarleyKozushko/Presentations/MD5.pdf
or try to duplicate C (or java) implementation ...
OVERALL STRATEGY
To keep time and have processus faster, you must also think about the use of your function:

if you use it once, for one unique file, better is to reduce work, by selecting before other files on their size.
if you use it regularly (and want to do it fast), scan regularly new files in background to keep an hash base up to date. Detection of new file is straightforward.
if you want to get all files duplicated, better scan everything, and use Set Strategy also

Hope this helps
Comment: It's not very smart to compute the MD5 for all files. The first thing you should do is get a list of all files, sort by size, and remove from the list any files that don't have the same size as some other file. Because two files can't be identical if they're different sizes. That greatly reduces the number of files you have to process. *Then* do the MD5 comparisons, but only compare files that have the same lengths.
Comment: Thank you for your advice and some useful link you've share, I was looking for the algorithm for a week, and I found some useful website too, I understand the algorithm, but I don't understand is what must I write before entering the algorithm, to scan the duplicate document.
Comment: @JimMischel , thank for the smart remark, I append some strategy in my answer.
Comment: @All, Nice, thank you so much, I will try in my project :)
Comment: what should i do next to get duplicate images if i get Set files_hash , after that i am unable to proceed further, please help @guillaumegirod-vitouchkina
Comment: Same images don't return same md5 value.
Answer: You'll want to recursively scan for files, then, for each file found, calculate its MD5 or whatever and store that hash value, either in a Set<...> if you only want to know if a file is a dupe, or in a Map<..., File> if you want to be able to tell which file the current file is a duplicate of.
For each file's hash, you look into the collection of already known hashes to check if that particular hash value is in it; if it is, you (most likely) have a duplicate file; if it is not, you add the new hash value to the collection and proceed with the next file.

Scan duplicate document with md5


Hibernate, JDBC and Java performance on medium and big result set

Question: #Issue#
We are trying to optimize our dataserver application.
It stores stocks and quotes over a mysql database.
And we are not satisfied with the fetching performances.
#Context#
- database
- table stock : around 500 lines
- table quote : 3 000 000 to 10 000 000 lines
- one-to-many association : one stock owns n quotes
- fetching around 1000 quotes per request
- there is an index on (stockId,date) in the quote table
- no cache, because in production, queries are always different
- Hibernate 3
- mysql 5.5
- Java 6
- JDBC mysql Connector 5.1.13
- c3p0 pooling
#Tests and results#
##Protocol##

Execution times on mysql server are obtained with running the generated sql queries in mysql command line bin.
The server is in a test context : no other DB readings, no DB writings
We fetch 857 quotes for the AAPL stock

##Case 1 : Hibernate with association##
This fills up our stock object with 857 quotes object (everything correctly mapped in hibernate.xml)
session.enableFilter("after").setParameter("after", 1322910573000L);
Stock stock = (Stock) session.createCriteria(Stock.class).
add(Restrictions.eq("stockId", stockId)).
setFetchMode("quotes", FetchMode.JOIN).uniqueResult();

SQL generated :
SELECT this_.stockId AS stockId1_1_,
       this_.symbol AS symbol1_1_,
       this_.name AS name1_1_,
       quotes2_.stockId AS stockId1_3_,
       quotes2_.quoteId AS quoteId3_,
       quotes2_.quoteId AS quoteId0_0_,
       quotes2_.value AS value0_0_,
       quotes2_.stockId AS stockId0_0_,
       quotes2_.volume AS volume0_0_,
       quotes2_.quality AS quality0_0_,
       quotes2_.date AS date0_0_,
       quotes2_.createdDate AS createdD7_0_0_,
       quotes2_.fetcher AS fetcher0_0_
FROM stock this_
LEFT OUTER JOIN quote quotes2_ ON this_.stockId=quotes2_.stockId
AND quotes2_.date > 1322910573000
WHERE this_.stockId='AAPL'
ORDER BY quotes2_.date ASC

Results :

Execution time on mysql server : ~10 ms
Execution time in Java : ~400ms

##Case 2 : Hibernate without association without HQL##
Thinking to increase performance, we've used that code that fetch only the quotes objects and we manually add them to a stock (so we don't fetch repeated info about the stock for every line). We used createSQLQuery to minimize effects of aliases and HQL mess.
String filter = " AND q.date>1322910573000";
filter += " ORDER BY q.date DESC";
Stock stock = new Stock(stockId);
stock.addQuotes((ArrayList<Quote>) session.createSQLQuery("select * from quote q where stockId='" + stockId + "' " + filter).addEntity(Quote.class).list());

SQL generated :
SELECT *
FROM quote q
WHERE stockId='AAPL'
  AND q.date>1322910573000
ORDER BY q.date ASC

Results :

Execution time on mysql server : ~10 ms
Execution time in Java : ~370ms

##Case 3 : JDBC without Hibernate##
String filter = " AND q.date>1322910573000";
filter += " ORDER BY q.date DESC";
Stock stock = new Stock(stockId);
Connection conn = SimpleJDBC.getConnection();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("select * from quote q where stockId='" + stockId + "' " + filter);
while(rs.next())
{
    stock.addQuote(new Quote(rs.getInt("volume"), rs.getLong("date"), rs.getFloat("value"), rs.getByte("fetcher")));
}
stmt.close();
conn.close();

Results :

Execution time on mysql server : ~10 ms
Execution time in Java : ~100ms

#Our understandings#

The JDBC driver is common to all the cases.
There is a fundamental time cost in JDBC driving.
With similar sql queries, Hibernate spends more time than pure JDBC code in converting result sets in objects.
Hibernate createCriteria, createSQLQuery or createQuery are similar in time cost.
In production, where we have lots of writing concurrently, pure JDBC solution seemed to be slower than the hibernate one (maybe because our JDBC solutions was not pooled)
Mysql wise, the server seems to behave very well, and the time cost is very acceptable.

#Our questions#

Is there a way to optimize the performance of JDBC driver ?
And will Hibernate benefit this optimization ?
Is there a way to optimize Hibernate performance when converting result sets ?
Are we facing something not tunable because of Java fundamental object and memory management ?
Are we missing a point, are we stupid and all of this is vain ?
Are we french ? Yes.

Your help is very welcome.
Comment: Have you profiled this using Yourkit profiler or a similar product? Where do you lose the most speed? With TCP/IP? With object creation? Note: I doubt that a properly configured JDBC solution would ever be slower than a Hibernate one...
Comment: I agree, the pure JDBC solution was probably suffering lack of pooling.
Comment: The difference in time for the pure JDBC solution could very well be the time that is needed to send the data over the network. The call `rs.getByte("fetcher")` seems to indicate you are transferring BLOB data. How big is that data?
Comment: @a_horse_with_no_name: You meant `rs.getBytes()`? `getByte()` shouldn't cause any issues, I guess?
Comment: @LukasEder: oh! I missed the fact that there was no `s` (blush).
Answer: Can you do a smoke test with the simples query possible like:
SELECT current_timestamp()

or
SELECT 1 + 1

This will tell you what is the actual JDBC driver overhead. Also it is not clear whether both tests are performed from the same machine.

Is there a way to optimize the performance of JDBC driver ?

Run the same query several thousand times in Java. JVM needs some time to warm-up (class-loading, JIT). Also I assume SimpleJDBC.getConnection() uses C3P0 connection pooling - the cost of establishing a connection is pretty high so first few execution could be slow.
Also prefer named queries to ad-hoc querying or criteria query.

And will Hibernate benefit this optimization ?

Hibernate is a very complex framework. As you can see it consumes 75% of the overall execution time compared to raw JDBC. If you need raw ORM (no lazy-loading, dirty checking, advanced caching), consider mybatis. Or maybe even JdbcTemplate with RowMapper abstraction.

Is there a way to optimize Hibernate performance when converting result sets ?

Not really. Check out the Chapter 19. Improving performance in Hibernate documentation. There is a lot of reflection happening out there + class generation. Once again, Hibernate might not be a best solution when you want to squeeze every millisecond from your database.
However it is a good choice when you want to increase the overall user experience due to extensive caching support. Check out the performance doc again. It mostly talks about caching. There is a first level cache, second level cache, query cache... This is the place where Hibernate might actually outperform simple JDBC - it can cache a lot in a ways you could not even imagine. On the other hand - poor cache configuration would lead to even slower setup.
Check out: Caching with Hibernate + Spring - some Questions!

Are we facing something not tunable because of Java fundamental object and memory management ?

JVM (especially in server configuration) is quite fast. Object creation on the heap is as fast as on the stack in e.g. C, garbage collection has been greatly optimized. I don't think the Java version running plain JDBC would be much slower compared to more native connection. That's why I suggested few improvements in your benchmark.

Are we missing a point, are we stupid and all of this is vain ?

I believe that JDBC is a good choice if performance is your biggest issue. Java has been used successfully in a lot of database-heavy applications.
Comment: What do you think about deactivating Hibernate's 2nd level caching (if that is even possible)? Couldn't that speed things up remarkably?
Comment: @LukasEder: Hibernate's 2nd level cache is disabled by default. I believe (although I encourage you to profile this) that most of the time is spent in reflection code in Hibernate - which you can't really avoid.
Comment: I didn't know it was deactivated by default, thanks for the hint. I agree with the need for profiling.