Counting multigraphs up to isomorphism

Question:
Problem: Let $a_m$ denote the number of multigraphs without loops on 4 vertices with $m$ edges (counting up to isomorphism). Find closed formula for the power series $\sum \limits _{m=0}^\infty a_mx^m$.

I just could not come up with a recurrent formula.
I know I could employ the Burnside's theorem. If I considered $S_4$'s action on the set of all suitable multigraphs and calculated the number of graphs fixed by each $\sigma \in S_4$, this would be easy, but the overall solution would be tedious given there are $4!$ elements in $S_4$. (Now this could probably somehow be simplified to discuss only types of permutations in $S_4$, but that's still not quite what I want.)
I cannot guess the solution from the first few members for small $m$. Not even after generalizing this problem to $n$-vertex multigraphs and considering sequence $a_{n,m}$ and looking at small values.
Edit: I am also considering taking the $11$ simple graphs on $4$ vertices and trying to obtain multigraphs by summing them, but I can't get it to work properly.

I am looking for a hint, not for a solution. Ideally, I'd love to obtain the recurrence and apply generating functions to find a closed formula.
Answer: Consult the following link on how to compute the cycle index $Z(G)$ of
the      edge      permutation      group     of      $K_4$:      MSE
link. (We would not
be adding anything here as we would essentially quote it verbatim.) It
was found that
$$Z(G) = \frac{1}{24}
\left(a_1^6 + 8 a_3^2 + 9 a_1^2 a_2^2 + 6  a_2 a_4\right).$$
As a  sanity check  let us compute  non-isomorphic colorings  of $K_4$
using at most $N$ colors. We get using Burnside the count
$$\frac{1}{24}
\left(N^6 + 8 N^2 + 9 N^4 + 6  N^2\right)
= \frac{1}{24}
\left(N^6 + 9 N^4 + 14 N^2\right)$$
which yields the sequence
$$1, 11, 66, 276, 900, 2451, 5831, 12496, 24651, 45475, \ldots $$
which points us to OEIS A063842 where this
sequence  is  described  as  the  number of  multigraphs  rather  than
colorings, a  claim that has  to be investigated.  (Remark, somewhat
later. This  issue has  now been  fixed.) With this  in mind  we now
compute (use PET rather than Burnside for generating functions)
$$[z^n] Z(G)\left(\frac{1}{1-z}\right)$$
which has OGF as requested in the problem statement
$$\frac{1}{24}\frac{1}{(1-z)^6} + \frac{1}{3}\frac{1}{(1-z^3)^2}
+ \frac{3}{8}\frac{1}{(1-z)^2}\frac{1}{(1-z^2)^2}
+ \frac{1}{4}\frac{1}{1-z^2}\frac{1}{1-z^4}.$$
The repertoire  that goes into PET  represents the fact  that there is
exactly one multi-edge consisting of $q$ single edges, yielding an OGF
of $\sum_{q\ge  0} z^q  = 1/(1-z),$ which  includes the off  edge with
weight zero (no edge present between pair of vertices).
This yields the sequence
$$1, 3, 6, 11, 18, 32, 48, 75, 111, 160, 224, 313,
420, 562, 738, \ldots$$
which points us  to OEIS A003082. This has
a perfect match to the problem  definition and it looks like the first
entry needs to be qualified or possibly even corrected.
The values  look good,  e.g. the six  multigraphs with  three edges
are: a path, a tree with three leaves, a triangle, a double edge and a
single edge, not  connected, a double edge and  a single edge attached
at one of the vertices of the double one and a triple edge between two
vertices.

Extracting coefficients  is not terribly rewarding here  but we may
use     the     Maple     code     from     the     following     MSE
link  to  get  the
set   of   polynomials   (period    due   to   roots   of   unity   is
$\mathrm{lcm}(2,3,4) = 12):$

               5          4   13   3         2   1309     53
  n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + ---- n + ---
                              288                2880     192

                5          4   13   3         2   203     13
   n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + --- n + --
                               288                360     24

                5          4   13   3         2   181     39
   n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + --- n + --
                               288                320     64

                5          4   13   3         2   203
   n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + --- n + 2/3
                               288                360

               5          4   13   3         2   1309     53
  n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + ---- n + ---
                              288                2880     192

                5          4   13   3         2   27
   n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + -- n + 7/8
                               288                40

               5          4   13   3         2   1309     53
  n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + ---- n + ---
                              288                2880     192

                5          4   13   3         2   203
   n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + --- n + 2/3
                               288                360

                5          4   13   3         2   181     39
   n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + --- n + --
                               288                320     64

                5          4   13   3         2   203     13
   n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + --- n + --
                               288                360     24

               5          4   13   3         2   1309     53
  n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + ---- n + ---
                              288                2880     192

                 5          4   13   3         2   27
    n -> 1/2880 n  + 1/192 n  + --- n  + 7/32 n  + -- n + 1
                                288                40

The Maple code for this goes as follows:

HZ := 1/24*1/(1-z)^6+1/3*1/(1-z^3)^2
+3/8*1/(1-z)^2*1/(1-z^2)^2+1/4*1/(1-z^2)*1/(1-z^4);

PSEQ :=
proc()
    option remember;
    local n, lambda, offs, res, locs, vals, cfs;

    res := [];
    lambda := lcm(2,3,4);

    cfs := series(HZ, z=0, 6*lambda+2);

    for offs from 0 to lambda-1 do
        locs := [seq(offs+p*lambda, p=0..5)];
        vals := map(loc -> coeff(cfs, z, loc), locs);

        res :=
        [op(res),
         unapply(interp(locs, vals, n), n)];
    od;

    res;
end;

X :=
proc(n)
    local Fseq, lambda;

    Fseq := PSEQ();
    lambda := nops(Fseq);

    Fseq[1+(n mod lambda)](n);
end;

Addendum.  Some  additional  material  which may  perhaps  inspire
further exploration  of these types  of problems.  Here is  a
routine  for  computing  the  cycle  indices  of  the  edge
permutation group  of the complete graph  $K_n.$ It is  about twice as
fast    as    the   routine    posted    at    the   following    MSE
link, which includes
some explanatory  material. We obtain  e.g. the following  cycle index
for $K_6:$
$$Z(G_6) =
{\frac {{a_{{1}}}^{15}}{720}}+1/48\,{a_{{1}}}^{7}{a_{{2}}}^{4}+1
/18\,{a_{{1}}}^{3}{a_{{3}}}^{4}+1/12\,{a_{{1}}}^{3}{a_{{2}}}^{6}
\\+1/4\,a_{{1}}a_{{2}}{a_{{4}}}^{3}+1/6\,a_{{1}}a_{{2}}{a_{{3}}}^{
2}a_{{6}}+1/5\,{a_{{5}}}^{3}\\+1/18\,{a_{{3}}}^{5}+1/6\,a_{{3}}{a_
{{6}}}^{2}.$$
This can of course be used to count multigraphs. Working with ordinary
graphs we obtain the generating function
$$Z(G_6)(1+z) =
{z}^{15}+{z}^{14}+2\,{z}^{13}+5\,{z}^{12}+9\,{z}^{11}+15\,{z}^{
10}+21\,{z}^{9}\\+24\,{z}^{8}+24\,{z}^{7}+21\,{z}^{6}+15\,{z}^{5}+
9\,{z}^{4}+5\,{z}^{3}+2\,{z}^{2}+z+1$$
for a total  of $156$ graphs.  E.g. the  nine non-isomorphic graphs on
six  vertices with  four edges  would appear  to be,  singletons being
omitted: the  path on five vertices,  a tree with four  leaves, a tree
with three leaves, a square, a  triangle with a node attached to it, a
path on  four nodes and  one on two  nodes, a triangle and  a detached
connected pair, a tree with three leaves and a connected pair, and two
paths on three nodes.
The  code follows.  Do  consult it  to clarify  the details  of the
technique being used, it is  included here in place of introducing new
notation to describe the algorithm.

pet_cycleind_symm :=
proc(n)
option remember;

    if n=0 then return 1; fi;

    expand(1/n*add(a[l]*pet_cycleind_symm(n-l), l=1..n));
end;

pet_cycleind_edg :=
proc(n)
option remember;
local all, term, termvars, res, l1, l2, inst1, u, v,
    uidx, vidx;

    if n=0 or n=1 then return 1; fi;

    all := 0:
    for term in pet_cycleind_symm(n) do
        termvars := indets(term); res := 1;

        # edges on different cycles of different sizes
        for uidx to nops(termvars) do
            u := op(uidx, termvars);
            l1 := op(1, u);

            for vidx from uidx+1 to nops(termvars) do
                v := op(vidx, termvars);
                l2 := op(1, v);

                res := res *
                a[lcm(l1, l2)]
                ^((l1*l2/lcm(l1, l2))*
                  degree(term, u)*degree(term, v));
            od;
        od;

        # edges on different cycles of the same size
        for u in termvars do
            l1 := op(1, u); inst1 := degree(term, u);
            # a[l1]^(1/2*inst1*(inst1-1)*l1*l1/l1)
            res := res *
            a[l1]^(1/2*inst1*(inst1-1)*l1);
        od;

        # edges on identical cycles of some size
        for u in termvars do
            l1 := op(1, u); inst1 := degree(term, u);
            if type(l1, odd) then
                # a[l1]^(1/2*l1*(l1-1)/l1);
                res := res *
                (a[l1]^(1/2*(l1-1)))^inst1;
            else
                # a[l1/2]^(l1/2/(l1/2))*a[l1]^(1/2*l1*(l1-2)/l1)
                res := res *
                (a[l1/2]*a[l1]^(1/2*(l1-2)))^inst1;
            fi;
        od;

        all := all + lcoeff(term)*res;
    od;

    all;
end;

pet_varinto_cind :=
proc(poly, ind)
local subs1, subs2, polyvars, indvars, v, pot, res;

    res := ind;

    polyvars := indets(poly);
    indvars := indets(ind);

    for v in indvars do
        pot := op(1, v);

        subs1 :=
        [seq(polyvars[k]=polyvars[k]^pot,
             k=1..nops(polyvars))];

        subs2 := [v=subs(subs1, poly)];

        res := subs(subs2, res);
    od;

    res;
end;

VGF :=
proc(n)
option remember;
    expand(pet_varinto_cind(1+z, pet_cycleind_edg(n)));
end;

Remark. I only just now noticed  that the OP was asking for a hint
rather than a  solution. It is hoped that there  are enough details to
be filled in here for it to be instructive to assemble a comprehensive
answer.
Addendum Nov 18  2018. The result for the number of ordinary graphs is shown below. It is perfectly sufficient to use the Maple functions
degree and indets  to implement an interface  to the monomials
from the cycle index as multisets of cycles. This is shown below:
$$1, 2, 4, 11, 34, 156, 1044, 12346, 274668, 12005168,
\\ 1018997864, 165091172592, 50502031367952,
\\ 29054155657235488, 31426485969804308768,
\\ 64001015704527557894928, \ldots $$
which points us to OEIS A000088. The Maple code is (Burnside)

Q := proc(n)
local cind, v;
option remember;

    cind := pet_cycleind_edg(n);
    subs([seq(v = 2, v in indets(cind))], cind)
end proc;

Counting multigraphs up to isomorphism


Can't Repeat Column Groups AND ALSO hide Static Columns within a Matrix

Question: UPDATE
When I first asked this question, I didn't fully understand what the problem was.  Your best bet is to glance over the issue below and then read my answer.

I have a report with a matrix where the data looks like the following:
<code>Name    Id   Activity 1     Activity 2     …Acitivity N
Smith   1    77             100             nn
Johnson 2    88             99              nn
</code>
Name and Id are in a group.
When the number of activities are greater than 11, I need the columns Name and Id to repeat when the report renders in a PDF.

Here is an example of report recently run:

Page 1 (NOTE:  red boxes indicate
  personal data filtered out):

Page 2:

I've been fiddling with the properties RepeatRowHeaders and RepeartColumnHeaders, but I have had no success.
How do I make this column repeat?
Sadists can check out the rdl file here.
Answer: Short Answer (and a plea from me):
Please vote on this Page to conditionally hide non-group matrix columns.

Long Form:
I found the solution to my problem, but first was figuring out the exact issue.  An SSRS Matrix has the ability to repeat columns on subsequent pages as its width grows when RepeatRowHeaders and RepeatColumnHeaders are set to True.  So that part of my report setup was fine.
My problem was that my grid was set up as follows:

What you see above is a Tablix containing a Matrix.  The blue represents a normal Tablix with four columns.  The green Represents a Matrix within Column 4 that has N number of columns.
So even with the properties RepeatRowHeaders and RepeatColumnHeaders set to true, Columns 1-3 would not repeat as the data in the Matrix extended to multiple pages.  This is because the outer Tablix (e.g. the blue part) only saw Column 4  as growing outwards instead of the individual columns in the inner matrix.
In order to make the columns repeat, I had to alter the grid like so:

This grid is only one matrix; not a Matrix within a Tablix as the first image depicted.  I pulled the matrix out and added "Static Columns" to the left.  This made Columns 1 - 3 repeat on multiple pages as the Matrix grew.
All seemed well, but here's the rub:  Static Columns cannot be hidden or, more aptly put, you can hide the data within a column, but the column itself will not shrink.
So, when I hid Static Column 3, my report looked something like this:

Notice the white-space Gap.  This solution simply would not work.  When I found that I was stuck in an either/or situation (either have the ability to conditionally hide columns or Repeat Columns on Subsequent Pages), I used one of our Microsoft Support Incidents to analyze the report and come up with a solution.
Microsoft was nice and knowledgeable and even pointed to a article in their forum stating they were considering allowing SSRS to have the ability to hide static columns.  However, in the end, they said we were stuck with the either/or situation.  (BTW, note that forum thread was answered in 2006.)
So after that long process with Microsoft, we were still stuck with our dilemma.  Fortunately, one of our fellow programmers came up with a workaround:  put the columns you want to toggle visibility  within the Matrix itself.
And that's what I did.  It took some finagling with the report SQL (and creating a statement I'm not particularly proud of), but it works.
So now the working report looks like this:

That's one Matrix, two static columns that always need to repeat on every page and Column 3 which is now part of the dataset used for the Matrix.
Once again, I plead with the community to vote on this Page to conditionally hide non-group matrix columns.  It would make constructing future reports like this much easier :)


XMLSerializer Deserialization - array element order

Question: We are using XMLSerializer.Deserialize(XMLReader) on .Net 4.5 to deserialize XML into an Object Graph generated by xsd.exe.
I would like to know what the expected deserialization behavior is for an array which is annotated with the XMLElementAttribute - specifically with regard to ordering.  For example:
For the following property:
<code>[System.Xml.Serialization.XmlElementAttribute("GivenName")]
  public string[] GivenName {
 // get() and set() methods
}
</code>
And the following XML:
<code><root>
  <GivenName>One</GivenName>
  <GivenName>Two</GivenName>
  <GivenName>Three</GivenName>
</root>
</code>
Will this always deserialize as ['One', 'Two', 'Three']
So that the array order always matches the XML order
Also is there any documentation I can reference that clearly states this.
Thanks
Rob
Answer: Yes Its deserialize same order. Check the code sample code and output:

<code> [System.SerializableAttribute()]
public class SampleClass
{
    [System.Xml.Serialization.XmlElementAttribute(Order = 10)]
    public string Foo { get; set; }
    [System.Xml.Serialization.XmlElementAttribute(Order = 5)]
    public string Bar { get; set; }
    [System.Xml.Serialization.XmlElementAttribute("GivenName", Order = 15)]
    public string[] GivenNames { get; set; }
}

class Program
{
    static void Main(string[] args)
    {
        string[] names = new string[3] { "One", "Two", "Three" };

        SampleClass TestObj = new SampleClass { Bar = "dsdfsdf", Foo = "test", GivenNames = names };
        XmlSerializer SerializerObj = new XmlSerializer(typeof(SampleClass));
        // Create a new file stream to write the serialized object to a file
        TextWriter WriteFileStream = new StreamWriter(@"C:\files\test.xml");
        SerializerObj.Serialize(WriteFileStream, TestObj);
        // Cleanup
        WriteFileStream.Close();

        // Create a new file stream for reading the XML file
        FileStream ReadFileStream = new FileStream(@"C:\files\test.xml", FileMode.Open, FileAccess.Read, FileShare.Read);

        // Load the object saved above by using the Deserialize function
        SampleClass LoadedObj = (SampleClass)SerializerObj.Deserialize(ReadFileStream);

        // Cleanup
        ReadFileStream.Close();

    }
}
</code>
Comment: Please see my comment below.  I am after an explicate documentation reference that states the order is retained.  The cost of loosing order in my situation (even in a small number of scenarios) is high.  I also have many test cases that demonstrate the ordering is generally respected, but I am after some amount of certainty whether it is in all cases or not.
Answer: You can find some awesome information about it on msdn which is provide by microsoft.
This is the "official" documentation
The serialization is linear so you will alway get the same order normally.
Comment: I've looked through the documentation and can't find anything that states specifically the element order is respected when translated into an array.

In my situation there is a significant amount of business risk associated to the order being dropped even if it occurs in a small number of cases.  Consequently I am after a specific reference - (even if it is code) that either states the order is respected (or isn't)

I acknowledge the correct solution is to explicitly order with an ordinal attribute, but I do not own the interface and the cost of change is high.
Comment: System.Runtime.Serialization Namespace
https://msdn.microsoft.com/en-us/library/system.runtime.serialization(v=vs.110).aspx
The order of deserialization relative to other objects in the graph is non-deterministic.
This is the only thing found.


Guitar sounds distorted only when tone knob is turned on

Question: i've had this Jackson 7 string equipped with EMG's for several years and just got this problem even after changing the 9 volt battery. When i roll back the tone all the way, it sounds clean and fine but anything more than that and the signal seems overloaded. Any ideas what the problem might be?
Edit: At first i plugged straight into my Eleven Rack which goes through studio monitors that have no issues. Then i plugged into my Fender combo and am using a brand new cable. I know for sure it's the guitar.
The tone knob on my guitar has nothing to do with the volume (as far as i know? There's no difference in level of volume and i believe it works as a high pass filter). I think it must either be an issue with my active pickups (perhaps the signal is overloading the amplifier), or a wiring issue with the tone potentiometer since i'm only having issues when the tone is above zero.
Comment: If your tone pot is acting like a volume adjustment, then you probably have a short, a bad capacitor, or other broken wiring problem. Didn't make an answer because it is not completely clear if your tone is adjusting volume.
Comment: Can you edit your question to add in the rest of your signal chain and as many settings as possible? If you have a huge number of pedals or effects in your chain, I suggest your first step is to eliminate them all and plug right into the amp. If you just have the amp, then please edit to add what make and model amp it is and what settings you have for each of the knobs.
Comment: I would change the pots first since it is a (relatively) cheap and (relatively) easy upgrade.
Answer: This is normal for active pickups. Just turn down your gain, if that doesn't help something is wrong with your boost in the guitar. I use 3 9V batteries in my Schecter JL 7 String and it works just fine after lowering the gain a bit.
Comment: This may be true.  A lot of the distortion is in the highs.  By dialing up the tone, the OP is putting the highs back in to the sound and, by proxy, the distortion also.  Turning down the gain on the amp should take care of it.
Answer: Is this still an issue?  The tone knob is a low-pass filter.  At 10, the tone should be bright.  At 10, it is almost as if the tone circuit wasn't there.  The fact that it is crackling leads me to believe their may be an issue with the pickup.
Assuming that you have a known good battery, I would try the following:

Verify that it is happening on all pickups.  It isn't clear to me if it is just one or all.  This might isolate it to just one pickup.  If it's both pickups, then it's likely the tone control.  If it's just one pickup, then it's likely the pickup.
Verify no loose or almost loose wiring (maybe a loose ground wire or something) or bad solder jobs.  This is actually not uncommon.
Try removing the tone knob from the circuit entirely.  It's not needed to get a sound.  See if the problem goes away.  If so, then bad pot or cap or wiring.

Even if it is no longer a problem, I would love to hear what the resolution was.


Removing history or line numbers from ZSH history file

Question: For some reason I get these odd line numbers or history numbers of some sort into my ~/.zsh_history file like so:
: 1266694881:0;rake db:migrate

I'd rather not have to resort to regexing out the beginning, but I'd like my lines to be something like:
rake db:migrate

Does anyone have any suggestions for how to solve this? I've found setopt EXTENDED_HISTORY but it doesn't seem to do much.
Comment: `grep $1 ~/.zsh_history | cut -c 16-`
Comment: Maybe I can just remove those first 15 characters easily with regex... uhh.. any thoughts.
Comment: I don't understand everything : do you have these numbers on every lines ? Do you know what these numbers are or do you only want to remove them ?
Comment: I actually don't know what they are for, if you are using ZSH, check it out in `~/.zsh_history` I'd rather ZSH not save that portion with the history at all, my interim solution for searching history is as such `cat ~/.zsh_history | grep $1 | sed "s/...............//"` But this is ridiculous.
Comment: I don't have any of these numbers in my zsh history. What version of zsh are you running ? what is the content of your .zshrc ?
Answer: Try:
unsetopt EXTENDED_HISTORY

The numbers are the number of seconds since the Unix epoch of when the command began and the duration in seconds that the command ran.
Edit:
I forgot to mention that you should use the fc command to interact with history instead of parsing the history file. There are at least a couple of reasons for this. One is that the history file doesn't have the in-memory entries until you exit the shell, a threshold is met or you explicitly write it with the fc -AI command (I believe). Secondly, you can leave EXTENDED_HISTORY turned on and still interact with the entries without having to remove that information. If it's off, then when you do fc -ld, the shell uses the time that the shell started for the timestamp for commands in the history before that time. If it's off, it remembers the actual date and time (the numbers you're seeing in the file).
See man zshbuiltins.
List the most recent entries:
fc -l

List them without command numbers:
fc -ln

List the most recent 20 commands without command numbers:
fc -ln -20

Include a timestamp and print only the most recent command:
fc -lnd -1

Show all the commands (within the last 50) that include the string "setop" (shows setopt and unsetopt):
fc -l -m '*setopt*' -50

Comment: yah I tried this it doesn't seem to affect anything.
Comment: Does it show "extendedhistory" among others when you enter the `setopt` command without any arguments at the shell prompt?
Comment: Fantastic! Thanks, that solves alot of things, and it also shows incomplete lines that were never sent from ZLE, that's sort of amazing.
Answer: Add to ~/.zshrc:
setopt no_extended_history no_share_history

Add to ~/.bashrc, ~/.config/fish/config.fish, etc for other shell.


Why is $E[Z|Z>c] = \int_c^\infty z_i \phi({z_i})\mathrm{d}z_i $ ($Z$ is censored)

Question: In a problem set I proved this "lemma," whose result is not intuitive to me. $Z$ is a standard normal distribution in a censored model.
Formally,
$Z^* \sim Norm(0, \sigma^2)$, and $Z = max(Z^*, c)$. Then,
\begin{align}
E[Z|Z>c]
&= \int_c^\infty z_i \phi({z_i})\mathrm{d}z_i \\
&= \frac{1}{\sqrt{2\pi}}\int_c^\infty z_i \exp\!\bigg(\frac{-1}{2}z_i^2\bigg)\mathrm{d}z_i \\
&= \frac{1}{\sqrt{2\pi}} \exp\!\bigg(\frac{-1}{2}c^2\bigg) \quad\quad\quad\quad\text{ (Integration by substitution)}\\
&= \phi(c)
\end{align}
So there is some sort of connection between the expectation formula over a truncated domain and the density at the point of truncation $(c)$. Could anyone explain the intuition behind this?
Comment: That it turns out that way is a consequence of the fact that the $z$ term is the negative of the derivative of the term in the exponent; it's one of many neat results for the standard normal but it doesn't necessarily have intuition behind it. On the other hand it wouldn't surprise me at all if one of the clever people here could come up with some kind of intuition for it.
Comment: Your latest edit asks for a proof (or intuitive explanation) of an incorrect statement.  The _conditional_ density of $Z \sim N(0,1)$ conditioned on $Z > c$ is $\frac{\phi(z)}{1-\Phi(c)}\mathbf 1_{\{z\colon z > c\}}$ and the _conditional_ expected value is thus $$E[Z\mid Z > c] = \int_c^{\infty}z\frac{\phi(z)}{1-\Phi(c)}\,\mathrm dz = \frac{1}{1-\Phi(c)}\int_c^{\infty}z\phi(z)\,\mathrm dz$$ and not what you have in your revised title.
Comment: @Glen_b What you are saying is that $\int_c^\infty \left(-\frac{d}{dz}\log(f(z))\right) f(z) dz = -\int_c^\infty f^\prime(z)dz = f(c)$ where $f$ is the PDF of *any* continuous distribution $F.$
Comment: @whuber That's certainly the case, and it's worth emphasizing that result, since it's directly relevant to the result in the question, but actually in my comment I was referring specifically to the case where the first of those terms is $z$ (since the term "the expectation formula" was in the question, I took it to be about $E(Z|Z>c)$, which is specific to the normal.
Comment: (at least up to the obvious multiplicative constant, about that conditional expectation). However, $E(g(Z)|Z>c)$ for that particular $g=-\frac{d}{dz}log f$ is probably worth discussing in an answer.
Comment: @DilipSarwate you're correct. My latest edit made it seems like a truncated normal, in which the density must be divided by $1-\Phi(c)$ as you suggested. In fact, the problem is about a censored model, in which the density of $Z$ is still standard normal. I'll add details.
Comment: Almost the same question--with the same solutions--appears at http://stats.stackexchange.com/questions/166273.
Answer: Would the Fundamental Theorem of Calculus work for you as intuition?
Let $\phi(x)$ denote the density function $\frac{1}{\sqrt{2\pi}}e^{-x^2/2}$
of a standard normal random variable.  Then, the derivative is $\frac{\mathrm d}{\mathrm dx}\phi(x) = -x\phi(x)$. The Fundamental Theorem of Calculus then gives
us that
$$\phi(x) = \int_{-\infty}^x -t\phi(t)\,\mathrm dt = \int_{-x}^\infty u\phi(u)\,\mathrm du = \int_x^\infty u\phi(u)\,\mathrm du$$
where the second integral is obtained on substituting $u = -t$ and using the
fact that $\phi(-u) = \phi(u)$ and the third upon noting that $\phi(-x) = \phi(x)$.
Alternatively, write the second integral as the integral from $-x$ to $+x$ plus
the integral from $+x$ to $\infty$, and note that integrating an odd function
from $-x$ to $+x$ results in $0$.


Complexity analysis of logarithms

Question: I have two functions, f(n)=log(base 2)n and g(n)=log(base 10)n. I am trying to decide whether f(n) is O(g(n)), or Ω(g(n)) or Θ(g(n)). I thinks i should take the limit f(n)/g(n) as n goes to infinity, and i think that limit is constant so f(n) must be Θ(g(n)). Am i right? Thanks in advance
Comment: The quantity $f(n)/g(n)$ is constant, that is, independent of $n$.
Answer: You can rely on a useful and often-forgotten result, that $(\log_a b)(\log_b c) = \log_a c$, so in your problem we'll have
$$
(\log_2 10)g(n) = (\log_2 10)(\log_{10} n) = \log_2 n = f(n)
$$
so $f(n)$ is just a constant multiple ($\log_2 10\approx 3.32$) of $g(n)$ and hence $f(n)=\Theta(g(n))$.

Complexity analysis of logarithms


Logic issue at the end of a loop

Question: I have a spreadsheet with a column filled with strings. There are sometimes several instances of one string (I've sorted it though so they're all grouped together) so I've created a macro that looks at the current row in the column, determines if its already been posted in the second worksheet, and if it has, it moves to the next row, if it hasn't it posts it and then moves on.
It works perfectly until the end where it completely ignores the last entry. The last entry is one of the instances where there's several of the same strings so the issue almost definitely isn't that its just ignoring the last row.
It's ignoring the last entry completely no matter how many times it repeats. New to vba so any help appreciated.
Sub RCFS()
    Dim ProfCtr As String
    Dim S2FreecellH As Long
    Dim ProfCenCellH As Long

    S2FreecellH = 3
    ProfCenCellH = 2
    ProfCtr = Cells(ProfCenCellH, 4)
    Worksheets("Sheet2").Cells(S2FreecellH, 1).Value = ProfCtr

    While IsEmpty(Cells(ProfCenCellH, 4).Value) = False
        If Cells(ProfCenCellH, 4).Value <> ProfCtr Then
            Worksheets("Sheet2").Cells(S2FreecellH, 1) = ProfCtr
            S2FreecellH = S2FreecellH + 1
            ProfCtr = Cells(ProfCenCellH, 4)
            ProfCenCellH = ProfCenCellH + 1
        Else
            ProfCenCellH = ProfCenCellH + 1
        End If
    Wend
End Sub

Comment: Are you trying to copy the unique values in Sheet1 column D to Sheet2 Column A?  If so is there any preexisting data Sheet2 Column A?
Answer: Indeed, your loop needs to still flush the value of ProfCtr even when you reached the empty cell. You can make it work by looking ahead one row in your If condition (add + 1).
Also included a couple of other improvements:
Do While Not IsEmpty(Cells(ProfCenCellH, 4).Value)
    If Cells(ProfCenCellH + 1, 4).Value <> ProfCtr Then
        Worksheets("Sheet2").Cells(S2FreecellH, 1) = ProfCtr
        S2FreecellH = S2FreecellH + 1
        ProfCtr = Cells(ProfCenCellH, 4)
    End If
    ProfCenCellH = ProfCenCellH + 1
Loop

Answer: Let me see if I have this right
    ' Stop when you reach an empty cell
    While IsEmpty(Cells(ProfCenCellH, 4).Value) = False

    ' If the cell's value matches the variable ProfCtr then
    ' ignore it and move to the next row.
    If Cells(ProfCenCellH, 4).Value <> ProfCtr Then

       ' But if it -doesn't- match, then copy the variable
       ' ProfCtr to Sheet 2.
        Worksheets("Sheet2").Cells(S2FreecellH, 1) = ProfCtr

       ' Increment your rows
        S2FreecellH = S2FreecellH + 1

       ' Set the next bit of text to copy
        ProfCtr = Cells(ProfCenCellH, 4)
        ProfCenCellH = ProfCenCellH + 1
    Else
        ProfCenCellH = ProfCenCellH + 1
    End If

Shouldn't you be setting Prof Ctr = Cells(ProfCenCelH, 4) BEFORE doing the copying over to Sheet2?
Answer: trincot has the correct answer.  Here is an alternate technique, using built-in method, that will accomplish the same task.
With Worksheets("Sheet1")
    .Range("D1", .Range("D" & .Rows.Count)).AdvancedFilter Action:=xlFilterCopy, CopyToRange:=Sheet2.Range("A1"), Unique:=True
End With

Logic issue at the end of a loop


Is the following function twice differentiable?

Question: Let $\alpha : I \rightarrow \mathbb{R}^3$ be a twice differentiable curve such that $\alpha ' $ and $\alpha ' ' $ are nonzero everywhere. Let $$s(t):=\int_{t_0}^{t}|\alpha ' (u)|du$$ and let $\beta := \alpha \circ s$. Then $\beta$ is an arc-length parametrization of $\alpha$. Is $\beta$ twice differentiable?
Comment: Use chain rule $$\frac{d}{ds} = \frac{1}{\frac{ds}{dt}} \frac{d}{dt}$$
Comment: I meant to write $\beta := \alpha \circ s^{-1}$, where $s^{-1}$i is the inverse of $s$. As I have written it, $\beta := \alpha \circ s$ is not an arc-length parametrization of $\alpha$. This all being said, it was my mistake and I think it's better to make a new question as opposed to editing this one.
Answer: Yes, it is. One differentiation gives
$$\beta'(t)=\alpha'(s(t))\cdot s'(t)=\alpha'(s(t))|\alpha'(t)|$$
The function $t\to \alpha'(s(t))$ is differentiable because $\alpha$ is twice differentiable, and the function $|\alpha'(t)|$ can be written as
$$\sqrt{\alpha_1'(t)^2+\alpha_2'(t)^2+\alpha_3'(t)^2}$$
where $\alpha(t)=(\alpha_1(t),\alpha_2(t),\alpha_3(t))$. Since $\alpha'(t)$ is nonzero everywhere, its norm is a differentiable function. Hence the product of $\alpha'(s(t))$ and $|\alpha'(t)|$ is also differentiable, so $\beta$ is twice differentiable.


Read data from Buffer

Question: How can i access the data from a buffer?
int WINAPI mRecv( SOCKET s, char *buf, int len, int flags ) {
    // Code to access the data from buffer
}

Comment: `buf[n]` to extract the nth byte. But make sure you don't go out of bounds.
Answer: You are probably looking for std::istream::read. It is used to extract characters from stream.
And if you want to iterate over the data then you can use the std::istreambuf_iterator
std::string
getBufferContent(const std::string& path)
{
  std::ifstream file(path);
  std::string content((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
  return content;
}

Read data from Buffer


How to use nested template tags with arguments?

Question: I would like to learn how I can use nested template tags where the child template tag takes an argument like below:
<code>{% parent_tag {% child_tag arguments %} rest_of_parent_arguments %}
</code>
In the above line, I would like to use the return value of child tag as an argument to the parent tag.
Answer: I would replace the child_tag with a custom filter. Something like:
<code>{% parent_tag argument1|filtername:argument2 rest_of_parent_arguments %}
</code>
assuming "arguments" consists of at most 2 arguments. See here for custom filters:
http://docs.djangoproject.com/en/dev/howto/custom-template-tags/#writing-custom-template-filters
Comment: thanks for your answer. It's the exact solution for my question.


Black band with 100% width gets a gap when table gets min-width

Question: I have a band on top and a table on the bottom.
When the table is without min-width the band (with 100% width) works perfectly covering 100% of the screen, however, when I add the min-width:950px to the table I get a gap I can't get rid of on the right side of the screen whenever the screen resizes to a smaller size.
I've looked around but I haven't been able to fix it with any of the recommendations here. Any help appreciated
fiddle: http://jsfiddle.net/xxVUW/
<code>#band{width:100%;height:100px;background: black;}
#port-content{min-width:950px;}
</code>
Answer: It is unclear exactly what you're asking. If you mean to say that by adding a table width, the "band" div no longer matches the width of the table, then all you need to do is set the width of the "band" div to be equal to the width of the table.
Updated fiddle: http://jsfiddle.net/xxVUW/2/
<code>#band{min-width:950px; width:100%;height:100px;background: black;}
#port-content{min-width:950px; border:1px solid red;}
</code>
EDIT: Added "min-width" attribute to "band" class
Comment: it almost is what you're saying (I'll update my answer) except I need the black band to be 100% width to cover the entire screen all the time so it can't match the table's width
Comment: I added the "min-width" attribute to the "band" class. That should allow it to be at least 950px, but also stretch to 100% if the page width is larger.


How do I insert a close button inside my popup

Question: I am generating a popup using b.popup, I want to include a close button "x" inside the popup but its not working.
I have used it as follows:
I dont have jquery installed hence I have to use the following script headers


jQuery .append javascript doesn't work

Question: snip of html

when user scroll down I add more items (div id="item") to div "list". I get more items using
$.get("/test/items/",
     function (data) {
        if (data != '') {
            $('#list').append(data);
        }
     }
)

returned data contains javascript and looks like this.

but after doing append I dont see javascript did anything. in original html as mentioned before works fine.
Comment: list is id... but ur selector takes it as a class.. is this a typo ' $('.list').append(data);' ????
Comment: yea it was typo :) thank
Answer: Try This one
$.get("/test/items/",
     function (data) {
        if (data != '') {
            $('#list').append(data);
         }

Comment: Even if the op were to add the script this way, would it still execute on append? This is for my curiosity
Comment: @ryadavilli What do you mean? This should append `data` to `#list`.
Comment: @Jan Dvorak, In the original post, it was mentioned that "but after doing append I dont see javascript did anything". I assumed this meant, that after the script got appended, the OP expected the newly added script to execute and perform some action. In the first example, the OP has a
Comment: @ryadavilli please read "but after doing append" as "but after having attempted append" ;-) . The `.list` most likely didn't select anything.
Comment: it was typo (my question still holds :). @ryadavilli - you understood question correct) . It should be $('#list').append(data); instead of $('.list').append(data);
Answer: It's not clear what you're trying to do here. You're making a request to get HTML, and in that html is a script reference. Are you trying to load a script?
If so, you should have your server return simply the url, not the entire HTML. You can then use jQuery to get a script and execute it.
// This GET should return, "http://email@example.com&time=today"
$.get("/test/items/",
  function (data) {
    $.getScript(data);
  }

The getScript() function is just shorthand for an AJAX call, and you can supply callbacks for success, failure etc.
Comment: Even if the op were to add the script this way, would it still execute on append? This is for my curiosity
Comment: @user1715729 - Oh ok.. Well I changed my answer. I don't know if it helps but it's completely different now that I know it was a typo.

jQuery .append javascript doesn't work


How can i install mysql_config on Red Hat Linux 4?

Question: I've been tasked with implementing a few upgrades on a Red Hat Linux 4 box that is already running MySQL.  I've found the need for the mysql_config tool, but i can't locate it as already being installed anywhere.
The package manager i'm using is RPM.
rpm -qa | grep -i mysql
MySQL-client-enterprise-gpl-5.0.56sp1-0.rhel4
MySQL-shared-compat-enterprise-gpl-5.0.56sp1-0.rhel4
MySQL-server-enterprise-gpl-5.0.56sp1-0.rhel4

Here's some info about the MySQL installation:
mysql> SHOW VARIABLES LIKE '%version%';
+-------------------------+-------------------------------+
| Variable_name           | Value                         |
+-------------------------+-------------------------------+
| protocol_version        | 10                            |
| version                 | 5.0.56sp1-enterprise-gpl-log  |
| version_comment         | MySQL Enterprise Server (GPL) |
| version_compile_machine | i686                          |
| version_compile_os      | redhat-linux-gnu              |
+-------------------------+-------------------------------+

I've done a bit of looking around online, but not having much luck.
How can i install mysql_config on this box?
Comment: 4? Isn't that way out of support?
Comment: Indeed :(.  My pleas for upgrade have been heard, and the wheels of IT beauracracy are moving, but slowly, and i need to move forward.
Answer: Since mysql_config is a developer tool, it will probably be in the equivalent devel package. You seem to have obtained these packages directly from Oracle, so ask Oracle to provide you with the equivalent MySQL-devel-enterprise-gpl package.
(Oh, and that's about 40 patches out of date; it's on 5.0.96 now.)
(And RHEL 4 went out of support two years ago...)


Update Service status when service instance is shutdown

Question: I implemented request using Spring Cloud Feign client. I tried this:
Feign client:
@FeignClient(name = "mail-service")
public interface EmailClient {

    @RequestMapping(method = RequestMethod.POST, value = "/register")
    void setUserRegistration(RegisterUserDTO registerUserDTO);

    @RequestMapping(method = RequestMethod.POST, value = "/password_reset")
    void setUserPasswordReset(PasswordResetDTO passwordResetDTO);
}

Feign configuration:
@Configuration
public class LoadBalancerConfiguration {

    @Bean
    public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) {
        return ServiceInstanceListSupplier.builder()
                .withBlockingDiscoveryClient()
                .withSameInstancePreference()
                .withHealthChecks()
                .build(context);
    }
}

Request DTO:
@Getter
@Setter
public class PasswordResetDTO {

    private int id;
}

Controller:
@Autowire
EmailClient emailClient;

@PostMapping("/dummy")
public ResponseEntity<?> test() {

    RegisterUserDTO obj = new RegisterUserDTO();
    obj.setId(12);

    emailClient.setUserRegistration(obj);

    return ok().build();
}

Feign configuration:
spring:
    cloud:
    loadbalancer:
        ribbon:
            enable: false
feign:
    client:
        config:
            default:
                connectTimeout: 5000
                readTimeout: 5000
                loggerLevel: basic
eureka:
    client:
        serviceUrl:
            defaultZone: ${EUREKA_URI:http://localhost:8761/eureka}
    instance:
        preferIpAddress: true

POM.xml
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-netflix-eureka-client</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
    </dependency>

I started 3 instances of endpoint services which are consumed as endpoint from Feign client in a round-robin order. So far so good, it's working as expected. But when I shutdown for example the first service instance the Feign client is not notified of this change and continues to send requests to the inactive service.
Do you know how consumer service can be notified when endpoint service is shut down and out of the list of active services? I suppose that I need to configure some health checks?
Do you know how I can solve this issue?
Source code for testing the issue: https://github.com/rcbandit111/eureka-discovery-poc/tree/master
Comment: You have Java code for Spring Cloud load balancer then have a ribbon dependency. Ribbon should be removed
Comment: ok, done. Any advice about the main problem?
Comment: Check eureka UI for what status is your inactive instance? Check [this](https://stackoverflow.com/questions/33921557/understanding-spring-cloud-eureka-server-self-preservation-and-renew-threshold) and [this](https://blogs.asarkar.com/technical/netflix-eureka/) it should help with an understanding of eureka works.
Comment: Are you getting 500 error code in response?
Answer: According to the documentation, Ribbon will not do anything to mark the server as down when a request fails, instead it relies on pinging the servers or clients of the load balancer to notify the ribbon by calling markServerDown(Server server).
As a result you should either change the interval of which the ribbon pings the server by providing your own implementation:
public class CustomLoadBalancer {
  @Bean
  public ILoadBalancer ribbonLoadBalancer(IClientConfig config,
                                          ServerList<Server> serverList, ServerListFilter<Server> serverListFilter,
                                          IRule rule, IPing ping) {
      return new ZoneAwareLoadBalancer<>(config, rule, ping, serverList,
                                         serverListFilter, new PollingServerListUpdater(config));
  }
}

and then use it in your SpringBootApplication configuration like this:
@RibbonClients(defaultConfiguration = CustomLoadBalancer.class)

or you should provide an implementation to call markServerDown(Server) method of ribbon in your client code when you think the time is right.
I suggest you to read this stackoverflow post, I think is directly related to your question.
Comment: I checked on Google. Looks like Ribbon is deprecated `https://stackoverflow.com/questions/65469735/versions-spring-boot-spring-cloud-ribbon-not-working` It will be replaced by spring-cloud-starter-loadbalancer Do you know how to solve the problem using the new framework?
Comment: if you want to use spring-cloud-starter-loadbalancer instead of Ribbon, alongside eureka, you should first disable ribbon by setting `spring.cloud.loadbalancer.ribbon.enabled=false`
then make sure `spring-cloud-starter-loadbalancer` is in your classpath. It will automatically use eureka to create its server list and will be used as your client-side load balancer.
Comment: but according to the post above ribbon is not included into Spring parent anymore.....
Comment: That depends on the release train version you are using. look at table-1 in here: https://spring.io/projects/spring-cloud
Comment: Also there are known issues to https://github.com/spring-cloud/spring-cloud-release/wiki/Spring-Cloud-2020.0-Release-Notes , I don't suggest using it yet if you need those features.
Comment: I use the latest Spring cloud
Comment: as it is stated here https://github.com/spring-cloud/spring-cloud-release/wiki/Spring-Cloud-2020.0-Release-Notes in the breaking changes section ribbon support is removed from this release train. I have not tried this but only including spring-cloud-starter-loadbalancer and spring-cloud-netflix dependencies should suffice.
Comment: ok, any idea how to solve the original post problem?
Comment: Yes, I suggest you to try my solution in the answer.
Comment: But something is not clear for me here. Why have you included ribbon in your POM if you are using spring 2020 and you are not intended to use ribbon at the first place?
Comment: If you are building your project from the ground up, https://start.spring.io/ is a great tool to get you started and  help you to decide what dependencies and which versions can be included into your project.
Answer: Ribbon - Load balancer
Out of the box it's providing you with round robin loadbalancing, but you can implement your own @RibbonClient (even for a specific service) and design your custom loadbalancing for example based on eureka metadata. The loadbalancing happens on the client side.
Feign - Http client
With @FeignClient you can rapidly develop clients for you other services (or services outside of your infrastructure). It is integrated with ribbon and eureka so you can refer to your services @FeignClient(yourServiceNameInEureka) and what you end up with is a client which loadbalances between the registered instances with your preferred logic. If you are using spring you can use the familiar @RequestMapping annotation to describe the endpoint you are using.
I have mentioned below example with RandomRule. Also can check Github here for the source.
@RibbonClient(name = "cloud-provider", configuration = CloudProviderConfiguration.class)
public class ConsumerApplication {
    /* ... */
}

class CloudProviderConfiguration {
    @Bean
    public IRule ribbonRule(IClientConfig config) {
        return new RandomRule();
    }
}

ALso, See the Use of Feign to setup High Availability Load Balancer


How to embed google search console anlytics reports into website using javascript?

Question: I want to embed the google search console results into my website.
I have already done with google analytics embed according to following API.
https://developers.google.com/analytics/devguides/reporting/embed/v1/
Is there any embed api for search console analytics report?
or any other javascript functionality which will fulfill my requirements
I want to display following report from google search console into my web-site
Answer: Interesting.. i see this tutorial --> https://ga-dev-tools.appspot.com/embed-api/basic-dashboard/
I think that can help you
Comment: Thank for your reply. That embed api is used for google analytics. I want to embed google search console analytics.


gym anytrading python dataframe format

Question: I am very new to gym anytrading i have this pandas dataframe where there is a column with a list of lists that are different lengths I am trying to figure out how to put that into the gym anytrading environment. Below is a link to a csv of the sample data in the dataframe and a code snippet. I keep getting this error TypeError: cannot unpack non-iterable NoneType object
import gym
import pandas as pd
import numpy as np
from gym_anytrading.envs import TradingEnv

class CustomTradingEnv(TradingEnv):
    def __init__(self, df):
        super().__init__(df, window_size=10)
        self.reward_range = (0, 1)

    def _process_data(self):
        # Process your DataFrame to ensure it's in the correct format
        # Here, you can perform any necessary preprocessing steps
        pass

    def reset(self):
        # Initialize the environment with the data from the DataFrame
        self._process_data()
        return super().reset()

    env = CustomTradingEnv(df)

    observation = env.reset()
    for _ in range(100):  # Run for 100 steps
    action = env.action_space.sample()  # Sample a random action
    observation, reward, done, info = env.step(action)
    if done:
        break

https://docs.google.com/spreadsheets/d/1-LFNzZKXUG44smSYOy2rgVVnqiygLfs00lAl2vFdsxM/edit?usp=sharing
Comment: Which version of `gym` (which `gym` repo) are you using?
Answer: First of all, you are not using the right gym package:
import gym

needs to be
import gymnasium as gym

since gym_anytrading also uses gymnasium (which is subtly different from the no-longer-maintained, older gym package).
Then the indentation in your code is incorrect - I assume this is just a typo. The iteration should be:
for _ in range(100):  # Run for 100 steps
    action = env.action_space.sample()  # Sample a random action
    observation, reward, done, info = env.step(action)
    if done:
        break

The actual error is coming from the __init__ function and the _process_data function. The full traceback is:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...", line 24, in test
    env = CustomTradingEnv(df)
  File "...", line 9, in __init__
    super().__init__(df, window_size=10)
  File "/.../anaconda3/envs/py310/lib/python3.10/site-packages/gym_anytrading/envs/trading_env.py", line 35, in __init__
    self.prices, self.signal_features = self._process_data()
TypeError: cannot unpack non-iterable NoneType object

To fix this you need to modify process_data so that it returns a tuple of  numpy arrays, prices (a 1-dim array of floats) and signal_features (a 2-dim array). This is also clearly explained in the gym_anytrading README.
With your dataframe you could do something like:
    def _process_data(self):
        env = self.unwrapped
        start = 0
        end = len(env.df)
        prices = env.df.loc[:, 'low'].to_numpy()[start:end]
        signal_features = env.df.loc[:, ['close', 'open', 'high', 'low']].to_numpy()[start:end]
        return prices, signal_features

When you do that your test loop will work fine, apart from the fact that you didn't provide any reward function. A custom env needs to implement a _calculate_reward function that returns a reward signal for any of the possible actions (in this case there are only two).
So that should look similar to:
    def _calculate_reward(self, action):
        # I'm assuming 0 and 1 stand for sell and buy (or viceversa)
        match action:
             case 0: return 0.1 # or some other more suitable value
             case 1: return -0.1 # or something more suitable
             case _: raise Exception("bug")

Also the _update_profit(self, action) function needs to be implemented (it needs to update the internal env state, self.unwrapped._total_profit, but you can let it pass as well).
Finally, the env.step function in your code is incorrect (for the latest version of gymnasium). It should be used as:
   observation, reward, done, terminated, info = env.step(action)

gym anytrading python dataframe format


Can't open json file to blender using python

Question: I am using blender on windows, following the tutorial: Visualize JSON Data in blender but I am stuck trying to open the JSON file.
My code (windows):
    import json

    with open('C:\Users\Franktabs\Documents\export.json') as f:
        j = json.load(f)
        print(j)

His code(linux):
import json

with open('/home/chris/Downalds/tutorial.son') as f:
    j = json.load(f)
    print(j)

Error Message:
Python:   File "C:\Users\Franktabs\Documents\Json2.blend\My Script", line 3
     with open('C:\Users\Franktabs\Documents\export.json') as f:
                   ^ SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
 location: <unknown location>:-1

Solutions Tried:

Using import bpy
Using double \
Using 'r'
Using utf-8
Checking the file type is actually .json
Checking the route and name of the file in properties

Solutions sources from a person with the same problem:
post1,post2
Comment: Does this answer your question? [using backslash in python (not to escape)](https://stackoverflow.com/questions/3380484/using-backslash-in-python-not-to-escape)
Comment: Backslashes inside a string (such as your Windows path) have a special meaning and must be *escaped* to be used literally. See the answers to the question linked above for details/workarounds.
Comment: Change your backward slashes ``\`` to forward slashes `/`.
Comment: @MattDMo Why? `os.path.join` should be used instead.
Comment: @OneCricketeer `os.path.join` is still going to need a base path, which will almost certainly include `\U...`, which is what is triggering the unicode error. Forward slashes are perfectly acceptable in Windows path literals, and OP claimed to have tried ``\\`` and `r` literals (which I doubt were done correctly, as they failed, but that's another topic). It's just something else to try without changing the actual code.
Comment: @MattDMo `pathlib.Path.home()` should return the Users directory, though
Comment: @OneCricketeer I agree, there are lots of things that *could* be done to make this problem go away, I just chose the simplest one.
Answer: As mentioned \U is a unicode escape character, and is likely part of the issue. Without seeing the \\ usage or r'' code or error, hard to know what the issues there could be...
This should work for an OS-agnostic solution, assuming you have a Documents directory where the file exists
import json
import pathlib

with open(pathlib.Path.home() / 'Documents' / 'export.json') as f:
  data = json.load(f)


How to change the color of values that fall outside the determined breaks of the pheatmap function in R

Question: I have 1 question regarding the pheatmap function in R and would be extremely thanksful if anyone could help me out.
1) When doing pheatmaps, I always set my breaks using the break function. However, if a value fall out of the determined breaks it is automatically represented in white. Does anyone know how to set the values that fall out of my breaks to the same color as my max or min? Here is a code-example using the "ALL" library:
library(pheatmap)
library(RColorBrewer)
library(ALL)

data("ALL")
expressionData = exprs(ALL)

#setting breaks to 0 to 5 then 5.01 to 10
breaks = c(seq(0, 5, length.out = 101),seq(5.01, 10, length.out = 101))
breaks2 <- breaks

#setting colors
my_palette <- colorRampPalette(c("blue", "white", "red"))(n = 201)

#running pheatmap with 10 first samples
pheatmap(expressionData[1:10, ], color = my_palette, breaks = breaks2)

image of the pheatmap result
Because some of the value for the sample 1005_at are higher than my highest breakpoint of 10, they end up being white. How I could set that values higher than 10 are the same color as the value of 10 (and the other way around, if there were to be values lower than 0, how to set them to the same color as the value 0)?
Thank you very much!!!
Comment: Are you happy modifying your breaks?
Comment: Yes. I'm doing multiple heatmap with different datasets and I need to keep the breaks the same among all of them.
Answer: This seems to be a possible solution:
library(pheatmap)
library(RColorBrewer)
library(ALL)

data("ALL")
expressionData = exprs(ALL)

#setting breaks to 0 to 5 then 5.01 to 10
breaks = c(seq(0, 5, length.out = 101),seq(5.01, 10, length.out = 101))
breaks2 <- breaks

#setting colors
my_palette <- colorRampPalette(c("blue", "white", "red"))(n = 201)

breaks2[length(breaks)] <- max(max(expressionData),max(breaks))
breaks2[1] <- min(min(expressionData),min(breaks))
#running pheatmap with 10 first samples
pheatmap(expressionData[1:10, ], color = my_palette, breaks = breaks2)

Comment: This is indeed a really nice solution!! It worked really well. Thanks!!


O que são variáveis livre e ligada em Prolog?

Question: O que se define como variável livre ou ligada. Dei uma leve pesquisada, mas não consegui entender muito bem o conceito matemático e diferenciá-las.
Se possível, explicar com exemplos definidos em Prolog mostrando quando uma variável é dita livre e quando ligada.
Answer: Uma variável livre é uma variável que ainda não possui um valor. Uma ligada é uma que possui (e na ausência de backtracking após uma variável receber um valor ela está ligada a ele para sempre, não sendo possível lhe atribuir um outro valor).
Exemplo:
?- member(X, Y).
Y = [X|_G3168] .

Ao chamar member/2, tanto X quanto Y eram variáveis livres. Após a chamada, Y passou a estar ligada a uma lista (iniciada com X e terminada com qualquer coisa). X continua livre, de modo que você pode atribuir-lhe o valor que você quiser:
?- member(X, Y), X = 42.
X = 42,
Y = [42|_G3116] .

Enquanto Y não está mais livre, se você tentar unificá-la com algo diferente do que ela é (uma lista com no mínimo um elemento) a execução vai falhar (provocando um backtrack):
?- member(X, Y), !, Y = 42.
false.

(o corte é pra evitar um loop infinito, pois member/2 continuaria gerando novos Ys, falhando e voltando)
Note que como o conteúdo de Y contém variáveis livres, você pode unificá-la com outra lista compatível com ela. Você não estaria mudando o seu valor - a lista continuaria a mesma. Só seus elementos (cabeça e cauda) que antes eram livres e vão deixar de ser:
?- member(X, Y), Y = [1,2,3,4,5].
X = 1,
Y = [1, 2, 3, 4, 5] .

(nesse ponto você poderia pedir "mais soluções" para obter os demais valores de X, mas no final também cairia num loop infinito).
Verificando se uma variável é ou não livre
Se você quiser saber se uma variável possui ou não um valor, não adianta compará-la com alguma outra coisa - já que compará-la vai acabar por unificá-la com aquela coisa e portanto torná-la ligada. Em vez disso, use os predicados var/1 e nonvar/1 que têm sucesso se a variável é livre/ligada respectivamente:
?- member(X, Y), var(X), nonvar(Y).
Y = [X|_G4611629] .

?- member(X, Y), !, var(Y).
false.

?- member(X, Y), !, nonvar(X).
false.


freebase cURL and mqlwrite with PHP

Question: Hi am doing some testing on the Freebase sandbox, but I can't get the following request to work. It hangs for ages then returns nothing. I have also outputted curl_getinfo() if it helps anyone work out where I am going wrong.
echo $uri = "http://".$this->config['apiSandboxHost'].'/'.$this->config['apiWritePath'].'={"create":"unless_exists","type":"/user/docs/music/note","name":"A","id":null}';

$ch = curl_init($uri);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$headers = array();
$headers[] = 'X-Metaweb-Request : idio';
$headers[] = 'Content-length: 78';
$headers[] = 'Content-type: application/x-www-form-urlencoded';
$headers[] = 'Cookie: '.  implode(';', $this->authCookies);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$output = curl_exec($ch);
pr(curl_getinfo($ch));
curl_close($ch);
echo $output;

Output
http://sandbox.freebase.com/api/service/mqlwrite?query={"create":"unless_exists","type":"/user/docs/music/note","name":"A","id":null}

Array
(
    [url] => http://sandbox.freebase.com/api/service/mqlwrite?query={"create":"unless_exists","type":"/user/docs/music/note","name":"A","id":null}
    [content_type] =>
    [http_code] => 0
    [header_size] => 0
    [request_size] => 693
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 285.371
    [namelookup_time] => 0
    [connect_time] => 0.185
    [pretransfer_time] => 0.185
    [size_upload] => 0
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 0
    [download_content_length] => 0
    [upload_content_length] => -1
    [starttransfer_time] => 285.371
    [redirect_time] => 0
)

Thanks in advance for any help!
Answer: It ended up being an issue with the way I was sending data to freebase. Once I added
curl_setopt($ch, CURLOPT_POSTFIELDS, $query);

it all worked fine.
It was weird as I could only get it working when I sent the data as a post variable and also added content to the query string of the request.

freebase cURL and mqlwrite with PHP


Laravel 5.1 use session to restrict direct access using urls users based on user role

Question: I have 2 laravel projects, 1 for the front end where i m using html css angularjs. The second for api controllers. I call using http post and get the api controllers functions using angularjs to get content data.
In the front end i have a menu this menu appears differently based on user role, if admin or no.
This is done. My problem is the access for views using the url in the browser.
So I have a query where I get for each user what modules in the menu can he see. Now I'm putting the result in Laravel session.
$menu = DB::select menu by user id ... //Getting menu query based on user if admin or no

session(["menu" => $menu);
return session('menu');

I'm getting the results and the menu is showing good in the website based on the logged user if he s admin or no.
Now, to solve the direct url access issue, I want to use this session and compare the url to this session, if the url exists in the session i will let him access, if no i will redirect him to somewhere.
any idea?
Answer: I would strongly suggest looking at the Laravel documentation on Authorization before going too far down a custom implementation:
https://laravel.com/docs/5.1/authorization
Without knowing more about how your front-end and back-end applications interact with each other, it is a little difficult to get into speciifics but i shall do my best.
Each page returned by Laravel has access to a Request object which contains information about the request which returned the page. You can access this Request and its assocaited Route using Laravels helper functions (if you are not passing it to the view already). The getPrefix() method will return the root relative url which you can then use as you see fit. For example:
// Return and store the URL as a string
$url = request()->route()->getPrefix();

// Check your session for the URL/s you want to allow and compare to the stored URL
if (session()->get('URL') == $url) {
    // User is allowed access to the page
    // Do something ...
} else {
    // User is not allowed access to this page
    // Redirect back or to a route of your choice
    return redirect()->back();
}

I hope this gives you some ideas, good luck!


Using FBS's get resource method in qss definitions?

Question: I have a seperate qss stylesheet that contains my stylesheet definitions for my entire app. I need to add a close button as background image since I couldnt find documentation on using inbuilt icons.
customTabBar::close-button {
    padding: 0px;
    margin: 0px;
    border-radius: 2px;
    border-color: rgba(0, 0, 0, 50%);
    background-image: url(%(closeIcon)s);
    background-position: center center;
    background-repeat: none;
}

from fbs_runtime.application_context.PyQt5 import ApplicationContext

import qstylizer.parser

customIcons = {
    "customTabBar::closeButton.backgroundImage": f"url({app.get_resource('ui/close.png')})",
}

app = ApplicationContext()

with open(app.get_resource("app.qss"), "r") as stylesheet:
    css = qstylizer.parser.parse(stylesheet.read())
    for key, value in customIcons.items():
        property = key.split(".")
        css[property[0]][property[1]].setValue(value)
    app.app.setStyleSheet(css.toString())

the files are stored in the default fbs structure, under
src/main/resource/base/ui/*.png
since I cannot use fstrings with curly braces being a part of qt. And this answer using python string formatting, but I keep getting key errors due to me having some rgba color values that also have % in it.
Since I cant use %ages or curly braces, I was thinking of building a qproperty out of get_resource, but I am not sure how. I need my qss cross compatible and cannot escape my curly braces.
My main problem is that the image wont be available when I package the application wtih fbs, using FBS freeze
Comment: plop, you have to use pyrcc5: `pyrcc5 your.qrc -o your_rc.py`
Comment: please provide a [mre]
Comment: @eyllanesc I thought my issue was a bit straight forward but sure I added an MRE.
Comment: there are things that are not defined, for example what is fileImporter, where are the icons, etc.
Comment: my apologies, my file importer is an extension to get resource with some additional conditions. For the image file, you can use anything for now. All i care about is the file loading properly, while maintaining the qss structure
Comment: do not delete the above from your question, instead add it to the end as I have done with my answers so that the question is understood by other readers
Comment: the problem has nothing to do with FBS but you want to dynamically modify the qss
Comment: my problem stems from fbs, since after releasing any file paths in the qss would not be valid, the only workaround i know of now, would be to have the qtabbar stylesheet in my qtabbar class itself.
Comment: Why don't you use qresource?
Comment: @eyllanesc because qresource demands relative path (starting with a /) and I simply cannot maintain that if package the application
Comment: mmm, it seems that you have not handled qresource. On the contrary, the qresource generates a virtual path. Only the relative path is necessary at the time of converting the .qrc to .py.
Comment: a qrc can be converted to py? i was using qresource.registerResource() ...
Comment: Building a qrc file, running pyrcc5 and then executing my code , with background-image: url(:/closeIcon.png); is a no go. All i get is could not create pixmap from :\closeIcon.png.
I followed this https://stackoverflow.com/questions/36673900/importing-resource-file-to-pyqt-code
Comment: I do not understand you, explain yourself better. Even so it already seems that I have solved the qstylizer bug, I will send the PR in a few moments
Comment: No worries, I can wait, He has merged the PR and hopefully will release soon.
Comment: I will test it, thing is even without the properties i am getting a parse error after passing the .to_string() to setstylesheet. with the css I added in an edit to the original question. I ran the output through a text comparator against the original stylesheet, but I just cant see the issue.
Comment: The author of qstylizer has already done the merge, so I will remove my repo, use qstylizer's gh, or wait for me to post it on pypi
Answer: Parsing and modifying Qt Style Sheet using the python formatting tools can be complicated, for example in your case you try to format (0, 0, 0, 50%) causing errors, so I recommend using qstylizer(python -m pip install qstylizer), so you can easily modify the properties:
QTabBar::close-button {
    padding: 0px;
    margin: 0px;
    border-radius: 2px;
    border-color: rgba(0, 0, 0, 50%);
    background-position: center center;
    background-repeat: none;
}

import functools

from fbs_runtime.application_context.PyQt5 import ApplicationContext

import qstylizer.parser

customIcons = {
    "QTabBar.closeButton.backgroundImage": f"url({app.get_resource('ui/close.png')})",
}

app = ApplicationContext()

with open(app.get_resource("app.qss"), "r") as stylesheet:
    css = qstylizer.parser.parse(stylesheet.read())
    for key, value in customIcons.items():
        obj = functools.reduce(getattr, key.split("."), css)
        obj.setValue(value)
    app.app.setStyleSheet(css.toString())

Update:
Analyzing the source code:
if key and key[0] not in ["Q", "#", "[", " "] and not key.istitle():
    key = inflection.underscore(key)
it seems that the classes are TitleCase so a possible solution is to change the name of the class to Customtabbar:
Customtabbar::close-button {
    padding: 0px;
    margin: 0px;
    border-radius: 2px;
    border-color: rgba(0, 0, 0, 50%);
    background-position: center center;
    background-repeat: none;
}

app = ApplicationContext()

customIcons = {
    "Customtabbar::close-button": {
        "background-image": f"url({app.get_resource('ui/close.png')})"
    },
}

with open(app.get_resource("app.qss"), "r") as stylesheet:
    css = qstylizer.parser.parse(stylesheet.read())
    for qcls, value in customIcons.items():
        for prop, v in value.items():
            css[qcls][prop] = v
    app.app.setStyleSheet(css.toString())

According to PEP the class names must be CapWords so I have created a fork by changing:
qstylizer/style.py
if key and key[0] not in ["Q", "#", "[", " "] and not key.istitle():

by
if key and key[0] not in ["Q", "#", "[", " "] and key != inflection.camelize(key):

Now accept the names of the classes that comply with PEP8.
CustomTabBar::close-button {
    padding: 0px;
    margin: 0px;
    border-radius: 2px;
    border-color: rgba(0, 0, 0, 50%);
    background-position: center center;
    background-repeat: none;
}

app = ApplicationContext()

customIcons = {
    "CustomTabBar::close-button": {
        "background-image": f"url({app.get_resource('ui/close.png')})"
    },
}

with open(app.get_resource("app.qss"), "r") as stylesheet:
    css = qstylizer.parser.parse(stylesheet.read())
    for qcls, value in customIcons.items():
        for prop, v in value.items():
            css[qcls][prop] = v
    app.app.setStyleSheet(css.toString())

Update2:
The PR has been accepted so it is only necessary to update the library: python -m pip install qstylizer --upgrade.
Comment: getattr didnt work and i had to change a few things
```"tabBarPlus::closeButton.backgroundImage": f"url({app.get_resource('ui/close.png')})",```
and ```property = key.split(".")
        qss[property[0]][property[1]].setValue(value)
``` however, my custom classes for example customView become custom-view resulting in an stylesheet not being understood by qt (Could not parse application stylesheet)
Comment: @Blaine that tells me that your example is not an MRE, provide and then I will try to help you
Comment: I have updated my original question with a newer MRE, replicating my current issue. According to qstylesheet source code, I see that only snake case is intended to be modified but I am unsure. All I have changed in the example is that I changed QTabBar to customTabBar, following my naming convention in my code. After parsing it becomes custom-tab-bar
Comment: @Blaine I think you discovered a bug, I am analyzing where is the cause to patch it but until now I discovered that if the class starts with Qt the problem does not happen, could you change your class to QCustomTabBar?
Comment: Ah, well I wouldnt mind changing one class but provided that you always need to subclass qt classes I would have to change class names for every class I have. I think for a workaround I will look into a regex replacement on the string returned by to string
Comment: @Blaine Update the library, the bug has been solved but you must use CustomTabBar
Comment: apparently imho, its better if no sanitization is performed. qproperty-defaultAlignment is the valid property, qstylizer will santitize it to qproperty-defaultalignment. Making the stylesheet incorrect.
Comment: I have updated my original question,and now I cant simply tell where the problem lies
Answer: As per @eyllanesc 's suggestion, this was my solution.
the rcc file,
<RCC>
  <qresource>
    <file alias="closeIcon">close.png</file>
  </qresource>
</RCC>

the shell command,
pyrcc5 -o resources.py resources.rcc

and here is the style sheet.
TabBarPlus::close-button {
    background-image: url(:/closeIcon);
    padding: 0px;
    margin: 0px;
    border-radius: 2px;
    background-position: center center;
    background-repeat: none;
}


How to secure Windows VISTA (Ultimate) to run as Server with Remote Desktop enabled?

Question: I need to run a Windows Vista Instance (Ultimate) on a dedicated Server with a dedicated public IP and the Remote Desktop Protocoll enabled. Vista will server as a "small windows" server to run a program via RemoteDesktop only for authenticated users.
What do I have to do, to secure this Vista Installation (after installing it on the server). Is Windows (after doing the updates and having autoupdates enabled) more or less secure by default? What do I have to tweak to make it secure?
It would be kind if you can list the steps I need to do (for example: disable this service and that service, enable the firewall under XYZ...)
Thank you very much!
Tim
Comment: vista will only let you run 1 rdp session. is this what you want?  you may have to switch to windows server.  Vista is s desktop operating sytem
Answer: Another thing you can do to make it even more secure is to actually tunnel RDP through SSH.
If you have a router that has an sshd or if the box itself can have something like cygwin installed, you can tunnel the RDP port to the machine you're connecting from and disable open internet access to terminal services entirely.
Example: http://theillustratednetwork.mvps.org/Ssh/RemoteDesktopSSH.html
Answer: Some points:

Place the machine in distinct
security zone (DMZ) with adequate
filtering
Forward to the machine only the
strict necessary ports
keep the machine OS and software  up
to date.
Use the strongest authentification
mecanism that you can.
Audit config files changes and failed
logons.

That's probably not an exhaustive answer. Anyway hope this helps.
Comment: Hello Maxwell. Thanks for your answer! The problem is, that I will (due to the server/network architecture) not be possible to place it in a DMZ or "frontend" firewalled network. The Vista installation will be directly be accessible over the internet via its public IP and I only have the possibility to secure Vista itself...
Answer: The firewall is enabled by default. Check it is configured appropriately.
Consider moving RDP to a non default port to minimise the no of casual attacks.
consider automatic account access lockout if too many failed attempts.
pick strong passwords, only allow essential users remote access.
require network level authentication for RDP sessions if you're sure all the potential clients support it.
I run an XP machine in this situation, and it's survived for several years without compromise, so don't be excessively concerned.
Comment: Hello Xenny. Great posting, thanks. It contained several very helpful information.
Answer: Adding to the other excellent answers:

As far as I recall, automatic lock-out will not prevent brute-forcing any administrator accounts with tools like TSGrinder as they cannot be locked out when interactively used (which remote desktop counts as). So make sure administrator accounts are explicitly denied access by removing them from the allowed remote logon list. See this neat old XP article about securing Remote Desktop for XP for some examples on how to do this and other useful things.
Require the highest encryption level (FIPS-grade) is tedious but well worth it, and your Vista SKU seems to support it according to this rather detailed article: Connect securely to Windows Vista Remote Desktop.
Answer: I think you should harden the IP Stack following some Microsoft's recommandations.
You should see on MSDN website for more informations about hardened IP Stack.


Get browser saying 'default browser'

Question: I am using get_browser to keep statistics for my web site.  However, all of the results are browser = 'Default Browser' and platform = 'unknown'.  I checked phpinfo() and the browscap.ini path is correct.  What else would be causing this problem?  It does this in all browsers.
My Code:
print_r(get_browser(null, true));

Comment: See this page: http://chrisschuld.com/projects/browser-php-detecting-a-users-browser-from-php.html
Comment: Please show your code.
Answer: You can use getallheaders()
http://php.net/manual/en/function.getallheaders.php
$headers = getallheaders();
$browser = $headers['User-Agent'];

Comment: Is that any different than $_SERVER ['HTTP_USER_AGENT']?
Comment: To tell the trouth I have never researched the topic but my best guess is: they should be just the same.
Comment: I tested that and it gives the same results.  Thanks, though.
Comment: How are you testing? I mean probably your test request is passing the default_browser flag if you are using curl or anything other than a browser.
Answer: I wouldn't use get_browser() since it uses the browscap file, which is constantly being updated and is more for seeing what the browser can do instead of getting the name of the browser. (browscap - short for BROWSer CAPabilities).  You can find many other ways to do this online, with a quick Google search.

Get browser saying 'default browser'


Disable 3rdparty hook from custom module once

Question: Im using the menu_trail_by_path which does a pretty good job for most of my URL's, however in a custom module I'm setting the breadcrumbs directly (drupal_set_breadcrumb in a page callback). These crumbs get messed up due the menu_trail_by_path module (sort of overwritten I guess =/)
This happens because menu_trail_by_path sets the crumbs using hook_page_delivery_callback_alter which is called after the page callback apparently.
I would like to keep my custom crumbs in the page callback as it has all the required (dynamic) data available.
Hero we go.. how on earth can I disable the page_delivery_callback_alter hook from the menu_trail_by_path module... once? I.e. for the current runtime /request only.
Thanks!
Roland Franssen
Answer: You could implement hook_module_implements_alter() to avoid a hook implemented by a module is invoked. The problem is that Drupal saves in cache the list of modules implementing a hook, but you can change the list of hooks implemented by modules back to the original one.
function mymodule_implements_alter(&$implementations, $hook) {
  if ($your_condition && ($hook == 'page_delivery_callback_alter')) {
    $old_implementations = &drupal_static(__FUNCTION__, array());
    if (empty($old_implementations)) {
      $old_implementations = array_merge(array(), $implementations);
      unset($implementations['menu_trail_by_path']);
    }
  }
}

function mymodule_exit() {
  $old_implementations = &drupal_static('mymodule_implements_alter', array());
  $implementations = &drupal_static('module_implements');

  if (!empty($old_implementations)) {
    $implementations['page_deliver_callback_alter'] = $old_implementations;
  }
}

This code works only in the case hook_exit() is invoked, which is normally done in drupal_page_footer().
You should be sure mymodule_implements_alter() is executed after the implementations made from other modules, in order to preserve the changes done from other modules.
Comment: Oh dear god :-)
Comment: This is inception!


spring+hibernate why is many-to-many relationship not persisted?

Question: I have an entity E1 defined as
@Entity
public class E1 {
    @Id
    @GeneratedValue
    public long id;
    @ManyToMany(cascade={CascadeType.PERSIST, CascadeType.MERGE})
    @JoinTable(
            name="e1_e2",
            joinColumns = @JoinColumn(name = "e2_id"),
            inverseJoinColumns = @JoinColumn(name = "e1_id")
    )
    public Set<E2> e2s = new HashSet<>();
}

an entity E2 defined as
@Entity
public class E2 {
    @Id
    @GeneratedValue
    public long id;
    @ManyToMany(mappedBy = "e2s")
    public Set<E1> e1s = new HashSet<>();
}

and controller defined as
@RestController
@RequestMapping("/")
public class C1 {
    private final E1Repository e1Repository;
    private final E2Repository e2Repository;

    @PersistenceContext
    EntityManager em;

    @Autowired
    public C1(E1Repository e1Repository, E2Repository e2Repository) {
        this.e1Repository = e1Repository;
        this.e2Repository = e2Repository;
    }

    @Transactional
    @RequestMapping(method = POST)
    public void c(){
        E1 e1 = new E1();
        E2 e2 = new E2();

        e1Repository.save(e1);
        e2Repository.save(e2);

        em.refresh(e1);
        em.refresh(e2);

        e1.e2s.add(e2);
        e2.e1s.add(e1);

        e1Repository.save(e1);
        e2Repository.save(e2);

        em.refresh(e1);
        em.refresh(e2);
    }

}

(E1Repository and E2Repository are @Repository annotated interfaces that extend JpaRepository and have empty bodies).
when I step through the c method in my debugger, I see that after the last two em.refresh lines, both e1 and e2 have their sets cleared.
based on other questions I found on Stack Overflow, I tried defining E2 as
@Entity
public class E2 {
    @Id
    @GeneratedValue
    public long id;
    @ManyToMany(cascade = CascadeType.PERSIST)
    @JoinTable(
            name="e1_e2",
            inverseJoinColumns = @JoinColumn(name = "e2_id"),
            joinColumns = @JoinColumn(name = "e1_id")
    )
    public Set<E1> e1s = new HashSet<>();
}

but this did not help.
Original question (from before I tried testing the simplified case above)
I have a class Robot defined similar to the following:
@Entity
class Robot{
    @Id
    @GeneratedValue
    private long id;

    @ManyToMany(mappedBy="robots")
    private Set<Match> matches = new HashSet<>();
}

and a class Match similar to
@Entity
class Robot{
    @Id
    @GeneratedValue
    private long id;
    @ManyToMany(cascade={CascadeType.Persist,CascadeType.Merge})
    @JoinTable(
        name = "match_robot",
        joinColumns = {@JoinColumn(name = "Match_id")},
        inverseJoinColumns = {@JoinColumn(name = "Robot_id")}
)
private Set<Robot> robots = new HashSet<>();

and a class Result defined similar to
@Entity
public class Result {
    @Id
    @GeneratedValue
    private long id;

    @ManyToOne(optional = false)
    private Match match;

    @ManyToOne(optional = false)
    @NotNull(groups = {Default.class, Creating.class})
    private Robot robot;
}

and attempt to save the relationship as follows:
resultRepository.save(result);
match.getRobots().add(result.getRobot());
result.getRobot().getMatches().add(match);
robotRepository.save(result.getRobot());
matchRepository.save(match);
entityManager.refresh(result);
entityManager.refresh(result.getRobot());
entityManager.refresh(match);

where *Repository are beans created by spring that implement JpaRepository.
When this code is ran, an insert statement is ran by hibernate for the result object (hibernate outputs all of its sql commands to the console), but none for the "match_robot" table. I should not that, before this snippet, result is in the transient state of the hibernate entity lifecycle, match is in the persisted state, robot is in the persisted state, and result's match and robot properties have been set to match and robot respectively.
Based on other questions on Stack Overflow and other sites, I have also tried defining the matches variable as
@ManyToMany(cascade = CascadeType.PERSIST)
@JoinTable(
        name = "match_robot",
        inverseJoinColumns = {@JoinColumn(name = "Match_id")},
        joinColumns = {@JoinColumn(name = "Robot_id")}
)
private Set<Match> matches = new HashSet<>();

which did not help. Other than that, the only recommendation I was seeing was to make sure both entities in the many-to-many relationship were consistant with each other before persisting, but, as far as I can tell, I am doing so here.
How can I persist this relationship?
Edit: the full method for context:
@Transactional
@RequestMapping(value = "/{match:[0-9]+}/results", method = RequestMethod.POST)
public ResultResource createResult(@PathVariable Match match,
                                   @Validated(Result.Creating.class)
                                   @RequestBody Result result) {
    if (match == null) throw new ResourceNotFoundException();

    result.setScorecard(scorecardRepository
            .findById(result.getScorecard().getId()));
    if (result.getScorecard() == null) {
        throw new ScorecardDoesNotExistException();
    }

    result.setMatch(match);

    //remove null scores
    result.getScores().removeIf(
            fieldResult -> fieldResult.getScore() == null
    );

    //replace transient robot with entity from database
    Robot existingRobot = robotRepository
            .findByNumberAndGame(result.getRobot().getNumber(),result.getRobot().getGame());
     if (existingRobot == null) { //create new robot
        //find team for this robot
        Team existingTeam = teamRepository
                .findByNumberAndGameType(
                        result.getRobot().getNumber(),
                        result.getScorecard().getGame().getType());
        if (existingTeam == null) {
            Team team = new Team();
            team.setNumber(result.getRobot().getNumber());
            team.setGameType(result.getMatch().getEvent().getGame().getType());
            team.setDistrict(result.getMatch().getEvent().getDistrict());
            teamRepository.save(team);
            result.getRobot().setTeam(team);
        }
    else result.getRobot().setTeam(existingTeam);
             result.getRobot().setGame(result.getMatch().getEvent().getGame());

        robotRepository.save(result.getRobot());
        entityManager.refresh(result.getRobot());
    } else result.setRobot(existingRobot);
    List<Robot> all = robotRepository.findAll();

    //replace transient FieldSections with entities from database
    //todo: reduce database hits
    //noinspection ResultOfMethodCallIgnored
    result.getScores().stream()
          .peek(fieldResult -> fieldResult.setField(
                  fieldSectionRepository.findByIdAndScorecard(
                          fieldResult.getField().getId(),
                          result.getScorecard())))
              .peek(fieldResult -> {
              if (fieldResult.getField() == null)
                  throw new ScoresDoNotExistException();
          })
          .forEach(fieldResult->fieldResult.setResult(result));

    if (!result.scoresMatchScorecardSections()) {
        throw new ScoresDoNotMatchScorecardException();
    }

    if (!result.allMissingScoresAreOptional()) {
        throw new RequiredScoresAbsentException();
    }

    if (!result.gameMatchesScorecard()) {
        throw new GameDoesNotMatchScorecardException();
    }

    resultRepository.save(result);
    match.getRobots().add(result.getRobot());
    result.getRobot().getMatches().add(match);
    robotRepository.save(result.getRobot());
    matchRepository.save(match);
    entityManager.refresh(result);
    entityManager.refresh(result.getRobot());
    entityManager.refresh(match);
    return new ResultResourceAssembler().toResource(result);
}

Comment: Could you please post your code before you call `resultRepository.save(result);` in order to get an idea who your object  net looks like when you start persisting it? Better the whole method? Is this method or containing class annotated with `@Transactional`?
Comment: @ansgar-schulte I had left out the preceding part of the method because, with the exception of creating and saving the `Robot` object and giving a 404 if the `match` object is null, it has no dealings with `Robot` or `Match`. I'll go ahead and add it though.
Comment: *when I step through the c method in my debugger, I see that after the last two em.refresh lines, both e1 and e2 have their sets cleared.*: well, of course they are, since you refreshed them without flushing the changes you just made. Why do you refresh? And why do you re-save managed entities? All those re-saves and refreshes are useless, or even counter-productive. Read http://docs.oracle.com/javaee/6/api/javax/persistence/EntityManager.html#refresh(java.lang.Object): *Refresh ... overwriting changes made to the entity, if any.*.
Comment: @jb-nizet I thought the save method on the JpaRepository flushed changes since it appeared to in all other circumstances. I'm refreshing so that I can see in the debugger if the changes were persisted by the save() method. replacing save with saveAndFlush leads to a "Unique index or primary key violation" on the join table (e1_e2).
Comment: @jb-nizet the re-saves and refreshes are for me viewing it in the debugger. I did not add them until after I was experiencing this issue.
Comment: @jb-nizet the exception I mentiones was from using the second definition of E2 described. with the first one, it works perfectly. Thank you.
Answer: JpaRepository.save() does not flush the changes to the database. It is necessary to flush the changes with JpaRepository.flush() or JpaRepository.saveAndFlush().


uWSGI doesn't execute chdir

Question: Title. When I run uwsgi as uwsgi *** or /usr/local/bin/uwsgi *** it works perfectly, but when I run it as /usr/bin/uwsgi *** (as it being runned as service) it doesn't execute chdir directive of .ini file what results in impossibility to load the script.
My .ini file:
[uwsgi]
socket = 127.0.0.1:3031
pidfile = /run/uwsgi/app/***.pid
chdir = /var/www/admin/data/www/test.***.##/***/
pythonpath = ..
file = wsgi.py

default.ini file:
[uwsgi]
autoload = true
master = true
workers = 2
no-orphans = true
log-date = true
uid = www-data
gid = www-data

How could I fix it?
Answer: If you run a different program, you'll get different results.  /usr/local/bin/uwsgi and /usr/bin/uwsgi are different programs.  My first thought would be that /usr/bin/uwsgi is perhaps an older version, which doesn't support the chdir parameter.
Comment: Yes, you was right, there was an old version from debian repositories in `/usr/bin/uwsgi` and new version which was installed by `pip` in `/usr/local/bin/uwsgi`. Service still doesn't start, but it's another problem already. Thanks!

uWSGI doesn't execute chdir


Trying to submit form values where only part of the form binds to a model (with ASP.Net MVC3)

Question: I apologize in advance for a wall of text below. I hope to provide all the info at once that may be needed to understand what I'm doing.
I've got a page where I'm collecting some basic user info (name, email, etc). In another portion of the page I have a multiple choice quiz form (using radio inputs for each question). My goal is to collect the contact info, process the quiz answers, and store a score with the contact info in a QuizResults table.
This may be a case of over thinking, or over engineering.. so feel free to tell me I'm going about this all wrong.
Here's my Quiz related models:
public class QuizType {

    public QuizType() {
        this.QuizQuestions = new HashSet<QuizQuestion>();
        this.QuizResults = new HashSet<QuizResult>();
    }

    public int Id { get; set; }
    public string Name { get; set; }

    public virtual ICollection<QuizQuestion> QuizQuestions { get; set; }
    public virtual ICollection<QuizResult> QuizResults { get; set; }
}

public class QuizQuestion {

    public QuizQuestion() {
        this.QuizAnswers = new HashSet<QuizAnswer>();
    }

    public int Id { get; set; }
    public string Question { get; set; }
    public int Order { get; set; }
    public int QuizTypeId { get; set; }

    public virtual ICollection<QuizAnswer> QuizAnswers { get; set; }
    public virtual QuizType QuizType { get; set; }
}

public class QuizResult {
    public int Id { get; set; }
    public string TesterName { get; set; }
    public string TesterEmail { get; set; }
    public string TesterCompany { get; set; }
    public string TesterPhone { get; set; }
    public string ApproverName { get; set; }
    public string ApproverEmail { get; set; }
    public bool HasCompanyIssuedIdBadge { get; set; }
    public int Score { get; set; }

    public virtual QuizType QuizType { get; set; }
}

public class QuizAnswer {

    public QuizAnswer() {
    }

    public int Id { get; set; }
    public bool isCorrectAnswer { get; set; }
    public string Answer { get; set; }
    public int QuizQuestionId { get; set; }

    public virtual QuizQuestion QuizQuestion { get; set; }
}

So basically I can create a Quiz Type, then for each type I can create multiple questions, each question can have multiple answers. I'm not trying to post back any data for QuizType, QuizQuestion, or QuizAnswer. Those are just there to help build my page[s].
This is where I start scratching my head. I need to be able to iterate through QuizQuestions and QuizAnswers to create the multiple choice form. But I also need to bind part of that form to QuizResults for posting back....  here's what I've got now in order to display the form (but not work correctly).
First, I've created a ViewModel:
public class QuizViewModel {
    public IQueryable<QuizQuestion> QuizQuestions { get; set; }
    public QuizResult QuizResults { get; set; }
}

Then in my controller:
    public ViewResult Index() {

        var questions =
            from q in unitOfWork.QuizQuestionRepository.Get()
            where q.QuizType.Name.Equals("QuizTypeOne")
            select q;

        var qvm = new QuizViewModel {
            QuizQuestions = questions
        };

        return View(qvm);
    }

I won't post my entire razor view, but I think these are the pertinent parts:
@model QuizViewModel

@* *** MISC STYLE, JS, ETC LEFT OUT FOR BREVITY *** *@

@using (Html.BeginForm()) {

    @Html.LabelFor(model => model.QuizResults.TesterName)

    @Html.EditorFor(model => model.QuizResults.TesterName)

    @* *** OTHER FORM FIELDS LEFT OUT FOR BREVITY *** *@

    foreach (var item in Model.QuizQuestions) {

@item.Question

            @{
                // I've been tinkering with this trying to find a way to get it
                // so that the input would have the right id to be picked up and
                // dropped into my post object correctly
                var inputId = "SelectedAnsers[" + item.Id + "]";
            }
            @foreach (var answer in item.QuizAnswers) {
                <input type="radio" id="@inputId" name="@inputId" value="@answer.Id" /> @answer.Answer @:(@answer.isCorrectAnswer)

            }

    }

}

Back in my controller I have a method setup to handle the post, but it's not doing anything at the moment. I'm just running in debug and looking at the values posted when I submit the form.
[HttpPost]
public ViewResult DukeDrive(QuizViewModel quizViewModel) {
    // I have a breakpoint set so that I can inspect quizViewModel
    return View();
}

In my head I want to be able to process the quiz form values, calculate the score, then create a new QuizResult object based on the contact info filled out and the newly calc'd score. I'll be darned if I can find the right way to do it though.
Any pointers?
Answer: You could add another parameter of type FormCollection which contains all posted values.
[HttpPost]
public ViewResult DukeDrive(QuizViewModel quizViewModel, FormCollection formData) {
    // I have a breakpoint set so that I can inspect quizViewModel
    return View();
}

Comment: Oh, I didn't realize both would pass through! I'll give that a shot real fast.
Comment: Yep, that gets me where I need to be... of course it's that simple.  Thanks!


Confused about conversion between windows-1252 and UTF-8 encoding

Question: I have a legacy database that claims to have collation set to windows-1252 and is storing a text field's contents as
I'd
When it is displayed in a legacy web-app it shows as I'd in the browser.  The browser reports a page encoding of UTF-8.  I can't figure out how that conversion has been done (almost certain it isn't via an on-the-fly search-and-replace).  This is a problem for me because I am taking the text field (and many others like it) from the legacy database and into a new UTF-8 database.  A new web app displays the text from the new database as
I'd
and I would like it to show it as I'd.  I can't figure out how the legacy app could have achieved this (some fiddling in Ruby hasn't showed me a way to affect converting a string I'd to I'd).
I've tied myself in a knot here somewhere.
Comment: Your dbase contains junk.  Inserted by a program that ignored the encoding and used utf8.  And as luck would have it, read by a program that ignored it as well.  Not usually luck.  It works 99% right.
Answer: It probably means the previous developer screwed up data insertion (or you're screwing up somewhere). The scenario goes like this:

the database connection is set to latin1
app actually sends UTF-8 to database
database interprets received data as latin1, stores it as such (interprets ' as ')
app queries for the data again
database returns ' encoded in latin1
app interprets the data as UTF-8, resulting in '

You essentially need to do the same misinterpretation to get good data. Right now you may be querying the database through a utf8 connection, so the database returns ' encoded in UTF-8. What you need to do is query through a latin1 connection and interpret the data as UTF-8 instead.
See Handling Unicode Front To Back In A Web App for a more detailed explanation of all this.
Comment: Yes that makes sense thanks @deceze. I'd like to do a one off conversion then. Don't want to touch the old database.