Can the hypothesis in this theorem on commutative monoids be weakened?

Question: Let $(M,*,1)$ be a commutative monoid. Define the binary relation $R$ on $M$, such that $xRy$ iff $(\exists z)(x*z=y)$. It is easy to show that, since $M$ is a commutative monoid, the relation $R$ is
both transitive and reflexive. I read in a paper the theorem that if additionally $M$ is cancellative and pure, (where pure means $1$ is the only invertible element), then $R$ is also antisymmetric. I am wondering if this hypothesis can be weakened. That is, if $M$ is merely cancellative, is $R$ antisymmetric? And also, if $M$ is merely pure, is $R$ antisymmetric? All of this is given the background assumption that $M$ is a commutative monoid.
Comment: If $x$ is invertible, then $x\mid 1$ and $1\mid x$, so pure is needed. What *can* be weakened is that $M$ need not be commutative.
Comment: @HagenvonEitzen What about cancellative? Is it needed?
Answer: After some tinkering around, I found the answer to my question. Pure is needed, because otherwise $x | 1$ and $1 | x$. As for cancellative, here is a counter-model I found where pure alone is not sufficient: Let $M = \{0,1,2\}$, let the identity of $M$ be $0$, and let $+$ be defined as $0+x=x+0=x$, $1+1=2+2=1$, and $1+2=2+1=2$. Then $(M,+,0)$ is a commutative pure monoid, where the divisor relation $R$ is not anti-symmetric, as both $1R2$ and $2R1$
Answer: Today, I replied to another related question which weakens the assumptions in another way:  A weak cancellation property for monoids
Essentially, the assumption of cancellativity can be weakened.  If the commutative monoid $M$ with identity $1$ has the property that for all $a,b\in M,$ $ab=a$ implies $b=1$ (perhaps called weak cancellativity), then $R$ is antisymmetric.
If a monoid is pure, it need not be antisymmetric.  Just look at $\{-1,1\}$ with multiplication.
Can you please post a reference to the article you mentioned?  I would be very interested in it.
Comment: Just realized that the example $\{-1,1\}$ is incorrect.  I believe everything else stated remains correct.


angularjs - confusion about nested named views with angular-ui-router

Question: Using angular-ui-router, an external routing plugin for angular found here, I am attempting to understand how the nested named views work. I got the initial demos working fine, but tried to make some modifications, like this;
$stateProvider
   .state('default', {
      url: "/default",
      views: {
         "viewA" : { templateUrl: "angular/partials/default.A.html" },
         "viewB" : { templateUrl: "angular/partials/default.B.html" }
      }
   })
   .state('state1.list', {
      url: "/state1list",
      views: {
         "viewAList" : {
            templateUrl: "angular/partials/state1.list.html",
            controller: function($scope) {
               $scope.items = ["A", "List", "Of", "Items"];
            }
         }
      }
   });

The idea is that I am setting the list contents to a named view that is nested inside of the default.A.html file.
Error
The error message I receive is this;

Error: Could not resolve 'state1.list' from state 'default'

HTML
The html looks like this;
index.html

angular/partials/default.A.html

Default.A.HTML

Show List

angular/partials/state1.list.html

List of State 1 Items

{{ item }}

angular/partials/default.B.html

Default.B.HTML

Comment: After further testing, it seems that the route name might have been the issue. When I changed the route to `default.list`, it worked. Can anyone confirm/elaborate on this?
Answer: The problem is that when you supply state1.list, the 'state1' state is considered to be the parent of the 'state1.list' state. But you don't have a state named 'state1'. Once you change it to 'default.list', the inferred parent is 'default', which works because you do have a 'default' state. If you created a new state named 'state1' and then another one named 'state1.list' then that would work as well.

Naming Your States (From https://github.com/angular-ui/ui-router/wiki/Nested-States-%26-Nested-Views)
No two states can have the same name. When using dot notation the
  parent is inferred if it is not explicitly provided, but this doesn't
  change the state name. When not using dot notation, the parent must be
  explicitly provided, but you still cannot name any two states the
  same, for example you can't have two different states named "edit"
  even if they have different parents.
Comment: An important detail that I wish I had found on the docs. Thank you so much for clarifying.


Keep textarea from changing size

Question: I'm using jQuery to alter the font size and line height of text in a textarea, but when these values are changed, the textarea gets really tall (if changing the line height to be bigger) or really tiny (shrinking the entire thing, if making the font size smaller.)
I would hopefully like to fix this problem without getting rid of the draggable resize feature that is built into the textarea, but if that's not possible then that's fine. It's more important to me that these other problems are fixed.
Here is my javascript (jQuery) code:
  $("#text").css("line-height", lineSpace);

  $("#text").css("letter-spacing", letterSpace+"px");

  $("#text").css("font-size", textSize+"px");

Any help would be greatly appreciated.
Comment: Have you tried CSS?
`textarea {
    resize: none;
}`
Answer: <textarea style="resize:none"  name="name" cols="num" rows="num"></textarea>

Answer:

$(document).ready(function() {
  var lineSpace = 2;
  var letterSpace = 5;
  var textSize = 20;
  $(".fixedTextarea").css("line-height", lineSpace);
  $(".fixedTextarea").css("letter-spacing", letterSpace+"px");
  $(".fixedTextarea").css("font-size", textSize+"px");
});

<textarea style="overflow:auto;resize:none;width:200px;height:200px;" class="fixedTextarea" col="10" rows="5">
 I'm using jQuery to alter the font size and line height of text in a textarea, but when these values are changed
</textarea>

Fixed your TextArea with width & height.

Keep textarea from changing size


How many isosceles triangles do you need to make any polygon?

Question: A while back, I had a question regarding constructing shapes with only isosceles triangles. I decided to give it a go again and it has once again stumped me. The question is:
How many isosceles triangles would you need to be able to construct any $n$ sided polygon?
I thought induction could work, but dealing with non convex polygons may make this difficult. I tried by finding polygons that require what I thought was the maximum amount, but this is unreliable and may not be enough to find a general expression in terms of $n$.
How would one go about solving a problem like this? Is this a well known result? Any help or guidance would be greatly appreciated!
Comment: Did you try it with quadrilaterals first? First, simple ones, then the difficult scalene-types.
Comment: I considered cyclic, non cyclic, convex and non convex, but I couldn't really get anywhere with it.
Comment: What about a $3-4-5$ right triangle?
Comment: A 3-4-5 triangle (or any right triangle) would be constructed with 2 isosceles triangles. But some triangles cannot be constructed with only 2 isosceles so the minimum needed to construct all triangles would be greater. @Somos
Answer: There is a two ears theorem which states for $n > 3$, any simple $n$-gon has at least two  ears.
If one split a $n$-gon at an ear, we get a triangle and a $(n-1)$-gon.
For any triangle $ABC$ with $BC$ being the longest side.

If $ABC$ is a right triangle, the circumcenter $O$ coincides with midpoint of $BC$, we can decompose $ABC$ into two isosceles triangle $ABO$, $AOC$.
Otherwise, let $D$ be the foot on $BC$. We can split $ABC$ first into two
right triangles $ABD$, $ADC$ and then into $4$ isosceles triangles.

In general, we can decompose any triangle into at most $4$ isosceles triangles.
By induction on $n$, we find we can split a $n$-gon into at most $4n$ isosceles triangles. This bound is probably not optimal but at least we know the decomposition is always possible.
Update
For general triangle, the bound $4$ is optimal.
If triangle $ABC$ is acute, we can decompose it into $3$ isosceles triangles: $AOB$, $BOC$ and $COA$. If $ABC$ is a right triangle, $2$ is enough. This leaves us with the case of obtuse scalene triangles.
A literature search indicate in $2004$, Kosztolányi, et al${}^{\color{blue}{[1]}}$ has studied the problem of decomposing obtuse scalene triangles into $3$ isosceles triangles.
With help of a computer, they found there are $23$ families of solutions.
Let $\alpha > 90^\circ > \beta > \gamma$ be the angles of the trangle.
In all these solutions, $\alpha, \beta$ are rational linear combinations of $180^\circ$ and $\gamma$.
In particular, this implies triangle with angles
$$(\alpha,\beta,\gamma) = \left( (5-\sqrt{2})\cdot 30^\circ, 30\sqrt{2}^\circ, 30^\circ \right)$$
cannot be decomposed into $3$ isosceles triangles. It is not hard to verify we cannot decompose this triangle into $2$ isosceles triangles. This means for general triangle, the bound $4$ is optimal.
Notes

$\color{blue}{[1]}$ - Kosztolányi, József & Kovács, Zoltán & Nagy, Erzsébet. (2004). Decomposition of triangles into isosceles triangles II. Complete solution of the problem by using a computer. Teaching Mathematics and Computer Science. 2. 275-300. 10.5485/TMCS.2004.0059.
An online copy can be found here.
Comment: @Schnappi I have no idea what is the optimal one. As you can see, by the time you make the comment, I already improve the bound from $6n$ to $4n$. This sort of problem tend to be hard. The best course of action is doing a literature search.
Comment: This way makes sense, but I'm looking for the proven ideal case. Is there any way to prove that this could be the ideal case? If not, how would I be able to find the ideal case (and prove it)?
Answer: If I combine the isosceles triangle side by side, I'll create a different polygon, but at some certain combination, it will still produce same shape
Just like when I join two parallelogram side by side, it also creates a parallelogram which is still a quadrilateral
so I'll assume your question about the minimum number of isosceles triangle needed to make a polygon of $n$ sides
$1$ ∆ is a $3$ sided polygon
$2$ ∆ makes a quadrilateral
$3$ ∆ makes a 5 sides polygon
so at least $n$ number of isosceles triangle makes a $(n+2)$ sided polygon
Comment: The question is more working the other way around. The question wants it for all possible polygons with n sides, if it were broken down into isosceles triangles, what would be the most ideal way to do this to minimise the amount of triangles? In other words, if you were trying to break up the shape in as least amount of isosceles triangles as possible, what would be the most triangles you require to make up an n sided polygon?


Best way to optimize and merge these Jquery functions

Question: I have 3 functions that are all set with click events that trigger the same functions within their respective divs. I'd like to optimize this so I'm not repeating myself over and over. What would be the best way to write this?
//Progress Bar animation when tab clicked
$('#tab1').click(function () {
    $('#stack .progress div').each(function () {
        var display = $(this),
        currentValue = parseInt(display.text()),
        nextValue = $(this).attr('data-values'),
        diff = nextValue - currentValue,
        step = (0 < diff ? 1 : -1);
        if (nextValue == '0') {
            $(display).css('padding', '0');
        } else {
            $(display).css('color', '#fff').animate({
                'width': nextValue + '%'
            }, 'slow');
        }

        for (var i = 0; i < Math.abs(diff); ++i) {
            setTimeout(function () {
                currentValue += step
                display.html(currentValue + '%');
            }, 20 * i);
        }
    })
});
$('#tab2').click(function () {
    $('#design .progress div').each(function () {
        var display = $(this),
        currentValue = parseInt(display.text()),
        nextValue = $(this).attr('data-values'),
        diff = nextValue - currentValue,
        step = (0 < diff ? 1 : -1);
        if (nextValue == '0') {
            $(display).css('padding', '0');
        } else {
            $(display).css('color', '#fff').animate({
                'width': nextValue + '%'
            }, 'slow');
        }

        for (var i = 0; i < Math.abs(diff); ++i) {
            setTimeout(function () {
                currentValue += step
                display.html(currentValue + '%');
            }, 20 * i);
        }
    })
});
$('#tab3').click(function () {
    $('#other .progress div').each(function () {
        var display = $(this),
        currentValue = parseInt(display.text()),
        nextValue = $(this).attr('data-values'),
        diff = nextValue - currentValue,
        step = (0 < diff ? 1 : -1);
        if (nextValue == '0') {
            $(display).css('padding', '0');
        } else {
            $(display).css('color', '#fff').animate({
                'width': nextValue + '%'
            }, 'slow');
        }

        for (var i = 0; i < Math.abs(diff); ++i) {
            setTimeout(function () {
                currentValue += step
                display.html(currentValue + '%');
            }, 20 * i);
        }
    })
});

Answer: Add a data attribute to the HTML tags representing the tab. For instance:
data-progress="#stack"

data-progress="#design"

data-progress="#other"

Define a function, like this:
function tabClicked() {
    $($(this).data("progress") + ' .progress div').each(function () {
        var display = $(this),
        currentValue = parseInt(display.text()),
        nextValue = $(this).attr('data-values'),
        diff = nextValue - currentValue,
        step = (0 < diff ? 1 : -1);
        if (nextValue == '0') {
            $(display).css('padding', '0');
        } else {
            $(display).css('color', '#fff').animate({
                'width': nextValue + '%'
            }, 'slow');
        }

        for (var i = 0; i < Math.abs(diff); ++i) {
            setTimeout(function () {
                currentValue += step
                display.html(currentValue + '%');
            }, 20 * i);
        }
    });
}

And use it as a click handler:
$("#tab1, #tab2, #tab3").click(tabClicked);

Comment: This is pretty, can I ask why you would add the 'data-progress' attribute? Wouldn't it work just fine without since the tabs already have an href with the same id reference?
Comment: @JuanPabloUgas, that's a good question. The data-progress attribute was added because the three separate functionalities used different selectors in that .each() call at the start of the function and on functionality level I believe it is more elegant to be agnostic to selectors and to use $(this).data("progress") instead. The .data() call can work if and only if we define the data attribute.
Answer: I prefer burning one thing in one oven. If you need to change one thing, you should need to change it at one place.
function doAnimation() {
    var display = $(this),
    currentValue = parseInt(display.text()),
    nextValue = $(this).attr('data-values'),
    diff = nextValue - currentValue,
    step = (0 < diff ? 1 : -1);
    if (nextValue == '0') {
        $(display).css('padding', '0');
    } else {
        $(display).css('color', '#fff').animate({
            'width': nextValue + '%'
        }, 'slow');
    }

    for (var i = 0; i < Math.abs(diff); ++i) {
        setTimeout(function () {
            currentValue += step
            display.html(currentValue + '%');
        }, 20 * i);
    }
}
//Progress Bar animation when tab clicked
$('#tab1').click(function () {
    $('#stack .progress div').each(doAnimation);
});
$('#tab2').click(function () {
    $('#design .progress div').each(doAnimation);
});
$('#tab3').click(function () {
    $('#other .progress div').each(doAnimation);
});

Comment: Although there I knew there were multiple ways of doing what I wanted, this seemed to make it pretty cut and dry and did not require adding any additional type tags on my view. Thank you!
Comment: This is generating the following error "Uncaught TypeError: Cannot read property 'call' of undefined"
Answer: var progressTargets = {tab1 : "stack", tab2 : "design", tab3 : "other"};
$('#tab1, #tab2, #tab3').click(function () {
    $("#"+progressTargets[$(this).attr("id")]+' .progress div').each(function () {
        var display = $(this),
        currentValue = parseInt(display.text()),
        nextValue = $(this).attr('data-values'),
        diff = nextValue - currentValue,
        step = (0 < diff ? 1 : -1);
        if (nextValue == '0') {
            $(display).css('padding', '0');
        } else {
            $(display).css('color', '#fff').animate({
                'width': nextValue + '%'
            }, 'slow');
        }
        for (var i = 0; i < Math.abs(diff); ++i) {
            setTimeout(function () {
                currentValue += step
                display.html(currentValue + '%');
            }, 20 * i);
        }
    });
});

try using array to target relevant progress bars.


Playing a Coltrane polychord and the limits of space

Question: As I was the reading sheet music for John Coltrane's A Love Supreme, I noticed a chord I had never seen before. It looks like this:

I'm familiar with slash chords, but never had to practically encounter a polychord like Gm over Fm7.
The notes would be G, Bb, D played on top of F, Ab, C, Eb. Is this feasible to play on a six-string guitar? In other words, is there any theory of consolidation in terms of playing polychords with limited space?
Comment: What is the source of this chord chart? I lost about 15 real/fake books about 8 years ago, (different people transcribe these things in different ways); and it's been a decade since I listened to this album... you would probably be well served listening to Coltrane's version to get a real sense of what's going on, a huge mistake many guitarists make is playing to many notes in a chord; soloists appreciate a little space to work with... this could be as simple as bass plays the root, guitar plays minor 3rd & 7th, and the soloist plays anything else he wants to add (give him room to blow)!
Comment: @DavidAxtellMooreII: The sheet music is from a Hal Leonard book called "The Music of John Coltrane." What is special about the chord chart in parentheses is that they were provided by Alice Coltrane. Besides being John's wife, she also played/performed as pianist with his band. So I certainly take her transcriptions as among the most respected.
Comment: @Aryeh - That does seem pretty definitive doesn't it? Although that does point out the difference between the way pianists think in terms of polychords (two-hands, ten fingers on 88 keys) vs. guitarists with six strings (7 or 8, but lets think traditionally for simplicities sake) and only 4 fingers in the left hand. Most of what I'e learned about polychords from a pianist's point of view were from Frank Mantooth's books on quartal harmony voicings [link](http://www.amazon.com/Voicings-jazz-keyboard-Frank-Mantooth/dp/B00071G21S). A good book for pianists/arrangers... TBC...
Comment: @Aryeh - oddly (ironically) these voicings are tricky on piano, but easy on guitar... guitar's are tuned in 4ths after all, except the 2nd/3rd strings of course. So oddly, I find cluster/close type chord voicings (2nd's etc.) much more interesting and colorful on guitar... which are easy on piano... but hard on guitar. Check out John Stowell type of "close" voicings as a contrast between the two concepts... (both really cool...)
Answer: Unless you have a seven string guitar, this chord is impossible to play on guitar if you want all chord degrees represented.  Since it is a G-minor chord over an Fm7, you can really think of the total composite chord as an Fm13, which is a pretty standard jazz chord for guitarists. . . or any jazz player for that matter.
What notes you leave out in part depends on the ensemble you're playing in.  For example, if you're playing in a jazz trio with guitar, sax, and drums then you're going to need to cover at least some of the rudimentary harmony (root, third, seventh, etc) with a frugal selection of upper-tertian harmony (say, the ninth and thirteenth.)  Choosing a five-note chord in this way is good for having an active bass line, which is important if you're playing fingerstyle.
If you're playing with a bassist or a pianist (or both) then usually with upper-tertian chords the guitar plays a lot of the "active" or "filler" harmonies - notably chord extensions and other active tones.
On the other hand, if you want to show a clear delineation between the chords (such as Coltrane notated) then it would be wise for you to voice the Fm7 below a Gm triad.  Since you will have to obviously omit one note, the fifth of the Fm7 would be omitted since it is a four note chord and the fifth is almost always omitted first as it is the least harmonically active of the pitches.
Thus, an example of an appropriate voicing would be (from lowest to highest):
F, Ab, Eb, G, Bb, D
That said, the lead sheet does not indicate the inversion of the chord, so feel free to experiment with shapes that give you the easiest flexibility - as long as the two chords remain separate.  As indicated in the score, one or both chords may be in inversion if necessary.
Hope that helps.
Comment: Harmonically speaking it is better to drop the C as it is least harmonically active.  Though if the OP were playing with a pianist / bassist they could drop the F and the C since those notes are typically covered by LH piano / bass.  Regarding the voicing, there are dozens of different voicings and positions for a chord on the guitar, which is why it's important to experiment!  Everyone has chord shapes they favor with their playing style.
Comment: ...and it'd be quite the finger twister on a seven string in standard tuning.
Comment: with a low B, that'd be 6-4-3-1-0-..  or would you barré at 3 and drop the Eb?
Comment: Not every note in the chord need be voiced perfectly simultaneously, merely within the bar at some point... and even then, not really, I agree with @jjmusicnotes, you can always ditch a natural fifth, this isn't metal, that's a boring note (INMHO, and the majority of teachers I've studied with); if you've got a bass player, he should have that covered already, in which case reducing redundancy is a good thing. If you are playing solo, the 3rd & 7th of the Fmin7 are the color tones (they're the ones that really define the tonality of Fmin7)
Comment: //^^Not every note in the chord need be voiced perfectly simultaneously, merely within the bar at some point... and really, not even then. I agree with @jjmusicnotes, you can always ditch a natural fifth, this isn't metal, that's a boring note, if you've got a bass player, he should have that covered already, in which case reducing redundancy is a good thing. The 3rd & 7th of the Fmin7 are the guide tones, they're the ones that really define the tonality as being Fmin7, the Gmin just adds the natural extensions of Fmin7 (i.e. 9, 11, 13); all 4 bars are just Fmin7 (extended).
Answer: Possibly the chord marked is not just for guitarists. To get a good voicing, the notes need to be spread over maybe 3 octaves - not easy on a guitar.With clashes like G and Ab and D and Eb,the notes won't sound good next to each other, on any instrument. When they're nearly an octave apart, they often sound great.As there are 7 notes anyway, at least one would have to be sacrificed- as usual on guitar when playing extended chords - just get someone else in the band to play the missing ones ! I guess that if the potentially clashing notes were played on different instruments with different tones/sounds, it could sound o.k. Try one basic chord on guitar while the keyboard plays the other.No, I've never seen this sort of 'slash chord' before either.
Comment: This type of notation is for a polychord, where two individual chords are overlayed onto one another to form a new composite chords.  This is different from "slash" chords which only tell you which note is in the bass.
Comment: Closely voiced notes are cool! this is Coltrane! I go out of my way to add seconds in a voicing, go look up a Guitarist named John Stowell.

>"Stowell holds his guitar in a diagonal position, which facilitates playing close, piano-like voicings more comfortably."[http://en.wikipedia.org/wiki/John_Stowell]
Comment: I guess one other way to call this chord could be Gm7b9b6sus4 so it does make more sense to name it with 2 separate well-known chords instead.
Answer: Play all fourths starting at G, you have the notes G, C, F, Bb, Eb, and Ab which is all the notes of the chord except for the D which is the 5th of the Gm.
Comment: While it does contain most of the notes of the polychord Gm/Fm7, this voicing makes it more of quartal than polychod thus not really not fully accomplishing what is desired.
Answer: The "lazy" way to play ordinary slash chords, is for something chordal in the mid-range (like a guitar, or right hand piano) to play the chord above the slash, and for something monophonic in the bass range (like a bass guitar, or left hand piano) to play the note below the slash.
I wonder whether an adaptation of that would apply here. The mid-range instrument plays Gm, the bass plays an arpeggio, or similar, using notes from Fm7.
Jazz!
Comment: Poly chords are intended to be a very harmonically full sounding chord.  Splitting the polychord between two instruments negates the polychordal musical effect and thus renders it useless.  It is best in this circumstance for guitarists to choose a couple notes from both chords and play them together - leaving the bass / piano to fill out any other necessary harmonies.


Given a row with a list too big to explode(), how do I do a preparatory explode() to bring list size down to fit into memory?

Question: I'm trying to do an .explode on a column, and stream or sink to file, but one of the lists has 300k items (6.7mil characters if combined into a string).
import polars as pl

test = pl.LazyFrame({'col1': 'string '*1_000_000})
(test
 .with_columns(explode_me = pl.col('col1').str.split(' '))
 .explode(pl.col('explode_me'))
 .collect(streaming=True)
 .write_parquet('file.parquet')
)

This issue was created, but "a single row explodes to more than fits into memory. There is not much what we can do with the current architecture. At absolute minimum, the explosion of a single row should fit."
How do I best split the oversized lists into lists with fewer items so my later .explode will fit into memory? (possibily using pl.when())
Basically, split the string every 50k words so I can explode to 6 rows, so I can then later explode 6 rows of 50k, instead of 1 row of 300k (which overloads memory).
EDIT:
My current solution
import polars as pl

test = pl.LazyFrame({'col1': 'string '*1_000_000})
(test
 .with_columns(explode_me = pl.col('col1').str.split(' '))
 .with_columns(
     pl.col('explode_me').apply(lambda x: [x[i:i+10_000] for i in range(0, len(x), 10_000)],
                                return_dtype= pl.List(pl.List(pl.Utf8)))
  )
 .select(pl.col('explode_me'))
 .explode(pl.col('explode_me'))
 .sink_parquet('file.parquet')
)

Comment: Your current approach raises `ComputeError: The output type of 'apply' function cannot determined` - why not set the `return_dtype`? `pl.apply(..., return_dtype=pl.List(pl.List(pl.Utf8)))` works in streaming for me. Although I imagine there is a proper way to do this without apply.
Comment: Fantastic, thank you! I always seem to forget about that piece. And agreed, I'd love to not use the `.apply`
Comment: Okay, so that gets rid of the exception, but just to confirm: does it actually run your use-case without overloading memory?
Comment: Welp, last night's rest seemed to fix the issues I was having, so now yes!
Comment: Interesting, thanks. Now to try figure out how to do it without apply.
Comment: I've added a follow-up: https://stackoverflow.com/questions/76628658/
Answer: You can use the list.slice method to chunk the list into smaller lists. The below takes your example (with only 10 strings) and chunks them into 5 columns of lists of two strings. I save the chunk names and expressions in the chunk_cols so you can set the chunking logic at run time with whatever logic you want.
chunk_cols = {f'chunk{i}': pl.col('explode_me').list.slice(2*i, 2) for i in range(5)}

test = pl.LazyFrame({'col1': ' '.join([f'string_{i}' for i in range (10)])})
(test
 .with_columns(explode_me = pl.col('col1').str.split(' '))
 .with_columns(
     **chunk_cols
 )
 .explode(pl.col('chunk1'))
 .collect(streaming=True)
 .select('col1','chunk1')
)

col1                chunk1
str                 str
"string_0 strin…    "string_2"
"string_0 strin…    "string_3"

Comment: My goal is to keep it all in one column (or a single additional created column) instead of creating numerous additional columns that need to be later combined. It would be only applied to columns with strings over a certain length, are you able to accomplish it in more of a `.explode`-chaining way?
I do think the `.slice` is on the right track.


Is it possible to access data in Google Drive using a regular account without user authentication?

Question: I have a web application which have a document upload/download feature. I'd like to use Google Drive for storing these documents. Is it possible to have the web app own a regular Google  account but that doesn't require user's consent for the app to access the Drive? Or is this only possible using a Service account?
Answer: Yes it's possible. see stackoverflow.com/questions/19766912/… for a set of steps to do this. "own" isn't quite the correct term, "access" would be better.
Comment: This requires consent by having the user go to the playground.
Comment: Wrong. It requires the app developer giving consent for the app to access the app developer's account. From the question "Is it possible to have the web app own a regular Google account"


DDD: need two events to be raised for single handler

Question: How can I make sure I have received event (A) from an aggregate and event (B) from another aggregate before invoking a method from a third aggregate in the event handler? For example, I need productPaid event and productInStock event to buy a product?
Answer: Maybe you can do a query in the handler to some aggregate to check if the other event already happened.
Or, you need a "process manager" instead of an "event handler".
You need a state that is shared by the two event handlers, something that when you receive an event allows you to record that it happened (some storage), and when you handle an event you can check if you already received the other one.
You can think to a process manager like an aggregate that receives events instead of commands (it cannot reject them!)


What's $-2^{-2^{-2^{-2...}}}$?

Question: Pretty simple question that I know someone has probably asked before:
What's $-2^{-2^{-2^{-2...}}}$? Or, more specifically, what is the number that this repeated sequence approaches?  I have calculated it out to about $-0.641185$, but I have no idea what that number is or where you could get it from.
Keep in mind that I am not asking for $(-2)^{(-2)^{(-2)...}}$, this sequence doesn't really work.
The order of operations looks like this: $-(2^{-(2^{-(2^{-(2...)})})})$
It's basically like taking $-2$, exponentiating it by 2 to get $2^{-2}$, then making that negative $-2^{-2}$, then exponentiating it again, etc.
Comment: You can rewrite it as $y = -(2^y)$, and then solve for $y$ using the Lambert W function.
Answer: Recursively, we might define this as the limit as $n \to \infty$ of
$$a_n = -(2)^{a_{n-1}}$$
with $a_1 = -2$. Letting $n \to \infty$ on the assumptions of continuity and convergence of $a_n$ to some value $L$ gives us
$$L = -2^L$$
This value $L$ is fundamentally what you seek. This equation will have to be solved by means of the Lambert W function and doesn't have a closed form outside of that, to my knowledge anyhow. Note that $-2^L = -e^{\ln(2) \cdot L}$. Thus, follow the following steps:

Use the identity above
Multiply both sides by $-\ln(2)$
Multiply both sides by $e^{-\ln(2)L}$

Then you get that
$$L = -2^L \implies -\ln(2) L  e^{-\ln(2) L}= \ln(2)$$
The Lambert W function is the inverse to $f(x) = xe^x$. That is, $W(xe^x) = x$. Applying it to both sides of the above, we obtain that
$$W(-\ln(2) L  e^{-\ln(2) L})= W(\ln(2)) \implies -\ln(2)L = W(\ln(2))$$
Solving for $L$ gives us
$$ L = \frac{-W(\ln(2))}{\ln(2)}$$
Of course, this is about as much simplification as we can get; many values for the W function have to be approximated. Of course, we also should note this is the principle value since it is a multivalued function otherwise. If we approximate the value by, say, Wolfram, we see that
$$L \approx -0.641186$$

What's $-2^{-2^{-2^{-2...}}}$?


Automatically delete orphaned SIDs

Question: Does someone know a tool that allows to delete orphaned SIDs on a fileserver without the need for any scripting?
Comment: Hi. Welcome to Server Fault. We require posts to be in English. A Google translation would do. If you post the German as well someone will be able to correct any translation errors.
Comment: @SvenW I'm 85% certain this can be done with PowerShell, giving him a script would probably be as good as a tool...  (and Thomas: Don't be afraid of scripting. Microsoft FINALLY gave you PowerShell - A scripting language that doesn't suck - You should use it as much as practical)
Comment: You can use "SetACL -on D:\ -ot file -actn delorphanedsids" as described here https://helgeklein.com/blog/2012/07/finding-removing-orphaned-sids-in-file-permissions-or-busting-the-ghosts-built-into-windows-7/  You may have to download setacl from https://helgeklein.com/setacl/
Answer: You could download SubInACL.
You can use the following format to clean out orpahned SID's
<code>/cleandeletedsidsfrom=domain[=dacl|sacl|owner|primarygroup|all]
</code>
If you want this done automagically you can create a script, though I have never done so myself.

Automatically delete orphaned SIDs


Is Hyper-V CSVFS supported on Linux VMs

Question: We have a Hyper-V guest failover cluster with a few Windows VMs that are using Cluster Share Volume (CSV).
Is it possible to use the same CSV for Linux VMs? Is CSVFS supported on Linux VMs at all ?
Answer: Linux has its own clustered file systems.  If you want to share a disk among Linux machines, you need to form a cluster of those Linux machines and run a clustered file system on them.  CSVFS is a Windows technology.
You'd still probably be happier with a file server.
Comment: Yes it seems that would be the case. We have setup everything and whole csvfs over svhdx is just one complexity over another. We will try file server in the end. Thanks

Is Hyper-V CSVFS supported on Linux VMs


Laravel 5 previous request path

Question: Is there any built-in method in Laravel to get the path of the previous URL?
I know about url()->previous(); but i only want to get previous path.
Also, I can extract previous path from the previous URL. But I would like to know if there is any built-in method in Laravel.
Comment: But that is the built in method, there is no other!
Comment: Possible duplicate of [Previous route name in Laravel 5.1](https://stackoverflow.com/questions/40690202/previous-route-name-in-laravel-5-1)
Answer: For Laravel 5.3
I've gone through same problem and I had searched for the same thing but i end up writing a code to manually extract path from previous URL.
Therefore I think there is no such method provided by Laravel(till 5.3) to get only path of previous URL.

Laravel 5 previous request path


Zend_Form: identical fields returning false when actually identical

Question: I've been struggling with this for an hour now and still can't understand what's wrong.
Here's how I set my two fields:
$password = $this->createElement('password', 'password');
$password->setLabel('Password');
$password->setRequired('true');
$password->addValidator(new Zend_Validate_Regex('/^([a-zA-Z0-9]*[0-9]+[a-z]*[A-Z]+[a-zA-Z0-9]*)|([a-zA-Z0-9]*[A-Z]+[a-z]*[0-9]+[a-zA-Z0-9]*)$/'));
$password->setAttrib('size', 25);
$password->setAttrib('length', 150);
$this->addElement($password);

$confirm_password = $this->createElement('password', 'confirm_password');
$confirm_password->setLabel('Confirm');
$confirm_password->setRequired('true');
$confirm_password->setAttrib('size', 25);
$confirm_password->setAttrib('length', 150);

$confirm_password->addValidator('Identical', false, array('token' => 'password'));
//$confirm_password->addValidator(new Zend_Validate_Identical(array('token' => 'password')));

$this->addElement($confirm_password);

I've tried both validators (+ the one commented) but none works.
These declarations come from www.wjgilmore.com/blog/entry/validating_identical_passwords_with_the_zend_framework and https://stackoverflow.com/a/3653416/1300454
Both declarations always return "The two given tokens do not match" eventho I'm sure these two fields contain the exact same string.
Any guess ? Thank you!
EDIT:
just noticed thanks to XDebug that when Zend_Validate_Identical's isValid() function is called, my $token equals 'password' while my $value equals 'P4ssword' (the password I entered in my confirm_password field. What's going on?
EDIT2:
tried with these two solutions: emanaton.com/code/php/validateidenticalfield and stackoverflow.com/a/1628611/1300454 but none of them work for me either. With the generic IdenticalField validator it can't find my "password" field and the second validator just returns that the two field don't match and with XDebug again I found out that it's still looking to match my password 'P4ssword' to the word 'password' that I gave as the field name...
Answer: Is it possible you're not using current Zend Framework version? The code in the class looks like it should work as you expect it (1.11). Are you using $form->isValid() or ->isValidPartial()? Try editing the code of ZF and dumping the contents of $context variable. It looks like your code is not filling the $context variable as it should.
public function isValid($value, $context = null)
{
    $this->_setValue((string) $value);

    if (($context !== null) && isset($context) && array_key_exists($this->getToken(), $context)) {
        $token = $context[$this->getToken()];
    } else {
        $token = $this->getToken();
    }

    if ($token === null) {
        $this->_error(self::MISSING_TOKEN);
        return false;
    }

    $strict = $this->getStrict();
    if (($strict && ($value !== $token)) || (!$strict && ($value != $token))) {
        $this->_error(self::NOT_SAME);
        return false;
    }

    return true;
}

Comment: is there a way I could for sure tell what version I'm using ?
Comment: $context contains:
('controller' => 'user', 'action' => 'dovalidation', 'module' => 'default', 'confirm_password' => 'P4ssword')
Comment: Than from my point of view the problem is you set the validator to password element, instead of confirm_password. The password value is in $value and confirm is in $context. You can either move the validator or change the token to confirm_password
Comment: I didn't really understand what you were saying. Did you mean I should add the validator to 'password' element or did you imply that I do it by mistake? Because my code is exactly how it looks in my question. Btw, I've also tried to put it on password element before but doesn't work any better. Weird stuff.


Oracle SQL loader issue

Question: I have a table TABLE1 with columns R_ROW_ID, NAME, MARKS, TIME, NUMBER,LAST_UPD_TIME,USER_ID...
All column values are not null.
TABLE1 has unique index on column R_ROW_ID.
I have a file which has only NAME, MARKS data and load this data into TABLE1 using SQL Loader.
File1.txt has below data:
JOHN|83
DON|54
LONIS|91

How to populate ROLLNO with unique value , NUMBER with default value 0 and LAST_UPD_TIME with current timestamp
This file loading is one time and I  have created a sequencer SEQ as mentioned below:
 CREATE SEQUENCE SEQ START WITH 90000 INCREMENT BY 1;

Grant on this sequencer SEQ is:
  GRANT SELECT ON SEQ TO LOAD_USER;

This is the control file code I have written:
 LOAD DATA APPEND
 INTO TABLE TABLE1 FIELDS TERMINATED BY "|"
 TRAILING NULLCOLS
 (
   R_ROW_ID  "SEQ.NEXTVAL",
   NAME      CHAR "NVL(TRIM(:NAME),' ')",
   MARKS      CHAR "NVL(TRIM(:MARKS),' ')",
   TIME       CHAR "NVL(TRIM(:TIME),' ')",
   NUMBER    CONSTANT "0",
   LAST_UPD_TIME EXPRESSION "CURRENT_TIMESTAMP(3)",
   USER_ID   CHAR "NVL(TRIM(:USER_ID),' ')"
   )

This is the error it is showing :
  SQL*Loader-951: Error calling once/load initialization
  ORA-02373: Error parsing insert statement for table TABLE1.
  ORA-02289: sequence does not exist

Let me  know is there anything wrong in above one.
Is there any other way we can populate R_ROW_ID value (which should be unique) apart from using sequences ?
Comment: The error and the grant suggest you have created the sequence in a different schema; unclear which owns the table. You might just be able to specify the sequence owner as `"owner.SEQ.NEXTVAL"`. Or default it in the table definition instead.
Comment: I have added owner.SEQ.NEXTVAL and still its not working
Answer: Well, who knows what's wrong if you aren't telling the truth.

You want to set ROLLNO to something, but that column doesn't exist in column list you specified in the 1st sentence.

Then, you want to set column named NUMBER to something else; you probably don't, because number can't be column's name as it is keyword, reserved for datatype (true, there can be exceptions, but I doubt this is the issue here)
SQL> create table test (number varchar2(10));
create table test (number varchar2(10))
                   *
ERROR at line 1:
ORA-00904: : invalid identifier

Why did you create sequence in one schema (we don't know which one) and granted it to another user? Why don't you do everything in the same schema? Or did you just get a wrong idea that you should grant privilege on sequence you created to yourself?
Here's an example; see if it helps. Everything is being done in the same schema.
SQL> create table table1
  2    (r_row_id      number       not null,
  3     name          varchar2(10) not null,
  4     marks         number       not null,
  5     time          varchar2(10) not null,
  6     c_number      number       not null,
  7     last_upd_time timestamp    not null,
  8     user_id       varchar2(10) not null
  9    );

Table created.

SQL> create sequence seq start with 90000 increment by 1;

Sequence created.

Control file:
load data
infile *
replace
into table table1
fields terminated by '|'
trailing nullcols
(
  name,
  marks,
  r_row_id      "seq.nextval",
  time          "nvl(:time, ' ')",
  c_number      constant "0",
  last_upd_time "systimestamp",
  user_id       "user"
)

begindata
JOHN|83
DON|54
LONIS|91

Loading session:
SQL> $sqlldr scott/tiger@pdb1 control=test13.ctl log=test13.log

SQL*Loader: Release 188.8.131.52.0 - Production on Mon May 1 20:05:07 2023
Version 18.104.22.168.0

Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.

Path used:      Conventional
Commit point reached - logical record count 2
Commit point reached - logical record count 3

Table TABLE1:
  3 Rows successfully loaded.

Check the log file:
  test13.log
for more information about the load.

Result:
SQL> select * from table1;

  R_ROW_ID NAME            MARKS TIME    C_NUMBER LAST_UPD_TIME              USER_ID
---------- ---------- ---------- ----- ---------- -------------------------- ----------
     90000 JOHN               83                0 01.05.2023 20:05:07:394000 SCOTT
     90001 DON                54                0 01.05.2023 20:05:07:394000 SCOTT
     90002 LONIS              91                0 01.05.2023 20:05:07:397000 SCOTT

SQL>

If you really are loading data using another user, then - in my example - "this" user is scott and he grants select on seq sequence to mike. Mike is doing the rest:
SQL> show user
USER is "SCOTT"
SQL> grant select on seq to mike;

Grant succeeded.

SQL> connect mike/lion@pdb1
Connected.
SQL> create table table1
  2    (r_row_id      number       not null,
  3     name          varchar2(10) not null,
  4     marks         number       not null,
  5     time          varchar2(10) not null,
  6     c_number      number       not null,
  7     last_upd_time timestamp    not null,
  8     user_id       varchar2(10) not null
  9    );

Table created.

SQL>

Control file; note modification for r_row_id (uppercase owner name!):
load data
infile *
replace
into table table1
fields terminated by '|'
trailing nullcols
(
  name,
  marks,
  r_row_id      "SCOTT.seq.nextval",
  time          "nvl(:time, ' ')",
  c_number      constant "0",
  last_upd_time "systimestamp",
  user_id       "user"
)

begindata
JOHN|83
DON|54
LONIS|91

Loading session:
SQL> $sqlldr mike/lion@pdb1 control=test13.ctl log=test13.log

SQL*Loader: Release 18.104.22.168.0 - Production on Mon May 1 20:10:39 2023
Version 126.96.36.199.0

Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.

Path used:      Conventional
Commit point reached - logical record count 2
Commit point reached - logical record count 3

Table TABLE1:
  3 Rows successfully loaded.

Check the log file:
  test13.log
for more information about the load.

Result:
SQL> select * From table1;

  R_ROW_ID NAME            MARKS TIME         C_NUMBER LAST_UPD_TIME              USER_ID
---------- ---------- ---------- ---------- ---------- -------------------------- ----------
     90009 JOHN               83                     0 01.05.2023 20:10:39.885000 MIKE
     90010 DON                54                     0 01.05.2023 20:10:39.885000 MIKE
     90011 LONIS              91                     0 01.05.2023 20:10:39.911000 MIKE

SQL>

Alternatively, you could create a synonym for owner's sequence (still connected as mike):
SQL> create synonym seq for scott.seq;

Synonym created.

Then you don't have to modify the 1st control file I posted at all, and loading does the job:
SQL> $sqlldr mike/lion@pdb1 control=test13.ctl log=test13.log

SQL*Loader: Release 220.127.116.11.0 - Production on Mon May 1 20:13:15 2023
Version 184.108.40.206.0

Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.

Path used:      Conventional
Commit point reached - logical record count 2
Commit point reached - logical record count 3

Table TABLE1:
  3 Rows successfully loaded.

Check the log file:
  test13.log
for more information about the load.

Oracle SQL loader issue


Why do we want to minimise the variance of our importance weights in SIS with respect to the proposal distribution

Question: Is there a clear and precise explanation of why minimising the variance of the weights in SIS with respect to a proposal ensures that the samples generated from the empirical distribution induced by the normalised weights will be closer to the posterior/target distribution? Also is there a reference which proves something about the distribution (mass function) induced by SIS in relation to the target/posterior density? Like random variables sampled from SIS with increasing particle counts converge in distribution to random variables sampled from the target? --Apologies for abuse of notation, or any mis-use of terminology. It just seems like most results for SIS / SIR are provided in terms consistency with respect to arbitrary test functions, instead of the actual samples generated under the normalised weights. Most of my understanding stems from~https://www.cs.ubc.ca/~arnaud/doucet_johansen_tutorialPF.pdf, so if I missed something obvious here let me know, and thank you in advance!
Answer: Is there a clear and precise explanation of why minimising the variance of the weights in SIS with respect to a proposal ensures that the samples generated from the empirical distribution induced by the normalised weights will be closer to the posterior/target distribution?
I tend to think of this problem in terms of the effective sample size (ESS). Quoting from Monte Carlo Strategies in Scientific Computing, Liu (2008) (pdf) pp. 34-36

Importance sampling suggests estimating $\mu = E_\pi\{h(x)\}$ by first
generating independent samples $x^{(1)}, \dots , x^{(m)}$ from an
easy-to-sample trial distribution, $g( \thinspace )$, and then correcting the
bias by incorporating the importance weight $w^{(j)} \propto \pi(x^{(j)})/g(x^{(j)})$
...
A useful "rule of thumb" is to use the effective sample size (ESS) to
measure how different the trial distribution is from the target
distribution. Suppose $m$ independent samples are generated from
$g(x)$; then, the ESS of this method is defined as $$ESS(m) = \frac{m}{1 + \textrm{var}_g(w(x))}$$
...
This can be interpreted as that the $m$ weighted samples is worth of $m / \{1 + \textrm{var}_g [w(x)]\}$ i.i.d. samples drawn from the target distribution.

There's much more detail in the book.
Also is there a reference which proves something about the distribution (mass function) induced by SIS in relation to the target/posterior density? Like random variables sampled from SIS with increasing particle counts converge in distribution to random variables sampled from the target?
If you're referring to the SIS algorithm on p. 11 of Doucet and Johanson's tutorial, with no resampling, then the distribution of the particles will just be sampled from $q_n(x_{1:n})$, regardless of the target/posterior distribution. SIS with a massive number of particles might still be useful in estimating an expectation in cases where the error decreases with the number of particles.
Once resampling is introduced to make SMC then you get the results given in Section 3.6 of the tutorial

Comparing (26) to (37), we see that the SMC variance expression has replaced the importance distribution $q_n(x_{1:n})$ in the SIS variance with the importance distributions $\pi_{k−1} (x_{1:k−1}) q_k (x_k| x_{1:k−1})$ obtained after the resampling step at time $k-1$.

The authors give two citations for the results about the density in section 3.6.


What is the hardest statistical concept to grasp?

Question: This is a similar question to the one here, but different enough I think to be worthwhile asking.
I thought I'd put as a starter, what I think one of the hardest to grasp is.
Mine is the difference between probability and frequency.  One is at the level of "knowledge of reality" (probability), while the other is at the level "reality itself" (frequency).  This almost always makes me confused if I think about it too much.
Edwin Jaynes Coined a term called the "mind projection fallacy" to describe getting these things mixed up.
Any thoughts on any other tough concepts to grasp?
Comment: I'd agree (In my surprisal) - I think its that $\pi$ pops up in many mathematical analysis.  Just a note you can write $\pi$ by with Latex commands as $\text{\pi}$ enclosed within $ signs.  I use the wiki page to get the syntax http://en.wikibooks.org/wiki/LaTeX/Mathematics . Another trick is to "right click" on an equation you see on this site, and select "show source" to get the commands that were used.
Comment: (I don't know enough to put this as an answer, hence adding a comment.) I always thought it was strange that PI crops up in statistical equations. I mean - what's PI got to do with statistics? :)
Comment: @Wiki If you accept that $\pi$ crops up when you go from measuring the length of a straigh piece of line to the length of a piece of circle, I don't see why it would not appear while going from measuring a probability to fall down on a segment to measuring the probability to fall down in a piece of circle ?
Comment: @Wiki Whenever you have trigonometric funcions (sine, cosine, tangent etc.) you risk having $\pi$ pop up. And remember that whenever you derive a function you're actually finding a tangent. What is surprising is that $\pi$ doesn't appear *more* often.
Comment: @Carlos I suspect the prevalence of $2\pi$ is mostly due to the use of the $\ell^2$ metric, leading to n-spheres. In the same vein, I would expect it's $e$ whose prevalence is due to analysis.
Comment: Monty hall problem? :)
Answer: for some reason, people have difficulty grasping what a p-value really is.
Comment: @shabbychef: Most of the people grasp it in the _worst_ possible way i.e. probability of making Type I error.
Comment: I think that's mostly related to how p-values are explained in classes (i.e.: just by giving a quick definition and without specifying what p-values are NOT)
Comment: @eduardo - yes a small enough p-value is sufficient to cast doubt on the null hypothesis: but it is calculated *in complete isolation* to an alternative.  Using p-values alone, you can never formally "reject" $H_0$, because *no alternative has been specified*.  If you formally reject $H_0$, then you must also reject the calculations which was based on the assumption of $H_0$ being true, which means you must reject the calculation of the p-value that was derived under this assumption (it messes with your head, but it is the only way to reason *consistently*).
Comment: I think this is mainly to do with how it is introduced.  For me, it was an "add-on" to the classical hypothesis test - so it appears as though its just another way to do a hypothesis test.  The other problem is that it is usually only taught with respect to a normal distribution, where everything "works nice" (e.g. p-value *is* a measure of evidence in testing a normal mean).  Generalising the p-value is not easy as there is no specific principles to guide the generalisation (e.g. there is no general agreement on how a p-value should vary with the sample size & multiple comparisons)
Comment: @shabbychef +1 though student often have difficulties with p-values (roughly because the concept in testing is a bit more subtle than a binary decision process and be cause "inverting a function" is not easy to aprehend). When you say  "for some reason" do you mean it is unclear for you why people have difficulties ? PS: If I could, I would try to make statistics on this site about the relation between "being a top answer" and "talking about p-value" :) . I also even ask myself if the hardest statistical concept to grasp can have the most upvote (if it is difficult to grasp ... :) )
Comment: @robin girard - in the spirit of the last part of your comment, one quote I like, I heard it from a comedian Bill Bailey is: *I analyse things too much...or maybe not enough?*.  Another one I like (from me, not Bill Bailey): *I used to be uncertain, but now I'm not so sure...* :).  It does look like p-value is going to win
Comment: If the null hypothesis is "your girlfriend hasn't been cheating on you with your best friend," and you catch them hugging, cuddling and holding hands in a dark place, then the p-value is **really** low. Unless she is ugly, in that case, the p-value might still be high. I'm completely sure even high-school students could understand p-values that way.
Comment: @Eduardo - a reasonable alernative hypothesis "your girlfriend was not chased by a rapist into a dark place, and your best friend found her and got the rapist to leave".  If you see them hugging, cuddling, and holding hands in that dark place, this is inconsistent with the above hypothesis (unless your best friend or your girlfriend are uncomfortable about touching people).  Hence it too will have a small p-value
Comment: @probabilityislogic: Well. At least we both understand p-values.
Comment: @Eduardo - I don't really get the "ugly" argument.  "Ugly" people have sex, and for some people the saying "forbidden fruit is the sweetest" holds true.  Another thing, is that it is your best friend who needs to think she is ugly for it to apply, and you cannot be certain of this (unless you can read his/her mind).
Comment: @probabilityislogic: I was just trying to use an example even the most mentally-challenged high-school student could understand.
Comment: @eduardo - I think the problem that comes about is that the data are insufficient for the problem.  If you replaced observing "hugging, cuddling, and holding hands" with "walking in on your best friend and girlfriend having sex", then it would be much harder to come up with alternatives (although having thought about it, an alternative could be assault or blackmail)
Comment: @probabilityislogic: This is rapidly going off-topic. We might argue elsewhere.
Comment: @eduardo - I'd say the example is off topic, but the analogy isn't.  This is because you can always come up with the hypothesis of "honesty" - i.e. you can get a small p-value by saying the hypothesis is "these data were not changed to be consistent with a different hypothesis".  Because the probability of observing the particular data you observed is always small, the p-value will also be small.  hence it is only by using your *prior information* about the integrity of the data, that you can eliminate these kinds of hypothesis.
Comment: @probabilityislogic: I know. I never claimed that the p-value would be zero. But a small enough p-value is sufficient to claim "Bullcrap!" when you are (1 minus p-value) sure they lying to you. Of course, how small does "small" mean to you is a matter of policy, and statistics doesn't deal with that.
Comment: @eduardo - using p-values in hypothesis testing is similar to the logical case of requiring a set of axioms to prove their own consistency.  It just can't be done.  However you can assess a given *theorem*, and work out which sets of axioms are consistent with the theorem. (axiom=hypothesis, theorem=data)
Comment: @probabilityislogic: Again, I know. If the null hypothesis is wrong, then the test statistic is meaningless.
Comment: @Eduardo - but if you think about it, Say we reject the null hypothesis, due to a small p-value.  Once we have rejected the null; this means that the decision to reject the null hypothesis was based upon a *meaningless statistic*.  Therefore, this invalidates the rejection we just made!
Comment: @probabilityislogic: No it doesn't invalidate the rejection we just made. We have already concluded that the null hypothesis can't be true. The p-value is meaningless because it is defined in terms of something that isn't true. From a strictly logical point of view, the rejection of the null hypothesis (on the premise that p-values shouldn't be that low) just forces you to come up with another null hypothesis or stay content with being ignorant about what's going on.
Comment: I think the above discussion between @eduardo and myself shows quite clearly why p-values are a conceptual nightmare.  even in a seemingly obvious example, they do not appear to be what we think they are.
Comment: @eduardo - So you would consider second order logic to be fallicious reasoning?  because that is what your answer implies.  If you reject the *conditions* ($H_0$ is a condition of the p-value) then you must reject the conclusions which were based on those conditions.  The same thing goes for axioms: you cannot simultaneously reject an axiom, but keep the theorem that depends on it for its proof.
Comment: @probabilityislogic: That's where you're wrong. The p-value isn't a _conclusion_ of the null hypothesis, but a _means to determine its believability_. Thus, if the p-value is too low, I can say, "Hahaha! Bullcrap!"
Comment: if the p-value isn't a conclusion of the null hypothesis, they why does it appear as one of the conditions required when calculating the null hypothesis.  presumably, your response implies that the p-value could be constructed without the null hypothesis.  And you cannot determine the *believability of the null* without referring to the prior probability of the null.  Otherwise, you are to believe incompatible null hypothesis simulatneously (the data is fake vs the data is not fake can give the same p-value)
Comment: @eduardo - suppose *every* p-value for every null you could think of (including faking the data) was smaller than $10^{-5}$, what are you to believe?
Comment: @probabilityislogic: Setting the alpha of a test is a matter of "policy," if you want to give it a name. Given that the null hypothesis is true, the p-value could be _anything_ from 0 to 1, even a small value. So, from a strictly logical point of view, you cannot _decide_ (in a deductive way) whether the null hypothesis is true or not. So, at the risk of being wrong, you establish that the p-value shouldn't be lower than alpha, otherwise, you _don't believe_ that the null hypothesis is true, regardless of whether it is actually true, which you will never know. There is no paradox.
Comment: @Eduardo - but if you don't believe the null is true, then you are acting *as if* $P(H_0)=0$.  And when this occurs, the conditional probability based on $H_0$ is *undefined* $P(T>T_{obs}|H_0)=\frac{P(T>T_{obs},H_0)}{0}$.  By putting your *prior probability* of $H_0$ in addition to the p-value $P(H_0)\times P(T>T_{obs}|H_0)$ the $P(H_0)$ in the denominator cancels out, and you are no longer dividing by zero, and you are just left with the joint probability.  This is a more sound way to reason because you are reject the null AND the data as a combination.
Comment: @probabilityislogic: No. Not believing it is plausible that the null hypothesis be null doesn't mean acting as if the probability of the null hypothesis were zero. It means "it might be true, but I won't _bother_ taking that possibility into consideration anymore." Remember that, in real life, we use statistical inference to make decisions when we don't have enough information to use the strictly deductive approach.
Comment: @Eduardo - but rejecting the null hypothesis *does* mean you are acting *as if it is false*.  If you are using p-values as a heuristic guide to your inference, then there is no quams with that.  But a low p-value on its own is *not a direct measure of the evidence* against the null( [Bernardo and Rueda](http://www.stat.duke.edu/research/conferences/valencia/IntStatRev.pdf) ).  [This paper](http://predictive.files.wordpress.com/2010/03/binder1.pdf)  also shows some of the logical flaws of p-values
Comment: @probabilityislogic: Who said statistical inference was a logical process? Basically, the situation is this: Someone would like to know something about a distribution (usually to make a decision), however, all he has is a limited subset of data drawn from that distribution. **From a strictly logical point of view, he simply does not have enough information to do anything at all. But he must do something anyway.** Thus, he introduces "premises" (such as, if a p-value were calculated, it shouldn't be lower than alpha) in order to make up for the information he doesn't have.
Comment: @Eduardo - I think you would have a hard time convincing someone that abandoning logic is the best thing to do.  You simply move from *deductive* logic to *inductive* logic when you have insufficient information to do *deductive* logic.  And of course he must do something - but the something that he does should be consistent, based on a proper test statistic.  A proper test statistic has to be defined when the null is true and when it is false. A p-value is not, as my previous comments show.
Comment: @probabilityislogic: Inductive "logic" is not logic at all; it is just (sometimes educated) guessing. Fortunately for us, some forms of induction (such as statistical inference) are _reliable enough_, so the risk associated to the possibility that the induction be wrong is overshadowed by the benefit of getting more information, even if it is somewhat uncertain. We don't seek to have perfect information, just enough information to make good decisions. Remember that both deduction and induction are just tools.
Comment: @Eduardo - The problem with most forms of statistical inference is that they are hard to generalise, becuase they are often based on intuition rather than solid foundations.  The p-value is such an example (how to adjust for multiple comparisons, how to deal with nuisance parameters, arbitrary choice of the statistic on which the p-value is based).  Without a set of more broad axioms or desiderata to guide the generalisation, p-values are basically restricted to the realm of 1 parameter problems, or where pivotal quantities exist.
Comment: @probabilityislogic: I don't think it's desirable to generalize p-values to n-parameter problems. I find it conceptually simpler to reduce those problems to 1-parameter goodness of fit problems. It might not fit your definition of beautiful (it certainly doesn't fit mine), but it works.
Comment: @Eduardo - but then how do you adjust the p-value to account for the multiple 1-parameter comparisons you are making?  there is no principles to guide you, only your intuition.
Comment: @probabilityislogic: I'm more often interested in the whole model than in individual 1-parameters. If the model works (i.e., helps me understand how a physical system works, and predict how it would behave given certain conditions), then each parameter has a good enough value.
Comment: @Eduardo: I think we're talking at different levels here.  p-values usually give good indications in models (such as OLS regression) of the size of parameters.  This is because they are based on a sufficient statistic, and no nuisance parameters (T-test).  But once you leave this "safe" area (e.g. non-linearity), p-values do not necessarily have good properties. They can be useful as a guide, simply because the tend to be easy to calculate.  It makes more sense to calculate the prob of the null, given the data, because we know what data we saw, but do not know what hypothesis to accept
Comment: ...cont'd... why do we calculate the probability of something which is certain (the data)?  And why do we condition on something which is uncertain (the null)?  This has always seemed backwards to me (although, if by prob of "data" we actually mean "future data", then it makes a bit more sense).  To me, it makes more sense to condition on what you observe, because you cannot "unobserve" anything once it has been observed.
Answer: Similar to shabbychef's answer, it is difficult to understand the meaning of a confidence interval in frequentist statistics. I think the biggest obstacle is that a confidence interval doesn't answer the question that we would like to answer. We'd like to know, "what's the chance that the true value is inside this particular interval?" Instead, we can only answer, "what's the chance that a randomly chosen interval created in this way contains the true parameter?" The latter is obviously less satisfying.
Comment: The more I think about confidence intervals, the harder it is for me to think of what kind of question they can answer at a conceptual level that cannot be answered by asking for "the chance a true value is within an interval, given one's state of knowledge".  If I were to ask "what is the chance (conditional on my information) that the average income in 2010 was between 10,000 and 50,000?" I don't think the theory of confidence intervals can give an answer to this question.
Answer: What is the meaning of "degrees of freedom"? How about df that are not whole numbers?
Answer: Conditional probability probably leads to most mistakes in everyday experience. There are many harder concepts to grasp, of course, but people usually don't have to worry about them--this one they can't get away from & is a source of rampant misadventure.
Comment: For starters: P(you have the disease|test is positive) != P(test is positive|you have the disease).
Comment: +1; could you add an example or two, favourite or current ?
Answer: I think that very few scientists understand this basic point: It is only possible to interpret results of statistical analyses at face value, if every step was planned in advance. Specifically:

Sample size has to be picked in advance. It is not ok to keep analyzing the data as more subjects are added, stopping when the results looks good.
Any methods used to normalize the data or exclude outliers must also be decided in advance. It isn't ok to analyze various subsets of the data until you find results you like.
And finally, of course, the statistical methods must be decided in advance. Is it not ok to analyze the data via parametric and nonparametric methods, and pick the results you like.

Exploratory methods can be useful to, well, explore. But then you can't turn around and run regular statistical tests and interpret the results in the usual way.
Comment: I think John Tukey might disagree http://en.wikipedia.org/wiki/Exploratory_data_analysis ;o)
Comment: I would partially disagree here.  I think the caveat that people miss is that *the appropriate conditioning operations are easy to ignore* for these kinds of issues.  Each of these operations change the conditions of the inference, and hence, they change the conditions of it applicability (and therefore to its generality).  These is definitely only applicable to "confirmatory analysis", where a well defined model and question have been constructed.  In exploratory phase, not looking to answer definite questions - more looking to build a model and come up with hypothesis for the data.
Comment: Ultimately the real problem with the issues raised in the initial answer is that they (at least partially) invalidate p-values.  If you are interested in quantifying an observed effect, one should be able to do any and all of the above with impunity.
Comment: For me, the "excluding outliers" is not as clearly *wrong* as your answer implies.  For example, you may only be interested in the relationships at a certain range of responses, and excluding outliers actually helps this kind of analysis.  For example, if you want to model "middle class" income, then excluding the super rich and impoverished outliers is a good idea.  It is only the outliers within your frame of inference (e.g. "strange" middle class observations) were your comments apply
Comment: I edited my answer a bit to take into account the comments of Dikran and probabilityislogic. Thanks.
Comment: @drknexus - I wouldn't say they invalidate the p-value per say, rather that the null hypothesis that it is based on is implicitly changed
Comment: @probablityislogic: Agreed.  In practice, since we assume a straightforward null and the underlying probabilities governing various choices the experimenter made are probably undefined, do we have any realistic hope of converting a mangled p value into our standard frame of reference?  I guess the difficulty of this task is what prompted me to call them invaldated.
Comment: As far as sample size is concerned: there is this adaptive sampling approach that people seem to take seriously. So I guess you don't always have to pick your sample size up front.
Answer: Tongue firmly in cheek: For frequentists, the Bayesian concept of probability; for Bayesians, the frequentist concept of probability.  ;o)
Both have merit of course, but it can be very difficult to understand why one framework is interesting/useful/valid if your grasp of the other is too firm.  Cross-validated is a good remedy as asking questions and listening to answers is a good way to learn.
Comment: I rule I use to remember: Use probabilities to predict frequencies.  Once the frequencies have been observed, use them to evaluate the probabilities you assigned.  The unfortunately confusing thing is that, often the *probability* you assign is equal to a *frequency* you have observed.  One thing I have always found odd is why do *frequentists* even use the word probability? wouldn't it make their concepts easier to understand if the phrase "the frequency of an event" was used instead of "the probability of an event"?
Comment: Interestingly, cross validation can be seen as a Monte Carlo approximation to the integral of a loss function in Decision Theory.  You have an integral $\int p(x) L(\textbf{x}_{n},x) dx$ and you approximate it by $\sum_{i=1}^{i=n} L(\textbf{x}_{[n-i]},x_i)$ Where $\textbf{x}_{n}$ is data vector, and $\textbf{x}_{[n-i]}$ is the data vector with the *ith* observation $x_i$ removed
Answer: From my personal experience the concept of likelihood can also cause quite a lot of stir, especially for non-statisticians. As wikipedia says, it is very often mixed up with the concept of probability, which is not exactly correct.
Answer: Fiducial inference. Even Fisher admitted he didn't understand what it does, and he invented it.
Answer: What do the different distributions really represent, besides than how they are used.
Comment: This was the question I found most distracting after statistics 101. I would encounter many distributions with no motivation for them beyond "properties" that were relevant to topics at hand. It took unacceptably long to find out what any represented.
Comment: Maximum entropy "thinking" is one method which helps understand what a distribution is, namely a state of knowledge (or a description of uncertainty about something).  This is the only definition that has made sense to me in all situations
Comment: Ben Bolker provides a good overview of this in the 'beastiary of distributions' section of [Ecological Models and Data in R](http://emdbolker.wikidot.com/)
Answer: I think the question is interpretable in two ways, which will give very different answers:
1) For people studying statistics, particularly at a relatively advanced level, what is the hardest concept to grasp?
2) Which statistical concept is misunderstood by the most people?
For 1) I don't know the answer at all.  Something from measure theory, maybe?  Some type of integration?  I don't know.
For 2) p-value, hands down.
Comment: Measure theory is neither a field of statistics nor hard. Some types of integration are hard, but, once again, that isn't statistics.
Answer: Confidence interval in non-Bayesian tradition is a difficult one.
Answer: I think people miss the boat on pretty much everything the first time around.  I think what most students don't understand is that they're usually estimating parameters based on samples.  They don't know the difference between a sample statistic and a population parameter.  If you beat these ideas into their head, the other stuff should follow a little bit easier.  I'm sure most students don't understand the crux of the CLT either.


Solarflare ef vi how to sniff TCP packet to a specific port?

Question: {
    in_addr addr;
    EFVI_CHECK(::inet_aton("22.214.171.124", &addr));

    ef_filter_spec filter_spec;
    ef_filter_spec_init(&filter_spec, EF_FILTER_FLAG_NONE);

    ef_filter_spec_set_ip4_local(&filter_spec, IPPROTO_TCP, addr.s_addr, htons(21723));
    ef_vi_filter_add(&res_.vi, res_.dh, &filter_spec, nullptr);
}

{
    // problem is here
    ef_filter_spec filter_spec;
    ef_filter_spec_init(&filter_spec, EF_FILTER_FLAG_NONE);

    ef_filter_spec_set_port_sniff(&filter_spec, 0);
    ef_vi_filter_add(&res_.vi, res_.dh, &filter_spec, nullptr);
}

I'm trying to sniff packets sent to a specific netword card at a specific port using Solarflare ef_vi API.
However, when I use ef_filter_spec_set_port_sniff(&filter_spec, 0), it actually sniff all packets received at that netword card. Basically the previous filters are ignored.
When I use ef_filter_spec_set_ip4_local alone, it intercepts the packet instead of sniffing (i.e the packet isn't sent to the kernel).
How can I do this with ef_vi?
Answer: what you can do is to interpret all the packets and process only packets matched your port. As far as I know, there is no way to sniff to a specific port using ef_vi APIs.


Componente ou Conversor PDF

Question: Alguém conhece algum componente para Delphi XE que converte PDF para DOC ou HTML ? Ou se possível editar o PDF direto e alterar o mesmo ?
Answer: A FoxIt disponibiliza um SDK para edição e visualização de PDF como pode ser visto neste link
Este SDK permite, entre outras coisas, suas solicitações

Edição de Documentos PDF
Rendenização para web de PDFs

Componente ou Conversor PDF


SSH Key Fingerprint Changes

Question: So I copied an SSH private key from my Linux machine to my M1 Mac. (I've attempted to do this both by uploading the cyphertext to a secrets manager as well as directly transfering the file) but when attempting to SSH to the same machine from each I can only succeed from the Linux machine.
Linux machine:
<code>$ ssh -V
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
$ md5sum id_rsa
7ed8b0c73e168ef48acc715d4c6bcf73  id_rsa
$ sha256sum id_rsa
969d0a35b36c50035881bdd2d8367b90dbff548131d5c72989d29e83b7a3aa19  id_rsa
$ ssh-keygen -l -f id_rsa
4096 SHA256:d19Ee+tlzVHitLzKuuZbdgYcnhla96ypXNtpBwrqfqE email@example.com (RSA)
$ ssh -i id_rsa firstname.lastname@example.com
[redacted MOTD; WORKS]
</code>
M1 Mac:
<code>$ ssh -V
openSSH_8.6p1, LibreSSL 2.8.3
$ md5 id_rsa
MD5 (id_rsa) = 7ed8b0c73e168ef48acc715d4c6bcf73
$ shasum -a 256 id_rsa
969d0a35b36c50035881bdd2d8367b90dbff548131d5c72989d29e83b7a3aa19  id_rsa
$ ssh-keygen -l -f id_rsa
4096 SHA256:rWsP9jN2otlchh8icr/jlnZxDjEfEkngDJOoPh4RYZI email@example.com (RSA)
$ ssh -i id_rsa email@example.com
firstname.lastname@example.com: Permission denied (publickey).
</code>
AFAICT these private keys are identical. However for some reason ssh-keygen does not fingerprint them thesame. Even though they have identical checksums.
Could this be an issue with the M1 SoC? Or is there something much more obvious happening here?
Comment: Do the corresponding `id_rsa.pub` files match? Yes, I'm specifically asking about the `.pub` files.
Comment: @user1686 that was it!  Apparently the -l flag on ssh-keygen causes it to fingerprint the public key even if the path is to the private.  And there is where the mismatch was.
Answer: The issue here was a misunderstanding with how ssh-keygen reports on figerprints. Even if you provide it with the path to a private key, with the -l flag it will use the corresponding public key file. And that's where I still had a mismatch. Upon fixing the public key everything worked as expected.

SSH Key Fingerprint Changes


Biholomorphic mapping between two domains

Question: I want to construct a biholomorphic mapping between
\begin{align}
G:= H \setminus \{iy : y \in (0,1]\},
\end{align}
where $H:= \{z \in \mathbb{C}: \operatorname{Im}(z) > 0\}$, and
\begin{align}
\mathbb{E}:= \{z \in \mathbb{C}: |z| < 1\}
\end{align}.
I know about the cayley map that transforms $H$ to $\mathbb{E}$ via $z \to \frac{z-i}{z+i}$ but I struggle to handle $G$.
This question is part of an exercise. Before this question I have shown that $z \to z^2$ maps $H$ to $\mathbb{C}\setminus [0, \infty)$. Does that somehow help me?
Answer: You are right that $z \to z^2$ maps $H$ to $\mathbb{C}\setminus [0, \infty)$, and the same function maps $G$ to $\mathbb{C}\setminus [-1, \infty)$.
Then $z \to z^2+1$ maps $G$ to $\mathbb{C}\setminus [0, \infty)$, and a suitable branch of the square root maps that to the upper half-plane.
Now use the Cayley map to map that to the unit disk.


Comparing two methods that predict experimental data

Question: I am comparing two methods that predict which residues in two proteins are in contact. I am a bit confused as to how I can compare these methods and draw some conclusion based on some statistic. Currently, I have been using a students t-test on the correlation between the methods and the experimental data. But from that I get that both are significant, i.e. that the p-value is << 0.05. I then conducted another t-test where I compare the means of the two methods, and that one gives me a p-value that confirms the plots that I made.
I have a feeling that I am missing some critical part when I am comparing these methods. In my mind it does not make any sense to compare the means of the methods and the experimental data since that would be like taking the mean of 50% like apples, 20% like bananas and 90% like pears.
<code>>>> import numpy as np
>>> from scipy import stats
>>> from matplotlib import pyplot as plt
>>> methodA = [0.0, 0.0, 0.0001, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0007, 0.0, 0.0, 0.0075, 0.0034, 0.1645, 0.1611, 0.0464, 0.0514, 0.0273, 0.0049, 0.0264, 0.0112, 0.001, 0.0143, 0.0299, 0.0003, 0.005, 0.0058, 0.0076, 0.1232, 0.0069, 0.001, 0.0, 0.0, 0.0, 0.0001, 0.0, 0.0002, 0.0051, 0.0564, 0.0165, 0.0916, 0.1397, 0.1468, 0.1002, 0.0031, 0.0, 0.0, 0.0, 0.001, 0.0008, 0.0, 0.0011, 0.0003, 0.0, 0.0147, 0.0034, 0.0731, 0.0777, 0.1516, 0.1282, 0.0349, 0.0695, 0.0134, 0.0102, 0.0001, 0.0001, 0.0002, 0.0045, 0.018, 0.022, 0.1027, 0.1663, 0.1726, 0.1473, 0.1402, 0.1605, 0.1603, 0.1322, 0.0976, 0.1379, 0.1289, 0.1477, 0.1413, 0.1634, 0.1672, 0.1453, 0.1487, 0.1584]
>>> methodB = [0.0005, 0.0024, 0.0229, 0.0099, 0.0078, 0.0159, 0.0208, 0.0277, 0.0291, 0.0473, 0.0076, 0.0, 0.0004, 0.0001, 0.0531, 0.1706, 0.1486, 0.1828, 0.1167, 0.0194, 0.1575, 0.0872, 0.0435, 0.0944, 0.1299, 0.0013, 0.0189, 0.0149, 0.0381, 0.0732, 0.0174, 0.0201, 0.0193, 0.0024, 0.0048, 0.0148, 0.0193, 0.0499, 0.0669, 0.072, 0.0582, 0.1345, 0.1353, 0.1321, 0.1846, 0.0071, 0.0204, 0.0, 0.004, 0.0145, 0.0165, 0.0025, 0.0079, 0.0085, 0.0017, 0.0784, 0.0004, 0.0934, 0.0673, 0.1815, 0.218, 0.0264, 0.0455, 0.0481, 0.0201, 0.0382, 0.0094, 0.0092, 0.0138, 0.0076, 0.0211, 0.018, 0.0099, 0.0508, 0.2212, 0.2321, 0.2052, 0.2411, 0.2098, 0.1637, 0.2424, 0.1796, 0.2234, 0.1549, 0.172, 0.2398, 0.1622, 0.14, 0.1928]
>>> ExperimentalData=[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.3381, 0.1166, 0.3265, 0.0233, 0.0, 0.0117, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0117, 0.0233, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0466, 0.0233, 0.0466, 0.3381, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.3148, 0.0117, 0.2215, 0.3381, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0466, 0.0233, 0.0117, 0.3381, 0.2565, 0.2682, 0.3381, 0.07, 0.0466, 0.0233, 0.035, 0.035, 0.035, 0.035, 0.0233]
</code>
<code>>>> plt.figure(figsize = [12.8, 5])
>>> xs = np.arange(len(methodA))
>>> plt.plot(xs, ExperimentalData, color = 'tab:gray', ls = '--', label = 'Experimental Data')
>>> plt.plot(xs, methodA, color = 'tab:blue', label = 'Method A')
>>> plt.plot(xs, methodB, color = 'tab:green', label = 'Method B')
>>> plt.legend()
>>> plt.show()
</code>

<code>>>> print(np.corrcoef(methodA, ExperimentalData)[0][1], np.corrcoef(methodB, ExperimentalData)[0][1])
0.46765048698864253 0.6367533471188255
</code>
<code>>>> print(stats.ttest_ind(methodA, ExperimentalData)[1], stats.ttest_ind(methodB, ExperimentalData)[1])
0.510637136331747 0.021951161091213506
</code>
From the plot I think it clearly looks as if method A is better than method B, and the t-test supports this but the correlation don't.
Answer:
I have a feeling that I am missing some critical part when I am comparing these methods.

Usually you want to evaluate some overall measure of the error or differences between observed and model-predicted values. Then the model with the lowest overall error would be the winner.
You should apply your knowledge of the subject matter to choose a measure that works best for your application. Is it the mean-squared error over all residues? The mean absolute error? An error beyond some level (below which you don't care about the error)? The number of residues estimated by the model to be in contact but not actually in contact? The number of observed contacts actually predicted by the method? Only you and your colleagues (guided by the relevant literature) can make that choice.*
Once you choose the appropriate measure of error, you can compare the errors between the two methods for this particular protein pair. A statistical test to distinguish the methods might be possible, depending on the measure you choose and the distributions of error values. But with only 1 protein pair you can't be sure that the "best" model here would be "best" in any other situation.

*The methods don't look that dissimilar overall to my non-expert eye. Method A seems to have fewer false-positive contact predictions where the observed values are 0. Method B seems to capture better the double-peak around residues 15-19 and the late peak near residues 75-80. Which is "better" will depend on your choice of the error measure.


Active Directory Username Formats

Question: I'm working on an open source project that will manage active directory users.  I am looking for feedback from Windows/Active Directory Admins on the formats of usernames they prefer or their organization uses.  I want to make sure the software allows admins to use the most popular formats when new users are created.  Here is the list I have so far:
<code> 1. <firstname><lastname>
 2. <lastname><firstname>
 3. <lastname><firstinitial>
 4. <lastname><firstinitial><middleinitial>
 5. <firstinitial><lastname>
 6. <firstinitial><middleinitial><lastname>
 7. <firstname><lastinitial>
</code>
In addition how do you handle multiple identical names? So if two John Smith's exist do you append a numeric number, or interject a middle initial or name to solve the problem?
Comment: What do you mean by manage?  I really don't understand what you are asking here. Can you elaborate?
Comment: Yea, its pretty simple, maybe its not clear.  When you create new users in active directory today what is the username format you use and why?  If the software I have allows you to create new users automatically through simple name input fields obviously you would want it to follow your naming convention.
Comment: In most of the environments I've managed, usernames were purposefully *not* related to the user's name to avoid issues with duplicate names, name changes, and all of the other headache involved with name based IDs.
Answer: We went a different direction with this question.  We ended up building dynamic elements that allow for any combination of the user's name to be used as a username.  However the formats above do appear to be the most used.
Brent

Active Directory Username Formats


How to read Exp(z|y) in statistics?

Question: I am calculating Customer Lifetime Value and have the following expression:
Customer Lifetime Value = Exp( visits | lifetime) * avg(transaction size)

In context of statistics, what does Exp(z | y) mean?
What is Exp in this context? (exponential, e^(z | y)..?)
What is | in this context? (z*y, z given y..?)
Thanks.
Comment: I don't know how about you, but I'm used to writing random variables with capitals, so it would be rather E(Z|Y). Small letters 'z' and 'y' usually mean observed value of random variable Z and Y, respectively.
Comment: CLV probably is a random variable, so on the right side of equation you should have a 'recipe' for random variable, and avg is a real number. That being said, Exp(visits|lifetime) could be a conditional expectation. See wikipedia for conditional expectation.
Comment: Thank you. So E(z | y) is more correct?
Comment: There are two different things: for a real integrale random variable $X$ and another random variable there is the conditional expectation $E[X|Y]$, see https://en.m.wikipedia.org/wiki/Conditional_expectation. This is yet another random variable. Then there is the factorization of that, denoted by $E[X|Y=y]$, that is, a function from the space Y maps to to the reals. This quantity is so important in ML because given features X and a real valued outcome Y, one can show that y->E[X|Y=y] is the best regression (with lowest RMSE). They mean E[Visits|Lifetime=lifetime].
Answer: Yes,  in this case Exp is meant to be $\mathit{E}$, the expectation of number of visits given the lifetime of customer.

How to read Exp(z|y) in statistics?


Nullable object must have a value

Question: Am getting this error when l tried to add data in my project.it point me to this code:
var currentuserid = WebSecurity.CurrentUserId;
var user = db.UserProfiles.SingleOrDefault(u => u.UserId == currentuserid);
List<Claim> claims = db.Claims.Include("Claim_Status").Include("Patient").Include("Hospital").ToList();
var userClaims = claims.Where(c => c.Hospitalid.Value == user.Hospitalid);
return View(userClaims.ToList());

how can l solve this problem.
am using Microsoft sql server 2008.
Comment: how can l remove the duplicate nullable object.
Answer: One or more of your HospitalIds are null. You can fix it by changing your Where clause to:
var userClaims = claims.Where(c => c.Hospitalid == user.Hospitalid);

which will filter out any claims with a null HospitalId.
Note that you're currently loading the entire Claims table and filtering the collection in memory. You probably want to execute the filter on the database, which you can do with:
List<Claim> claims = db.Claims.Include("Claim_Status").Include("Patient").Include("Hospital")
    .Where(c => c.HospitalId == user.Hospitalid)
    .ToList();

Comment: lee,thank u but l removed the `value` but the error still stays.
Comment: @UmarBachena - Which line is the exception on? It looks like `Hospitalid` is the only possible nullable value in your code.
Comment: it's on `Line 37:  var userClaims = claims.Where(c => c.Hospitalid == user.Hospitalid);`
Comment: @UmarBachena - Is it the same exception? Or is `user` null?
Comment: `Exception Details: System.InvalidOperationException: Nullable object must have a value.`
Comment: @UmarBachena - That code shouldn't throw an exception on a nullable value. Have you rebuilt everything?
Comment: @UmarBachena - Ok, then no match was found in your `UserProfiles` query, so you'll have to fix that first.
Comment: l rebuilt it but it still point me to the same error, how can l fix that in my `UserProfiles` query.
Comment: Lee, how can l fix this in my userprofiles?
Comment: @UmarBachena - Your query is probably ok, but the value returned by `WebSecurity.CurrentUserId` does not correspond to a record in your database, so I can't really help you with that.
Comment: l understand u.thank you Lee.

Nullable object must have a value


Can DB2 and SQL Server be installed on the same machine?

Question: I suspect there should be no problem as those are separate pieces of software, but ideally I would like to have some 'official' confirmation or denial as it will be production environment.
I'm targeting newest versions of both databases.
I was not able to find any information in official channels, but maybe I was looking in the wrong places.
Information about individual successes of failures would also be useful.
Comment: Given the products are from two different vendors, who would you expect to give an "official" blessing to the combination?
Comment: Well, but statement that setup with other vendor's software installed is not supported can realistically be expected. As I am still unable to find one, I'm assuming it will work. As stakes are high I want to be extra sure.
Comment: _Why_ are you doing this?  Obviously, you probably can't read the database files of one in the other.  If you're expecting to use this for a production environment, please be aware that the two systems will steal resources from each other when under heavy load.
Answer: Offcourse.
If you can't find anything against, that might be your answer.
Answer: The two products can co-exist peacefully because:

There is no overlap in the default installation path for the 2 products;
They don't require the same TCP port number for servicing requests - DB2 listens on port 50000 while SQL Server listens on port 1433.

Your real worries should be about server load during production.
Without knowing what your current usage is, the installation of another database server will at least impact resource contention and utilization (of CPU, memory, HDD etc).


Test whether two means are significantly different?

Question: I have an anonymized database of a fictional bank with approximately 30,000 entries.
These database contains for each customer whether he uses online banking (0 = he doesn't, 1 = he does) and the realised profit (in Euros) of the last year.
So I consider this database as a "sample", since the popularity of the bank's customers is higher. Now I calculated the mean of profits of all customers using online banking and the mean of profits of all customers NOT using online banking.
Now I'd like to test whether these two means differ significantly. Which is the right test to go with? I thought of the t-test, but I'm not sure, that these two means can be considered as based on two samples?
Comment: Do you mean that the dataset has 30,000 *rows* ? What are the summary statistics of the two samples (mean, median, standard deviation) ?
Comment: @XDAF Did my post answer your question?
Answer: The independent t test is for two $groups$. This means you can have one sample that you can divide into groups based on some characteristics like male vs. female or as in your case online banking yes vs. no. If you want to compare the means you maybe rather want to use the Welch t test and not the Student's t test (see here). Also keep in mind that with a big sample size the power of a t test increases, hence, it is easier to get a significant result. So you should also have a look whether the statistically significant difference in means is of any interest to you, because with a big sample size even little not relevant differences can be significant. See this question for some discussion, for example.
Comment: @XDAF The Welch test make an adjustment to account for the fact that the two groups may not have the same variance. If you do t.test in the R software package, it does the Welch test by default.


Using RoboCopy on multiple destinations with monitor option

Question: I am trying to copy a shared folder which its content (folders and files) are created by a third party app at no specified intervals.
I need to copy these files onto two different server shares.
The way I have set it up is to run two different DOS Batch files on a Server start up.
Today I had a problem where both stopped working. I wanted to know if there is a more efficient way to copy these files WITH the monitor option on two different shared folders.
I have the following options set: /mon:1 /mot:15 /r:2000 /z /LOG+:\\MyPC\share\BACKUPLOG.txt /TEE
In the batch file I had the following line at the begin gin so it could work otherwise it just displayed a loop of the command and never really ran: cd %windir%\system32\
Comment: Why not use a Distributed File Share arrangement? That way copying of data from one folder to the others would happen automatically.  See https://en.wikipedia.org/wiki/Distributed_File_System_(Microsoft)#DFS_replication for more details.
Comment: We haven't setup DFS because on the server that is creating the files we don't have much control over and is on another domain (cloud). We are using Robocopy to copy from this server to one of our 2008 R2 VM server and on a PC.
Answer: I would use George's suggestion and do DFS-R.  It's far more efficient for this kinda stuff..  But if like me, you insist on using Robocopy, you might want to just add a single pass to the task scheduler and run it at a normal interval.  (or at login which is available in the task scheduler as well).  This way you aren't creating a console window, that if interupted, will totally hose up your copy process.  In adding it to the scheduler, you can have it simply copy over the new updates.  And the machine/VM doesn't have to be logged in to run if you save the credentials to the task (advanced properties of the task)
The issues I have run into with the /MON option is that often the app writing the data doesn't finish in time before robocopy gets greedy and begins to copy it.  Of course then it will have to copy it again once the writing is completed.  Running it once and then exiting "mostly" solved the problem for me.  The contention can still happen during the run, but at least you aren't creating a race condition.
Set the task scheduler to say 5 mins and have it repeat for every 5 minutes for 24 hours, every day.  But of course this adds that extra load on the source server of having to scan the folder during each pass to see what has changed.  YMMV


how to make draggable transparent inside a Div

Question: I want to make my draggable transparent this code does not work
<code>$("#draggable").draggable({draggableClass:'transparent'});
</code>
Answer: <code>$( "#draggable" ).draggable({ opacity: 0});
</code>
Comment: it does not work I want the draggable background to be transparent this is my site http://amp.site88.net/
Comment: how can I get rid of the Widget it takes a lot of space I want to keep the draggable part
Comment: I really don't know... I just had a quick read in the docs... the default example is using a plain div... so that should get you started


Is there any way to use runasoriginaluser in uninstallrun in inno setup?

Question: I have written an application using docker-toolbox and inno setup script for the application installation in windows 10.
And I want to remove the docker-toolbox VM, while I uninstall my program. However, the VM cannot remove completely by following inno setup script.
[Setup]
PrivilegesRequired=none

[UninstallRun]
Filename: "{cmd}"; Parameters: "/C ""docker-machine rm -y myDocker"""

The command "docker-machine rm -y myDocker" always work whenever runs in my user cmd, but not works in inno setup uninstallrun.
And I checked and found out that docker-toolbox is based on VirtualBox. VirtualBox uses a per-user environment. Becoming root (or any other user) does not give you access or more powers to the original user's VMs. ALWAYS perform VM operations as the user that actually created the VMs.  Hence, I have to run a command as a original user in uninstallrun, but I cannot find a way to do so.
Looking forward for a help and support, I have spent lots of times in this problem.
Is there any way to use runasoriginaluser in uninstallrun in inno setup?
Answer: You should not modify a specific user profile from an (un)installer that runs with Administrator privileges (installs software for all users).
See Installing application for currently logged in user from Inno Setup installer running as Administrator.

runasoriginaluser flag is not supported in UninstallRun section. Probably because it won't be of any use there anyway. What the flag does in Run section is that it executes program with privileges with which the installer was originally executed. But uninstaller (for installer elevated to Administrator privileges) is executed with Administrator privileges straight away, when executed from Control Panel/Settings app.