I'm a returning piano player, after 30 years

Question: I am 74 years old, and learned piano as a child. I got quite far, my crowning glory was being able to play Rachmaninoff's Prelude in C# Minor. I practiced for about a year and managed to get through it with no mistakes, once, when I was about 17. But I have not played at all since I was about 30ish.
My musical (very) children have told me that the way to get back is to practise scales, and when I've finished that, practise more scales, and arpeggios. So I have been doing that, an hour every day (I wish I could do more, but I still have a full-time job, etc) for about 6 weeks now.
But I very sneakily found some sheet music on line for "intermediate" players. I am trying to to play an "easy" version of Jesu Joy of Man's Desiring. There are a few places that are tricky and every practise session, I try to work on those tricky parts individually over and over again to get them right. It's not the way I was taught as a child, but those same musical children have told me that it's the way.
It seems to be going well, but is slow going. I don't know if I will ever be as proficient as I even was at 17,but I'm hoping I will one day be able to get Jesu right and move on to other pieces.
Am I right to be hopeful?
Comment: @Michael: I do have arthritis in some of my fingers, but I just play slow enough not to let it bother too much.  I don't think I'll ever make Sinding's Rustle of Spring.  I'll try it at Slow Walk to Summer instead. :)
Comment: There is a lot of stuff that I've discovered I still remember. Like the fingering for the new piece I'm trying. I notice that I'm reading a bar or so ahead to see what's coming so that I can get the fingering right. And sometimes disagreeing completely with the suggested fingering. I seem to remember some "rule" about not using the same finger even when a note is repeated. I just automatically use another finger. It's all quite exciting, really.
Comment: There's a lot of interesting material here, but it led to a different answer than I expected. I thought it was setting up "am I going about it the right way?" (and note, the answer you've gotten already focuses on that). The question "can I hope for progress" (or, implicitly, "is it worth my time" or "am I progressing 'fast enough'") seems to be as common among adult learners as it is rare among children, and the proposed duplicate is in fact a collection of 40 questions and answers along those lines. Are you sure you don't want to edit to focus on the "how"?
Comment: (And if so, maybe add a bit more detail. My hot takes include the fact that it would be as wrong to practice *only* scales/arpeggios as an extended plan as it would be to omit them; you don't have to be "sneaky" to add actual repertoire. And that [finding your zone of proximal development](https://en.wikipedia.org/wiki/Zone_of_proximal_development) is important when re-learning as well as when learning. If *Jesu* is too challenging for now, pick something easier. If you find something too easy, pick something harder. And consider getting a teacher to pick for you?)
Comment: Also, please read about [how to avoid broad, discussion-based questions](https://music.stackexchange.com/help/dont-ask). There should be a central, answerable question; right now it seems to be "will I progress."
Comment: It's interesting that nobody has mentioned anything about (overuse) injuries (tendonitis etc.).
Comment: Sounds like OP played for some 15 years before stopping playing in his 30s? I once heard a piano teacher say that if you played piano for that long, then you can remember many pieces you played without having to refreshing them from time to time, but it seems OP now even has problem with scales, so my curirosity is, that teacher was wrong? No matter how long you played in the past, a long (years of) pause will take away most of what you learned?
Answer: If this is what you feel like doing, then totally go for it. Whether you get quite as good as you were in your youth again is a bit of an open question but that isn't what matters, what matters is that you enjoy yourself.
But speaking of enjoying yourself, I'm a little skeptical about your children's advice. I'm not disputing it's good for improving your technique, but is it also good at keeping you playing in the first place? If you find yourself getting bored or frustrated, then I suggest you set the exercises aside for a moment and play some actual music. Any kind of music you like. Doesn't even need to be challenging, but very much needs to be fun. Though, the pieces that'll best help you grow are those that will provide some challenge for you to overcome, just not so much they'd be overwhelming.
Wishing you the best with your rediscovered hobby.
Comment: @phoog: I formulated A and B as opposites in terms of priority. If a choice needs to be made between doing something that's not as fun but is a good learning tool, or something more fun that might not net long term skill acquisition; which will OP pick? I didn't mean to imply that a person should _only_ do one or the other, but rather I'm trying to highlight that while a person tends to do both, some people prioritize them differently, and that's what I think has happened between OP and their children.
Comment: @phoog: Different strokes for different folks. I'm the kind of person who needs to be faced with a problem before he can listen to the solution. For me, when learning guitar, I needed to first try and play the songs I liked, only to really struggle. This made me more receptive of "boring" skill exercises. If you'd started me on those exercises before I got to try and play a song that I liked, I probably would've given up then and there.
Comment: As to the second paragraph; the goal matters. Is the goal (A) to acquire as much skill as possible in a given timeframe (or reach a certain skill level in the least amount of time), or is the intention to (B) enjoy oneself on the road to (re)learning how to play the piano? Given the "very" musical nature of OP's children, I suspect they're answering the question from the perspective of achieving their (presumed) high skill level (A), but OP clearly is more interested in (B). None of the people in this story are wrong, they're just not realizing that they're answering subtly different questions
Comment: @Flater isn't (A) part of (B)?  Or at least potentially part of B?  As an amateur musician (e.g. in secondary school and university) I always found the most challenging pieces to be the most rewarding.  Isn't mastering this particular arrangement of _Jesu, Joy of Man's Desiring_ going to be more enjoyable than stumbling over it, and all the more so for the sense of accomplishment it will bring?  That's certainly how it works for me.
Comment: Thank you for all the comments, and encouragement.  I guess my children didn't rule out trying actual compositions by someone (like Bach), but I followed their insistence on scales for as long as I could (still do).  I found it quite fascinating how the brain works.  My first attempt at playing the C Major scale with both hands was very revealing.  I thought I could "just do it", but then, of course, it's not like that, left and right being mirror images, and having to cover an octave with only 5 fingers. But then after a while my muscle memory clicked in and I was away.  Same with Jesu.
Comment: @Flater all good points.  I'm just leaning toward the technical-practice-as-its-own-reward view because of Peter's age and history of having played fairly proficiently at a younger age.  For a beginner I would definitely tend in the other direction.  And of course everyone needs to find their own balance, including having a couple of pieces that aren't demanding (or have been well learned) to be able to pull out when it's time to stop struggling with difficult pieces or technical exercises and instead just enjoy playing some music.
Answer:
"...managed to get through it with no mistakes, once..."

The well-known 'keep taking a run at it, one day you'll get lucky' technique!
We've all done it.  But your children are correct.  Don't keep playing the tricky bits wrong in the hope they'll magically turn right.   Play that passage super-slowly, but right.  If you still fumble, it's a fair bet your hand's got into a position where 'you can't get there from here!'. Sort out the fingering so that doesn't happen, then play it 10 times over but SLOWLY. Sounds like your kids will take you from there!
Answer: Kids are like sponges - they soak up stuff far more easily than us old'uns! But, working on scales/arpeggios - mainly as warm-ups for those gnarled old hands - is a good start.
I've found that as an old'un, things take a lot longer to sink in, and my answer is to practise for very short periods, but many of them. As Laurence says, slow practice is the answer. Remember - amateurs practise till they get it right; professionals practise till they can't get it wrong. I appreciate you probably don't want to pursue a professional route at your time of life, but the premise is certainly a good one to embrace! Good luck, and keep going. It will do your kids a world of good to 'teach' you things. It's a known fact that when one teaches something, one gets to know it better as well...
Comment: I like the "professionals practise until they can't get it wrong"  I will try to see it that way.   In the beginning, I had real trouble playing the B Minor scale, but I avoided frustration with myself by saying, "Pretend this is a concerto that you're playing.  If you can't get  the B Minor scale right, how are you going to progress to Bach's Toccata and Fugue?"
Comment: @Peter another approach that is opposed to "until they get it right" but is more accessible and realistic for an amateur is "practice until you get it right three times in a row."  If you stop the first time you get it right then you've just done it 10 times wrong (or however many) and only once right; your muscles won't remember the last time too well.
Comment: @phoog yes, nice advice, thank you!
Answer: There are two important factors here:

Your goal/interests
How you practice

Goals
If you are motivated by the idea of developing a very good technique, you will need to be more technical in your practice. You must be aware this is an investment that does not pay off right away. You have decided to learn more slowly in order to gain a better general skillset that applies broadly to many types of music. Scales are the simplest technical exercise, and there are lots of others.
If you are motivated by actually playing music, you may do a little technical exercise to start your sessions, but you'll generally be trying to learn a specific piece, and you'll be focusing there.
How You Practice
This is the overlooked aspect so often left out. Even after you know your goals, you must practice well to actually ensure progress with any goal.
What does that mean?

Deliberate (you must have some objective with a phrase or portion of the music, e.g. play it smoothly, play it uninterrupted, practice a particular fingering, etc.)
Consistent (you must repeat the practice in the same way each time, e.g. play a phrase with the same fingering so you are reinforcing something consistent and not giving random info to your brain)
Correctness, at some speed (after having a goal, you must ensure you can achieve correctness of your objective, no matter how slow you must play to make it possible).
[Others...]

Correctness

Played with the right fingering, or at least a workable fingering
Played at a continuous rate without pauses
Played where each note has the right relative duration (even if tempo is slow)
Played with reasonable dynamics (reasonably consistent volume)
If more advanced, played with a good phrasing

Possible objectives

Play a section correctly with the left hand
Play a section correctly with the right hand
Play a section correctly with both hands
Practice a specific fingering for a small section (ensure the fingering is possible to transition to from the previous section, and leaves you ready to play the next)

Conclusion
The good news is, if you practice anything well, you will make progress with it. The better news is that you don't have to be an expert to make this work for you. Find something you can feel reasonably confident to try that fits your knowledge, but do it well.
While you don't have to practice this way the whole time at the piano, if you want to see noticeable results, you will need to do this some of the time.


What is the equation used in R's ccf and Julia's crosscor?

Question: I've been scratching my head a lot about this. Now, except for the fact that Julia gives out a seemingly backwards result with respect to the lags
julia> crosscor([1,2,3,4], [1,2,3,1])'
1x7 Array{Float64,2}:
 -0.30339  0.0  0.64049  0.13484  -0.37081  -0.40452  0.30339

> print(ccf(c(1,2,3,4), c(1,2,3,1), type="correlation", plot=F))

Autocorrelations of series 'X', by lag

    -3     -2     -1      0      1      2      3
 0.303 -0.405 -0.371  0.135  0.640  0.000 -0.303

which is easily solved by inverting the x and y ("easily" is relative when it takes a hour of your life to figure out), the numbers don't resemble any of the equations I'm familiar with (which aren't many, anyway).
So, I opened up my stats textbook and found a host of other formulas which I am not going to test with pen and paper. I suspected for a moment that removing the means from the vectors played a role, so I tried not doing that (sadly, this can only be done in Julia)
julia> crosscor([1,2,3,4], [1,2,3,1], demean=false)'
1x7 Array{Float64,2}:
 0.188562  0.518545  0.942809  0.848528  0.518545  0.235702  0.0471405

but this still does not resemble the normalised cross-correlation that I computed with pen and paper (pending mistakes which I may have made).
In short: I need to cite the formula, so what is it?
Comment: The R documentation `help("ccf")` contains this under "References":
"Venables, W. N. and Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth Edition. Springer-Verlag.
(This contains the exact definitions used.)"
Comment: Sorry, but I don't have a copy of MASS handy. It's always better to look up stuff like that yourself anyway.
Comment: I might need to check my maths because it doesn't, really… but then again I suppose this is why we have computers :)
Comment: Ah, yes. I checked my math again and I screwed up a couple of multiplications. You'd expect that ~25 years after first learning about it, one would have mastered multiplying single digit numbers :)
Comment: Yes, I thought I'd save myself a trip to the library :)
Comment: True that. Ah, well. My local library does not have a copy of it. Anyway, thanks for your input.
Comment: I bet your pen & paper cross-correlation, scaled as shown in Julia's [Statsbase.jl source](https://github.com/JuliaStats/StatsBase.jl/blob/master/src/signalcorr.jl#L254) (see [`sc`](https://github.com/JuliaStats/StatsBase.jl/blob/master/src/signalcorr.jl#L262)) would match.
Comment: I guess I expected you would calculate the equivalent of `x=[1,2,3,4];y=[1,2,3,1];crosscor(x,y,demean=false)*sqrt(dot(x,x)*dot(y,y))` which is `[4.0,11.0,20.0,18.0,11.0,5.0,1.0]`.
Answer: I assume that citing this excerpt here is ok by StackOverflow's terms of use?  From page 390 (section 14.1) of Venables and Ripley (2002) you can find the definition that they use here for the acf() function:

If you look at the source code of the ccf() function in R (type "ccf" at the prompt) you can see how acf() is being used and can interrogate how the above acf() implementation relates to the calculation.
Comment: If you're referring to `ij` they are the referring to the items of the series you're calculating correlations for at a particular lag.  These are the components that you'll see referred to in the source code for `acf()`.  I'd copy more of the explanation into the answer but don't want to infringe.  If there are specific details I can try to explain further?
Comment: That helps, but there are a few unclear things (like those indexes, what are they?). I suppose I'll have to grab a copy of the book.
Comment: Thanks, that helped. I also tracked down a copy of MASS-PL e they have the same equation. I'll try to figure it out and come back here (or on Cross Validated) and ask.


How to get change event in input field for a directive in angular 4

Question: I am a bit new to angular 4. I am trying to get change event from an input field inside a directive. Currently i am working with @HostListener
<code>@HostListener('keyup', ['$event'])
  inputChanged(event) {}
</code>
This is working correctly but this event is fired after some delay from releasing key and user can enter wrong input and is able to see that too. In my implementation i have removed the invalid input but it doesn't give a good exposure to user. Only thing i want is to get change event right the moment change happen in input field (character / string enter or remove both ).
Current HTML looks like this
<code><input type='text' class="form-control" placeHolder='hh:mm:ss' time-input [(ngModel)]="params.time" name="time"/>
</code>
PS. time-input is the name of directive and in directive i am trying to get the event change and i don't want to move any implementation to controller or component.
Comment: do you want to get native DOM event or event from `ngModel` directive?
Comment: i don't have any restriction for using any event type.i was using native DOM event at first but there was a problem in that but @gunter's answer worked and it is according requirements now
Comment: you need `input` event for the DOM native element. Gunter's solution will only work if there's `ngModel` directive on the `input`. So your `time-input` will only work alongside `ngModel`, it can't be used as a standalone directive
Comment: ok got it now i have two way i can cake ngModel necessory for that directive second update event (ngModelChange) to input event. Thanks for guidance. i will do it.:)
Answer: Use
<code>@HostListener('ngModelChange', ['$event'])
</code>
Comment: What if the change in value was not triggered by the user? Like reactive forms when the app dynamically sets the value of a forms
Comment: Thanks. Worked like a charm
Comment: Glad to hear :)
Comment: what If I'm not using ngModel?
Comment: @anaval you can also listen to `'input'` or `'change'` events fired by the input element directly. See also the end ofhttps://developer.mozilla.org/en-US/docs/Web/Events/input
Comment: Then there is no event. You can call your even handlers yourself from the code that changes the values.


POE-Component-IRC crashes under Windows 7 64-bit

Question: I'm trying to run the Perl module POE-Component-IRC using Perl v5.12.13 with ActiveState under Windows 7 64-bit and each time it is run it crashes Windows. I'm pretty sure it's something to do with this module since the only lines I have in the script is shown below.
use strict;
use warnings;
use POE qw(Component::IRC);

When I run this file Windows gets the gets the generic "This program has stopped responding" error message.
According to http://code.activestate.com/ppm/POE-Component-IRC/ it appears that it is compatible with my Perl version and version of Windows.
Any thoughts or ideas would be appreciated. I'd contact the developer on SourceForge but every time I report a but I get an error.
Comment: Don't report bugs on SourceForge, report bugs in the place that the distribution metadata says bugs are *supposed* to go to — in this case, https://rt.cpan.org/Public/Dist/Display.html?Name=POE-Component-IRC or firstname.lastname@example.com
Comment: In addition to hobbs' comment, you should always check search.cpan.org and/or metacpan.org for the proper place to report bugs.  While there may be a bug tracker at SourceForge, it's not necessarily canonical.
Answer: Check that you have latest version - 6.78.
Did you install it with ppm? In that case you should test it - run cpan -t POE::Component::IRC.
Correct place for reporting bugs is bug tracker


Help with Taylor polynomial

Question: I need to find the Taylor polynomial of order $2n$ of the function
$$f(x,y)=\frac{1}{1+xy}$$ on  $(x_0,y_0)=(0,0)$.
Can anyone give me a hand please?
Sorry, i make a mistake, their should say "polynomial" instead of "series, i have corrected now.
Answer: Hint, for $|u|<1$,
$$\frac{1}{1-u}=1+u+u^2+u^3+\ldots.$$
Comment: thanks for the answer, but i correct now, i mean Taylor polynomial, or the order $2n$ doesn´t make any sense.
Comment: @Dimitri so only keep a fraction of the terms...
Answer: A related technique. Here is the general formula for the Taylor series of a function in two variables around the point $(x,y)=(0,0)$
$$ f(x,y)=\sum_{k=0}^{\infty}\sum_{m=0}^{\infty}\frac{x^m y^k}{m! k!}\frac{\partial^{m+k}}{\partial x^m \partial y^k }f(x,y)|_{x=0,y=0} . $$
So, to get the order $2n$, you can consider the truncated series
$$ f(x,y)=\sum_{k=0}^{n}\sum_{m=0}^{n}\frac{x^m y^k}{m! k!}\frac{\partial^{m+k}}{\partial x^m \partial y^k }f(x,y)|_{x=0,y=0}. $$
You may want to show that
$$ \frac{\partial^{m+k}}{\partial x^m \partial y^k }\frac{1}{1-xy}\Big|_{x=0,y=0}=0\,\quad \mathrm{ when }\quad  m\neq n. $$
Otherwise, use Jonathan's technique to get the required series.

Help with Taylor polynomial


Welcome message after creating a mailbox user in zimbra

Question: We have deployed a zimbra mail server of version 5.0.x. I'm trying to setting up a automatic sending welcome message when email box is created. Is there any feature or that can be done.
Can anyone please help to do it..
Thanks in advance.
Comment: Are you creating the user through the interface, or using the back-end tools,or SOAP?
Comment: Using Admin User Interface and also I know that we can do it using back-end tools like zmmailbox, But that's not comfortable to my client so looking for good feature that can be added to it.
Answer: According to the relevant bug report, this is doable with a script. But they don't say how. I'd recommend asking there


Entity Framework: Model First, Inheritance?

Question: I have a Person table, an Employee table, and a Contractor table. All Employees are people, all Contractors are people and every Person is either an employee or a Contractor. Like so:
How would I be able to accomplish this concept using Model First? Inheritance?
Comment: Hey that picture looks famililar! :)
Comment: :P You know it! And I stole your sentence... couldn't have explained it any better.
Comment: Its all good.  I hope you get some answers.  Good luck!
Answer: You've got three options:
1 - Table-Per Hierarchy:
Good for performance, as one physical table is required. You'll need to add a discriminator field to Person - such as "PersonType". Problem with this approach (what I have found), is you end up will lots of nullable fields, and navigational properties between derived types are difficult (in my experience).
2 - Table-Per Type:
Requires separate tables, but good for flexibility if you want to another another "Person" type.
3 - Table-Per Concrete Type: Don't have experience with this, so can't really comment on it. AFAIK it's very similar to TPT.
I'd probably go with TPT, just because it's easier IMO.
Having said that though, the field in the "Contractor" and "Employee" tables are of the same type, so you could generalize this as a single field with TPH. But i'm guessing that's not the complete model.
Steps for Model-First:

Add those three entities to a blank EDMX.
Mark "Person" as abstract
Set "Contractor" and "Employee" as deriving from "Person". (Add -> Inheritance)
Remove the ID fields from the "Employee" and "Contractor" entities (not required - it will inherit the ID from "Person").
Generate database from model.
Answer: That table structure + inheritance == TPT. In the designer it will look something like this:

...and the raw EDMX for the entities and mappings to those tables:
<code><?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="2.0" xmlns:edmx="http://schemas.microsoft.com/ado/2008/10/edmx" xmlns:store="http://schemas.microsoft.com/ado/2007/12/edm/EntityStoreSchemaGenerator" xmlns:ssdl="http://schemas.microsoft.com/ado/2009/02/edm/ssdl" xmlns:edm="http://schemas.microsoft.com/ado/2008/09/edm" xmlns:annotation="http://schemas.microsoft.com/ado/2009/02/edm/annotation" xmlns:map="http://schemas.microsoft.com/ado/2008/09/mapping/cs" xmlns:codegen="http://schemas.microsoft.com/ado/2006/04/codegeneration" xmlns:huagati="http://www.huagati.com/edmxtools/annotations">
  <!--Updated by Huagati EDMX Tools version 2.16.4007.30259 on 2010-12-23 09:13:27-->
  <!-- EF Runtime content -->
  <edmx:Runtime>
    <!-- SSDL content -->
    <edmx:StorageModels>
      <Schema xmlns="http://schemas.microsoft.com/ado/2009/02/edm/ssdl" Namespace="Model1.Store" Alias="Self" Provider="System.Data.SqlClient" ProviderManifestToken="2008">
        <EntityContainer Name="Model1TargetContainer">
          <EntitySet Name="Person" store:Type="Tables" Schema="dbo" Table="Person" store:Name="Person" EntityType="Model1.Store.Person" />
          <EntitySet Name="Employee" store:Type="Tables" Schema="dbo" Table="Employee" store:Name="Employee" EntityType="Model1.Store.Employee" />
          <AssociationSet Name="FK_Employee_Person" Association="Model1.Store.FK_Employee_Person">
            <End Role="Person" EntitySet="Person" />
            <End Role="Employee" EntitySet="Employee" />
          </AssociationSet>
          <EntitySet Name="Contractor" store:Type="Tables" Schema="dbo" Table="Contractor" store:Name="Contractor" EntityType="Model1.Store.Contractor" />
          <AssociationSet Name="FK_Contractor_Person" Association="Model1.Store.FK_Contractor_Person">
            <End Role="Person" EntitySet="Person" />
            <End Role="Contractor" EntitySet="Contractor" />
          </AssociationSet>
        </EntityContainer>
        <EntityType Name="Person">
          <Documentation />
          <Key>
            <PropertyRef Name="PersonId" />
          </Key>
          <Property Name="PersonId" Type="bigint" Nullable="false" StoreGeneratedPattern="Identity" />
          <Property Name="Name" Type="nvarchar" Nullable="true" MaxLength="50" />
        </EntityType>
        <EntityType Name="Employee">
          <Documentation />
          <Key>
            <PropertyRef Name="EmployeeId" />
          </Key>
          <Property Name="EmployeeId" Type="bigint" Nullable="false" />
          <Property Name="EmployeeNumber" Type="nvarchar" Nullable="true" MaxLength="50" />
        </EntityType>
        <Association Name="FK_Employee_Person">
          <End Multiplicity="1" Role="Person" Type="Model1.Store.Person" />
          <End Multiplicity="0..1" Role="Employee" Type="Model1.Store.Employee" />
          <ReferentialConstraint>
            <Principal Role="Person">
              <PropertyRef Name="PersonId" />
            </Principal>
            <Dependent Role="Employee">
              <PropertyRef Name="EmployeeId" />
            </Dependent>
          </ReferentialConstraint>
        </Association>
        <EntityType Name="Contractor">
          <Documentation />
          <Key>
            <PropertyRef Name="ContractorId" />
          </Key>
          <Property Name="ContractorId" Type="bigint" Nullable="false" />
          <Property Name="ContractorNumber" Type="nvarchar" Nullable="true" MaxLength="50" />
        </EntityType>
        <Association Name="FK_Contractor_Person">
          <End Multiplicity="1" Role="Person" Type="Model1.Store.Person" />
          <End Multiplicity="0..1" Role="Contractor" Type="Model1.Store.Contractor" />
          <ReferentialConstraint>
            <Principal Role="Person">
              <PropertyRef Name="PersonId" />
            </Principal>
            <Dependent Role="Contractor">
              <PropertyRef Name="ContractorId" />
            </Dependent>
          </ReferentialConstraint>
        </Association>
      </Schema>
    </edmx:StorageModels>
    <!-- CSDL content -->
    <edmx:ConceptualModels>
      <Schema xmlns="http://schemas.microsoft.com/ado/2008/09/edm" xmlns:cg="http://schemas.microsoft.com/ado/2006/04/codegeneration" xmlns:store="http://schemas.microsoft.com/ado/2007/12/edm/EntityStoreSchemaGenerator" Namespace="Model1" Alias="Self" xmlns:annotation="http://schemas.microsoft.com/ado/2009/02/edm/annotation">
        <EntityContainer Name="Model1Container" annotation:LazyLoadingEnabled="true">
          <EntitySet Name="Person" EntityType="Model1.Person" huagati:InheritanceStrategy="TPT" />
        </EntityContainer>
        <EntityType Name="Person">
          <Key>
            <PropertyRef Name="PersonId" />
          </Key>
          <Property Type="Int64" Name="PersonId" Nullable="false" annotation:StoreGeneratedPattern="Identity" />
          <Property Type="String" Name="Name" Unicode="true" MaxLength="50" Nullable="true" />
        </EntityType>
        <EntityType Name="Employee" BaseType="Model1.Person">
          <Property Type="String" Name="EmployeeNumber" Unicode="true" MaxLength="50" Nullable="true" />
        </EntityType>
        <EntityType Name="Contractor" BaseType="Model1.Person">
          <Property Type="String" Name="ContractorNumber" Nullable="true" Unicode="true" MaxLength="50" />
        </EntityType>
      </Schema>
    </edmx:ConceptualModels>
    <!-- C-S mapping content -->
    <edmx:Mappings>
      <Mapping xmlns="http://schemas.microsoft.com/ado/2008/09/mapping/cs" Space="C-S">
        <Alias Key="Model" Value="Model1" />
        <Alias Key="Target" Value="Model1.Store" />
        <EntityContainerMapping CdmEntityContainer="Model1Container" StorageEntityContainer="Model1TargetContainer">
          <EntitySetMapping Name="Person">
            <EntityTypeMapping TypeName="IsTypeOf(Model1.Person)">
              <MappingFragment StoreEntitySet="Person">
                <ScalarProperty Name="PersonId" ColumnName="PersonId" />
                <ScalarProperty Name="Name" ColumnName="Name" />
              </MappingFragment>
            </EntityTypeMapping>
            <EntityTypeMapping TypeName="IsTypeOf(Model1.Employee)">
              <MappingFragment StoreEntitySet="Employee">
                <ScalarProperty Name="PersonId" ColumnName="EmployeeId" />
                <ScalarProperty Name="EmployeeNumber" ColumnName="EmployeeNumber" />
              </MappingFragment>
            </EntityTypeMapping>
            <EntityTypeMapping TypeName="IsTypeOf(Model1.Contractor)">
              <MappingFragment StoreEntitySet="Contractor">
                <ScalarProperty Name="PersonId" ColumnName="ContractorId" />
                <ScalarProperty Name="ContractorNumber" ColumnName="ContractorNumber" />
              </MappingFragment>
            </EntityTypeMapping>
          </EntitySetMapping>
        </EntityContainerMapping>
      </Mapping>
    </edmx:Mappings>
  </edmx:Runtime>
  <!-- EF Designer content (DO NOT EDIT MANUALLY BELOW HERE) -->
  <edmx:Designer xmlns="http://schemas.microsoft.com/ado/2008/10/edmx">
    <edmx:Connection>
      <DesignerInfoPropertySet>
        <DesignerProperty Name="MetadataArtifactProcessing" Value="EmbedInOutputAssembly" />
      </DesignerInfoPropertySet>
    </edmx:Connection>
    <edmx:Options>
      <DesignerInfoPropertySet>
        <DesignerProperty Name="ValidateOnBuild" Value="true" />
        <DesignerProperty Name="EnablePluralization" Value="True" />
      </DesignerInfoPropertySet>
    </edmx:Options>
    <!-- Diagram content (shape and connector positions) -->
    <edmx:Diagrams>
      <Diagram Name="Model1">
        <EntityTypeShape EntityType="Model1.Person" Width="1.5" PointX="5.125" PointY="2.125" Height="1.4033821614583331" />
        <EntityTypeShape EntityType="Model1.Employee" Width="1.5" PointX="3.875" PointY="4" Height="1.2110807291666665" />
        <InheritanceConnector EntityType="Model1.Employee">
          <ConnectorPoint PointX="5.875" PointY="3.5283821614583331" />
          <ConnectorPoint PointX="5.875" PointY="3.76" />
          <ConnectorPoint PointX="4.625" PointY="3.76" />
          <ConnectorPoint PointX="4.625" PointY="4" />
        </InheritanceConnector>
        <EntityTypeShape EntityType="Model1.Contractor" Width="2" PointX="5.875" PointY="4" Height="1.2110807291666665" />
        <InheritanceConnector EntityType="Model1.Contractor" ManuallyRouted="false">
          <ConnectorPoint PointX="5.875" PointY="3.5283821614583331" />
          <ConnectorPoint PointX="5.875" PointY="3.76" />
          <ConnectorPoint PointX="6.875" PointY="3.76" />
          <ConnectorPoint PointX="6.875" PointY="4" />
        </InheritanceConnector>
      </Diagram>
    </edmx:Diagrams>
  </edmx:Designer>
</edmx:Edmx>
</code>
For a more detailed description of the different inheritance types and how they relate to physical db tables, see:
http://huagati.blogspot.com/2010/10/mixing-inheritance-strategies-in-entity.html
...and...
http://blogs.msdn.com/b/adonet/archive/2010/10/25/inheritance-mapping-a-walkthrough-guide-for-beginners.aspx


Non-linear functional on a Banach space

Question: Let $(X,\lVert \cdot \rVert)$ be a (real) Banach space and let $p$ be a (real) functional on $X$ satisfying the following conditions:
(i) $p(x) \geq 0$ for all $x \in X$
(ii) $p(\alpha x) = \alpha p(x)$ for all $x \in X$ and $\alpha > 0$
(iii) $p(x_1 + x_2) \leq p(x_1) + p(x_2)$ for all $x_1, x_2 \in X$
(iv) if $x_n \rightarrow x$ in $X$, then $p(x) \leq \lim p(x_n)$
Show that there is an $M > 0$ so that $p(x) \leq M \lVert x \rVert$ for all $x \in X$.
I think this is going to come down to showing that $p$ is a norm, that $X$ is complete with respect to the metric defined by that norm, and then applying the open mapping theorem to the identity map $(X,\lVert \cdot \rVert) \rightarrow (X,p)$, but I'm not sure how to show that $p$ gives a norm.  I can show that $p(0) = 0$ (consider the sequence $\frac{1}{n} x$ for some nonzero $x$) and the triangle inequality follows straight from the properties of $p$, but I'm not sure how to show that $p(-x) = p(x)$ or how to show that $p(x) = 0$ implies $x = 0$.
Note: this isn't a homework problem, it's an old qual problem that I'm using to study for an exam.
Comment: $p$ doesn't have to be a norm, since it could be identically zero.
Answer: Hint: the property iv) enables you to show that $p$ is continuous: just consider the inverse image $U$ of an open interval and suppose it is not open, take $x= lim_nx_n, x_n \in U$, and $x$ is not in $U$, deduce the contradiction.
Deduce that $p$ is bounded in a ball and use ii) to show the existence of M.
Comment: Thanks!  Do you mean to take $x_n \notin U$ and $x \in U$, however?  This would derive the contradiction in the assumption that the complement of $U$ is not closed.
Answer: If property (iii) means that for all converging sequences $x_n\to x$ the limit $p(x_n)$ exists and satisfies $p(x) \le \lim p(x_n)$ then the claim can be proved by a simple contradiction argument:
Assume the claim is not true. Then for each $n$ there is $x_n$ such that $p(x_n) > n^2\|x_n\|$. Since $p(0)=0$ by (ii)(iv) it follows $x_n\ne0$. Due to (ii) we can divide the inequality by $\|x_n\|$. Thus, wlog $\|x_n\|=1$.
Dividing the inequality by $n$ yields
$$
p(n^{-1} x_n) > n.
$$
Now $n^{-1}x_n\to0$, so that $\lim p(n^{-1}x_n)$, exists, which is a contradiction to the inequality, where we got that $p(n^{-1}x_n)$ is unbounded.
Note that you can replace (iii) by ``for all $x_n\to x$ the limit $p(x_n)$ exists''.

Non-linear functional on a Banach space


What are the main factors to predict which team is going to win a soccer match?

Question: I'm building a logistic regression model to predict which team is going to win a soccer match. I'm looking for ideas to broaden the number of predictor variables. I need before as well as after match variables.
The variables that came to my head:
<code>*ELO-ratings of the teams
*Total shots & total shot on target
*Market value of the teams
*Penalties
*Fouls
*Free kicks
*Total pass
*Pass percentage
*Possession
</code>
I would greatly appreciate for any help you can provide.
Answer: A fairly important variable is home advantage. Last season in the Premier League, there were 172 home wins and 115 away wins, which is a significant difference. Though this will vary from league to league, it is still commonplace, and if you do the calculations on other leagues you will come up with similar results. Whilst there is no conclusive proof of this, teams with more compact stadiums often have a greater advantage. I support Gateshead FC and we do not have a particularly large home advantage, which is probably due to the large distance from the pitch to the stand, and the low attendances.
Answer: Over the last handful of years a great deal has changed in the analysis of likely outcomes in football. In no small part this has been influenced by approaches developed in American sports, hockey probably more than others.
Probably the best-known of the 'new' stats is Expected Goals (ExpG or xG). At its very simplest, ExpG tells us how likely a player or team is to score from a single shot. Suppose two teams from last weekend both had 5 shots on goal during a game, one team (A) scored 3 goals and the other (B) none. That suggests that shots on target are a poor predictor of scoring probability. However, if we know that team A's shots were all from inside the 6-yard box while B's were all from outside the penalty area, then we have a bit of context.
ExpG puts a quantitative quality score on each shot based on location and many other factors. As this is still a science in its infancy, there are many slight variations in the model that decides on the ExpG of a shot. That notwithstanding, if you know a team's average ExpG and average shot count per game then you can make a prediction of their likely scoring output.
This video by Dan Altman explains better than I.
One of the earliest mentions of ExpG was this blog post by Sam Green on the OptaPro blog.
A slightly later post introducing the concept is this one by Colin Trainor on StatsBomb.
ExpG is becoming more widely accepted with MLS probably leading the way, but UK newspapers and TV channels are starting to mention it.
Another is PDO -

The sum of a teams shooting percentage (goals/shots on target) and its save percentage (saves/shots on target against).

(From James Grayson's blogpost on the subject.)
I don't understand any of that so I won't even try to add anything useful.
A much simpler concept is Per90 -

Normalization for time, usually used in player stats. Instead of dividing a player's rate stat by games or appearances, you take all the minutes played and divide them into 90 minute chunks. Then take THAT number and use it as you would games played.

(From StatsBomb's Stat Definitions page.)
Answer: Goals scored and goals conceded in various forms (eg. think about using all goals scored in the season so far, goals scored against the specific opponent, goals scored in the last x games etc etc).
Comment: Do you have any evidence to show that these are *significant* factors?
Comment: If that is the case, this is not a suitable question for this site. However, I think the question can still be answered with evidence, as research has already been undertaken in this area.
Comment: Well, ELO ratings (at least for the international soccer calculation) do take into account goals in at least one way - I think it's goal differential, but don't remember the exact formula. Also, the post doesn't seem to be concerned with whether there's evidence that they're significant factors. Presumably Ubel will try to figure that out when building the logistic regression.
Comment: All answers on Stack Exchange should should be evidence-based when possible. Opinions are fine, but the entire answer generally shouldn't be primarily opinion based.
Comment: @studro: you have misunderstood the original question. At this stage he's brainstorming inputs to his model. He can take my suggestion and do the statistical verification on his own, which might reject or confirm my idea.
Answer: I think you have covered most of the important bases. But one thing to remember is current form which is very important in today's game. For example, just a few days ago, chelsea fc, a highly rated team who are the defending champions in the premier league  but are going through a bad patch, lost to west ham united fc, a team which finished in 12th position last season, 40 points behind chelsea. So maybe include a way to take into consideration the results in the last 5 games.
Answer: I think the defense quality of certain teams plays a big role. For example, Juventus in the season 2015/16 is a team that did not create very many chances in most games compared to teams like roma(seria A), Barcelona(laliga), Arsenal(epl) or Real madrid(laliga) just to name a few. By this I mean, in most cases their ELO ratings ranged between 30-50, whereas teams like arsenal and barcelona it would range mostly between 40-70. However, they were solid in defence, and even now I see them win so many games by single goal diffences especially away from home. To strengthen this fact, the season 2015/16, Juventus had an average conceeding rate on 0.53 whereas they scored 1.97 averagely, matches with odd goals where 21 while even goals were 17, but they only drew in goalless matches twice both away. This shows that their ability to defend was a major key factor in them winning matches.
refer these statistics from Punterslounge.com
Comment: Do you have any *quantitative* evidence to back up the assertions made in this answer? i.e. can you show that, overall, without cherry picking stats of one team, teams with strong defences do better than their ELOs would predict?
Comment: @Steve Thanks for your comment! I think your idea is getting more and more publicity lately, I see your point. For Example, Atlético Madrid or Juventus can compete other teams with higher net worth (Barcelona, Bayern). Can you elaborate this hypothesis with some data or analysis?


Animação com Problema no Safari

Question: Há um tempo percebi que no Safari as animações funcionam de uma maneira diferente.
Por exemplo, se tivermos um texto dentro de um span e queremos que ao hover as letras desse texto diminuam 1px, colocamos letter-spacing: -1px;. Colocando um transition: all 0.5 ease;, no Chrome e Firefox funcionam suavemente.
Mas no Safari, isso acontece abruptamente, ignorando a suavidade da transição. Por que ?
Porque eu não defini no elemento normal um letter-spacing: 0px;. Ou seja, é preciso que a propriedade esteja com um valor inicial para acontecer a transição. Apenas no Safari acontece isso.
Agora no caso abaixo não consigo contornar a situação.
Ao entrar no site, duas imagens executam uma animação. Mas ao hover dessas imagens, acontece outro efeito.
Daí quando tira o mouse de cima da imagem, ela volta nas suas propriedades iniciais e não para as propriedades que defini no 100% do keyframes da animação.
No CSS defini no elemento que a imagem vai começar com:
opacity: 0;
transform: scale(0);

E no KeyFrame defini que vai terminar com:
opacity: 1;
transform: scale(1);

Daí no Safari a imagem volta com opacity:0...
CSS
    .menu-language-full{
        position: absolute;
        right: 0;
        width: 100px;
        height: 50px;
        display: block;
        z-index: 1;
        top: 15px;
        img{
            margin-left: 5px;
            vertical-align: middle;
            opacity: 0;
            transform: scale(0);
            animation-name: ShowImgIdioma;
            animation-timing-function: cubic-bezier(0.68, -0.55, 0.27, 1.55);
            animation-fill-mode: forwards;
            animation-duration: 0.5s;
            transition: all 0.6s ease;
            &.pt-br{
                animation-delay: 2.2s;
            }
            &.en{
                animation-delay: 2.4s;
            }
            &:hover{
                transform: scale(1.1);
                filter: blur(1px);
            }
        }
    }

KeyFrames
@keyframes ShowImgIdioma{
    100%{
        opacity: 1;
        transform: scale(1);
    }
}

Comment: voce prefixou o `keyframe` ? se não me engano, essa é uma exigência do safari
Comment: Sim, prefiquissei.
Answer: Isso pode ocorrer pelo browser não estar reconhecendo as propriedades css tente como o exemplo abaixo, que é um exemplo onde o mesmo é executado em todos navegadores.
//OPACITY
-moz-opacity: 0.28;
-khtml-opacity: 0.28;
opacity: 0.28;
-ms-filter:"progid:DXImageTransform.Microsoft.Alpha"(Opacity=28);
filter: progid:DXImageTransform.Microsoft.Alpha(opacity=28);
filter:alpha(opacity=28);

//TRANSITION
-webkit-transition: all 01s ease;
-moz-transition: all 01s ease;
-ms-transition: all 01s ease;
-o-transition: all 01s ease;
transition: all 01s ease;

//TRANSFORM
-moz-transform: scale(1);
-webkit-transform: scale(1);
-o-transform: scale(1);
-ms-transform: scale(1);
transform: scale(1);

Comment: O meu está assim Rodrigo. É que eu programo em LESS e depois compilo em CSS com o Gulp.Js. E ele faz esses prefixes automaticamente.

Animação com Problema no Safari


All real numbers can be expressed as a limit of rational numbers?

Question: RTP

Let $C$ be a set of Cauchy sequences. $\forall x \in {\Bbb R}, \exists \{a_n\} \in C$ sucht that ${a_n} \to x$.

I have no clue to even start this problem.
All I know so far is that $\Bbb R$ is a set of equivalent classes of the limit of rational Cauchy sequences.
Can anyone give me a proof or a hint ?
I am new to analysis, so it would be nice if you could tell me the rationale of this problem as well.
Comment: $C$ is the set of Cauchy sequences with members in $\Bbb Q$ or in $\Bbb R$?
Answer: I'm going to write this answer on the assumption that you're working through a fairly rigorous construction of the reals; if you're not, it's more technical than you actually need.
Let $x\in\Bbb R$; then by the definition with which you're working, $x$ is an equivalence class of Cauchy sequences of rational numbers. Let $\sigma=\langle p_k:k\in\Bbb N\rangle$ be one of those Cauchy sequences. The whole idea of this construction of the real numbers is that $x$ should be the limit of the sequence $\sigma$. There's a technical problem, however: the rational numbers $p_k$ aren't actually in $\Bbb R$ when $\Bbb R$ is viewed as a set of equivalence classes of Cauchy sequences of rationals. However, if you've been shown this construction, you should have been shown that this $\Bbb R$ contains a copy of the rationals.
Specifically, for each $k\in\Bbb N$ let $\pi_k$ be the constant sequence $\langle p_k,p_k,p_k,\ldots\rangle$; then $\bar\pi_k$ is the real number that corresponds to the rational $p_k$, where I write $\bar\tau$ for the equivalence class of a sequence $\tau$ of rationals. Thus, we expect that $\langle\bar\pi_k:k\in\Bbb N\rangle$ converges to $x=\bar\sigma$ in $\Bbb R$. To show this, we must show that for each $\epsilon>0$ there is an $m_\epsilon\in\Bbb N$ such that $\bar d(\bar\pi_k,\bar\sigma)<\epsilon$ for each $k\ge m_\epsilon$, where $\bar d$ is the metric in $\Bbb R$. If you've done the rigorous development, you know that
$$\bar d(\bar\pi_k,\bar\sigma)=\lim_{n\to\infty}|p_k-p_n|\;.$$
Since $\sigma$ is a Cauchy sequence, there is an $m_\epsilon\in\Bbb N$ such that $|p_\ell-p_n|<\frac{\epsilon}2$ whenever $\ell,n\ge m_\epsilon$.

Now show that $\bar d(\bar\pi_k,\bar\sigma)\le\frac{\epsilon}2<\epsilon$ whenever $k\ge m_\epsilon$ and conclude that $\langle\bar\pi_k:k\in\Bbb N\rangle$ is a sequence of rationals in $\Bbb R$ that converges to $x=\bar\sigma$.
Comment: Do infinite decimals provide a rigorous construction of the reals?
Comment: @user72694: One could use them as the basis for a rigorous construction, but it would involve a lot of work; off the top of my head I suspect that overall it would be a bit more work than either the construction using Dedekind cuts or the construction using equivalence classes of Cauchy sequences.
Comment: There are textbooks that use specifically the infinite decimal construction. I am not claiming this approach is preferable to the others, but that it is feasible, and in certain cases preferable (as the case in point). My former colleague D. Barlet (the famous complex analyst) preferred this construction over the others.  Notice that the idea of representing all numbers by infinite decimals is older than people generally believe.  In fact, it is due to Stevin, over 300 years before Cantor, Dedekind, and Weierstrass.  I have a paper on the subject; if you drop me an email, I can send you a copy
Comment: OMG I am so intimidated by what you guys are saying. I really appreciate your help, but I feel as though I am not at that level where I can understand you at all.
Comment: @hyg17: What text are you working from? (It'll be at least half a day before I get back to this, but I will check.)
Answer: Consider an infinite decimal expansion of $x$, and truncate it at rank $n$.  This gives a sequence $(a_n)$ that tends to $x$.
Comment: Every real number has a decimal expansion, whether or not it is definable.  If the number is not definable, then its string of decimal digits will not be definable, but it certainly "exists" in the usual mathematical sense.  In fact, the reals can be constructed using infinite decimals, see http://en.wikipedia.org/wiki/Construction_of_real_numbers#Stevin.27s_construction
Comment: It was mentioned to me that "Any Goedel-style enumeration can cover only those real numbers that are definable by formulas (in some fixed language). Thus, if in one's set theory, there are uncountably many real numbers, then some of these numbers must be undefinable (by formulas)". My comment is: A real number x cannot therefore, in general, be given - and therefore cannot be truncated. This can only be done for a countable subset of the real numbers, such as Q.
Answer: Let $x\in Q$ Then define the sequence of rationals by $x_n=x-\frac{1}{n}$.(Note that $x_n\in R$) Then $x_n\to x$ as $x\to \infty.$
Let $x\in Q^c$.Let $A_n=(x-\frac{1}{n},x+\frac{1}{n})$.There exists $x_n\in Q\cap A_n$ using the claim 1 below. So we have $x_n\to x$ as $n\to \infty$.
Claim 1:Let $p\ne q$ and $p,q\in R$. I will prove that $\exists x\in Q$ with $p<x<q$.
By Archimedean principle $\exists m\in R$ such that $m(q-p)>1\Rightarrow mq-mp>1$
Now as $mq-mp>1$ so $\exists n\in N$ such that $mp<n<mq\Rightarrow p<\frac{n}{m}<q$ and as $\frac{n}{m}\in Q$ so $x=\frac{n}{m}$ satisfies the claim.
Proof of Archimedean property,
Let $a>0$ and $b\in R$ and $S=\{n|n\in N\text{ and }na\le b\}$ We know that as $a>0$ so $na\to \infty $ as $n\to \infty$ so $|S|<\infty$ i.e. $S$ is finite.And as $S$ is finite so $\sup S$ exists. Now by choosing $m\in N, m>\sup S$ we have a natural no. such that $ma>b$ . This proves the archimedean property.
Comment: Could you explain me how $x - \frac{1}{n}$ is a rational ? I don't see how $\{x_n\}$ can be a valid rational sequence.
Comment: If $x=p/q$ then $x-1/n=(np-1)/q$ so it is a rational.
Comment: But x is not necessarily rational.
Comment: In the first case only i have considered x to be rational.
Answer: The real numbers are, or can be, defined using the type limit of limit you mention so there is nothing to prove, except that axiom A implies A.


php: how to split up a really long string

Question: I am new to PHP and have a question.
I've made a chat-like application, it is working well but i can't seem to find exactly what i want to do in searching previous threads.
The user can input from a form field their $message. So...
$message = $_POST['message'];

The input box allows a maximum of 600 characters. Now if the user posts nothing but one long entire string that doesn't contain any white space, or spaces between any of the string characters.. short example..
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
If it goes to a certain length, it causes the page to break by creating a horizontal scroll frame.
How can I take $message and filter it so that it will break this up? But also taking into account that most users are posting a real $message, and not to accidentally split up any valid words etc. maybe something that checks to see if one of the $message array elements is longer than xxx amount of characters long and if so break it up? Or what's the easiest way to go about this?
Comment: this sounds like a layout problem. would you rather split the string or fix the layout issue with css?
Comment: You want to truncate the string ?
Comment: I'm open to any valid answer. im not much of a css guru either so any examples in either case would be appreciated
Comment: Use `word-wrap:break-word;` for CSS.
Answer: You should put the content of $message inside an element with a class set to break long lines (word-wrap: break-word):

i.e:
.php
<?php

if(isset($message = _$POST['message'])){
$message = _$POST['message'];

echo "
$message

}

.css
.message {
  word-wrap: break-word;
/* Be VERY careful with this, breaks normal words */
  word-break: break-all;

-webkit-hyphens: auto;
   -moz-hyphens: auto;
        hyphens: auto;
}

The above works in Internet Explorer 8+, Firefox 6+, iOS 4.2, Safari 5.1+ and Chrome 13+.
CODEPEN DEMO
Answer: Of course I found an answer online just after posting this. The answer is to put this into the body of my css:
overflow-x:hidden;
word-wrap: break-word;

Answer: I think the simplest way
<?php
$f = 'ffffffffffffffffffffffffffffffffffffffffffffffffffffff';
$row = 3;
$n = ceil(strlen($f) / $row);
for($i = 0; $i < $row; $i++){
    echo substr($f, 0, $n) . '
';
}
?>

Comment: This is `css` problem, and you won't solve with your code. What will happen if the device is 320*240 and it's font set to Large++ ?


E string of guitar broken so every wire will be replaced with new?

Question: During tuning a new set of strings, my top E string snapped.
My colleague says that you have to buy a complete set of strings because it does not come alone and every set have its own frequency so:

Can I buy complete set of strings instead of 1 broken string that is E?
Can the only string that is E will be easily available and replaced?
Answer: In England, at least, every guitar shop I've ever been in will sell individual strings. A good job too, as I use an unusual combination of gauges, meaning I'd have to buy three sets to make one for my guitars. So, yes, they are available, maybe you can buy 10 of each gauge on the internet; one individual from a shop. However, a set won't break the bank, and you only need to use the same as you broke - then you have spares.
It's also very dependent as to how old the original strings are. If you broke one while fitting a new set, obviously you wouldn't put a whole new set on. But if the set is quite old, rusty, or sounds dull, then yes, change the lot.
Answer: Yes - you can buy individual strings for guitars.
Typically I do keep a couple of spare E-strings in case I break one during a performance, but generally I replace all 6 at the same time

As Bayley pointed out, a set is really rather cheap
it's an opportunity to replace the other strings as they will be rusting and picking up dirt.

So if your strings were new and you just broke the top E, yes, I'd suggest just replacing it.
Answer: If you break the top e string during installation of a new set, you can certainly buy a single e string.  There is no need to buy a whole set if the strings you did not break are all new.
I shop at Just Strings dot com for individual strings.  Here is a link to their single guitar strings page Single Strings at Just Strings
Here you will find single strings from many of the major manufacturers in various gauges.  Try to pick one that most closely matches the one you are replacing in composition, type, gauge and brand.  The high e string is a plain steel string so most brands will be interchangeable.  Just get the same gauge as the one you broke.  Or you can get a different gauge if you prefer.
Some local guitar shops might carry a selection of individual strings as well - and you will save the shipping costs.  When I buy individual strings on line, I buy several because there is not much difference in shipping cost between one string or ten strings.
I buy single strings because I like to choose each gauge of my sets individually.  I might have heavier bottoms for more base and lighter tops for easier bending.
Also, I like to keep a few extras of the strings most likely to break.  In addition to the thinnest high e-string, a wound G string is very susceptible to breaking as well.  The reason is the steel core (where the tensile strength comes from) of a wound G string is actually thinner in diameter than the high e string.  So next time you hear a guitarist complain from the stage that their G String just broke, you will know they are not trying to be funny.  It was most likely the g string that broke.
If I break a string on stage, I can't take time to change an entire set and I don't like to rob a full set just to get one string because then when I go to install that set it will be a string shy of a full set.
Good luck with your next install.  Be sure to stretch your strings gently after installation.  Also, if your string breaks near the bridge - check to be sure there is nothing sharp on your saddle that might cut into the string.  And tune the high e up very slowly.
Answer: You can put any combination of strings as you see fit. There is no need to have a matching set. I have known guitarists that use sometimes two and three of the same strings and just tune them differently. It's all about the sound you are looking for.
Answer: In theory you should replace all strings with a new set, in order to maintain the same degree of aging in all strings.
As Bayley already stated in his answer, older strings need to be more stretched to be kept in tune, and metal strings (you don't specify which type of guitar and strings you are talking about) oxidize wich gives them a different tone coloration.
So if you replace only one string, you mat get differences in tone and tension that may harm your playing.
Having said that, I too keep getting my 1st string (high E) breaking, both in my electric and acoustic (folk type) guitar (specially since my son started using them regularly :-), and if I would change all strings every time it would be a small fortune.
Some string manufacturers provide sets with an additional E 1st string, so you can keep the same set while changing the E string once. This seems like a good compromise (remember, you should change all your strings anyway every so often, as they become dull and harder and harder to play).
But if you break your 1st string once or twice a month like my son does, you can buy just the one string. It's not easy to find in the suppliers but it can be found, or order directly from China through ebay. I've ordered 10 pieces each (~5$ for 10 strings) of 1st strings for electric and folk guitar of the Alice brand and I'm quite happy with them (for practising and home usage).
If you go that route, some degree of difference in tone and playability is probably anavoidable, but to minimize that be careful to buy the same gauge as the one you're replacing.
Comment: "older strings need to be more stretched to be kept in tune" -- don't you mean the other way round?
Comment: @Johannes, vibration frequency is proportional to string tension (actually to the square root of tension), so when a string losens by being under tension for a long time you have to stretch it **more** (i.e., turn the tuner so that the string stretches) to regain the necessary tension for a certain pitch. It's true that the frequency also diminishes with (the square root) of linear density, and as the strings losens its linear density decreases (it becomes "thinner"). However the effect of decreasing linear density is marginal compared to the effect of decreasing tension.
Answer: Dude, its like $11 bucks. Just buy the new pack. But to answer your question, no i have never seen individual strings for sale, with the expectation of some very low guitar tunings.
When replacing the E string its best to also do all the other strings as otherwise you will have tuning issues if only the newer string is stretching.
Comment: Replacing all will usually give even more tuning issues till they settle down. I'd rather be chasing one string than all 6.


What is a manifold?

Question: In dimensionality reduction technique such as Principal Component Analysis, LDA etc often the term manifold is used. What is a manifold in non-technical term? If a point $x$ belongs to a sphere whose dimension I want to reduce, and if there is a noise $y$ and $x$ and $y$ are uncorrelated, then the actual points $x$ would be far separated from each other due to the noise. Therefore, noise filtering would be required. So, dimension reduction would be performed on $z = x+y$. Therefore, over here does $x$ and $y$ belong to different manifolds?
I am working on point cloud data that is often used in robot vision; the point clouds are noisy due to noise in acquisition and I need to reduce the noise before dimension reduction. Otherwise, I will get incorrect dimension reduction. So, what is the manifold here and is noise a part of the same manifold to which $x$ belongs?
Comment: It's not really possible to use the term correctly without being mathematically precise
Answer: In non technical terms, a manifold is a continuous geometrical structure having finite dimension : a line, a curve, a plane, a surface, a sphere, a ball, a cylinder, a torus, a "blob"... something like this :

It is a generic term used by mathematicians to say "a curve" (dimension 1) or "surface" (dimension 2), or a 3D object (dimension 3)... for any possible finite dimension $n$. A one dimensional manifold is simply a curve (line, circle...). A two dimensional manifold is simply a surface (plane, sphere, torus, cylinder...). A three dimensional manifold is a "full object" (ball, full cube, the 3D space around us...).
A manifold is often described by an equation : the set of points $(x,y)$ such as $x^2+y^2=1$ is a one dimensional manifold (a circle).
A manifold has the same dimension everywhere. For example, if you append a line (dimension 1) to a sphere (dimension 2) then the resulting geometrical structure is not a manifold.
Unlike the more general notions of metric space or topological space also intended to describe our natural intuition of a continuous set of points, a manifold is intended to be something locally simple: like a finite dimension vector space : $\mathbb{R}^n$. This rules out abstract spaces (like infinite dimension spaces) that often fail to have a geometric concrete meaning.
Unlike a vector space, manifolds can have various shapes. Some manifolds can be easily visualized (sphere ,ball...), some are difficult to visualize, like the Klein bottle or the real projective plane.
In statistics, machine learning, or applied maths generally, the word "manifold" is often used to say "like a linear subspace" but possibly curved. Anytime you write a linear equation like : $3x+2y-4z=1$ you get a linear (affine) subspace (here a plane). Usually, when the equation is non linear like $x^2+2y^2+3z^2=7$, this is a manifold (here a stretched sphere).
For example the "manifold hypothesis" of ML says "high dimensional data are points in a low dimensional manifold with high dimensional noise added". You can imagine points of a 1D circle with some 2D noise added. While the points are not exactly on the circle, they satisfy statistically the equation $x^2+y^2=1$. The circle is the underlying manifold:
Comment: @RiaGeorge In the picture it is the *surface* that is a manifold.  It's continuous because you can move around it freely without interruption and never have to jump *off* the surface to get between any two places.  The holes you allude to are important in describing *how* you can get around on the surface between any two points in the simplest way, and counting them is an important technique in studying manifolds.
Comment: Explaining what topology is would be way too broad a question for this site, and a bit off topic.  I would search the mathematics stack exchange for information on that.  Manifolds and topology are not synonyms: manifolds are mathematical objects studied with the techniques of topology, topology is a sub-subject of mathematics.
Comment: Technical point, the solution set of a system of equations need not be a manifold. It's a variety, so it's mostly a manifold, but it can have points of self intersection where the manifold property fails.
Comment: This seems like a very good explanation to someone learning about the concept for the first time, with well-chosen, concrete examples. (I don't know for certain though since I have encountered the concept before.) As a minor quibble, I would recommend rephrasing the last sentence to be less absolute ("Anytime the equation is non-linear like..."): as it is written right now, it is not actually true. Apart from that minor quibble, I find this very well-written.
Comment: The answer misses all the fundamental points that make a manifold such, I don't get how it has so many upvotes. Topology, charts and smoothness are not even mentioned and the answer basically gives the impression that a manifold is a surface, which it is **not**.
Comment: Your manifold definition includes the requirement to be of *finite dimension*. But you include examples that do not meet that requirement—such as lines, planes, curves and surfaces. Could you please clarify what you meant?
Comment: @Mowzer: Finite dimensionality means that the number of dimensions is finite. For instance, a plane is a 2-dimensional object, and 2 is finite.
Comment: @RiaGeorge : About "topology", the question is interesting, but maybe the site "Mathematics" (https://math.stackexchange.com/) would be a better place. And like Matthew Dury says, it's very broad...
Comment: @Mowzer You may be confusing (set) cardinality https://en.wikipedia.org/wiki/Cardinality with (topological) dimension https://en.wikipedia.org/wiki/Lebesgue_covering_dimension For manifolds the dimension is simpler to understand, it is just the value of $n$ for which there are local homeomorphisms with $\mathbb{R}^n$, as a corollary of Brouwer's invariance of domain theorem, the dimension of manifolds is a topological invariant.
Comment: @Chill It's a very broad term that means the solution set of a system of equations. Generally what is meant is an algebraic variety implicitly, but there are other examples. See https://en.wikipedia.org/wiki/Complex-analytic_variety and https://en.wikipedia.org/wiki/Variety_%28universal_algebra%29
Comment: @Chill For the zero set of arbitrary smooth functions it's not really an interesting concept because it can be any closed set.
Comment: Thank you for your answer supported with a simple example and the picture. Can you please explain what a topology is in non-technical term as well? Is the term topology and manifold used interchangeably? Does the dimension has to be an integer number? What is it is a real number, then I think the structure is known as fractals if the entire structure is composed of each subpart is self-repeating. Lastly, in the picture there are three empty parts, hollow without the green material still why it is a continuous geometric structure?
Comment: Then, can you please include the case of noise that I asked in my question? In a point cloud data, say there are outliers (noise). Would the noise be considered part of the same manifold as the non-noise data?
Comment: @MattSamuel Technically that is true only when the equations are algebraic. For general non-linear equations (based on differentiable functions) the essence of your conclusion still does hold, since by the inverse/implicit function theorem in a neighborhood of any non-singular point the solution set will be a manifold, and by Sard's theorem non-singular points are generic. (But to the best of my knowledge the term variety is only appropriate for solution sets of algebraic equations.)
Comment: @Chill It's still a variety, just not an algebraic one. If it's not algebraic it's even easier for it to fail to be a manifold.
Comment: @MattSamuel would you mind providing a reference for this usage of the word variety? I have never heard it before and am curious.
Answer: A (topological) manifold is a space $M$ which is:
(1) "locally" "equivalent" to $\mathbb{R}^n$ for some $n$.
"Locally", the "equivalence" can be expressed via $n$ coordinate functions, $c_i: M \to \mathbb{R}$, which together form a "structure-preserving" function, $c: M \to \mathbb{R}^n$, called a chart.
(2) can be realized in a "structure-preserving" way as a subset of $\mathbb{R}^N$ for some $N \ge n$. (1)(2)
Note that in order to make "structure" precise here, one needs to understand basic notions of topology (def.), which allows one to make precise notions of "local" behavior, and thus "locally" above. When I say "equivalent", I mean equivalent topological structure (homeomorphic), and when I say "structure-preserving" I mean the same thing (creates an equivalent topological structure).
Note also that in order to do calculus on manifolds, one needs an additional condition which doesn't follow from the above two conditions, which basically says something like "the charts are well-behaved enough to allow us to do calculus". These are the manifolds most often used in practice. Unlike general topological manifolds, in addition to calculus they also allow triangulations, which is very important in applications like yours involving point cloud data.
Note that not all people use the same definition for a (topological) manifold. Several authors will define it as satisfying only condition (1) above, not necessarily also (2). However, the definition which satisfies both (1) and (2) is much better behaved, therefore more useful for practitioners. One might expect intuitively that (1) implies (2), but it actually doesn't.
EDIT: If you are interested in learning about what precisely a "topology" is, the most important example of a topology to understand is the Euclidean topology of $\mathbb{R}^n$. This will be covered in-depth in any (good) introductory book about "real analysis".
Comment: This is the only answer so far that pays attention to the modern mathematical idea of assembling a global object from local data.  Unfortunately, it doesn't quite make it to the level of simplicity and clarity required of a "non-technical" account.
Comment: @RiaGeorge $n$ stands for a natural number (integer $\ge 1$), as does $N$. There might be more advanced theory for fractional/real-valued dimensions, but it doesn't come up as often. "Topology" and "manifold" mean two very distinct things, so they are not interchangeable terms. A "manifold" has a "topology". The field of Topology studies spaces which have "topologies", which are collections of sets satisfying three rules/conditions. One goal of studying "topologies" is to describe in a consistent and reproducible way notions of "local" behavior.
Comment: Thank you for your answer: Can you please explain what a topology is in nnon-technical term as well? Is the term topology and manifold used interchangeably? Does the dimension has to be an integer number? What is it is a real number, then I think the structure is known as fractals if the entire structure is composed of each subpart is self-repeating.
Comment: @RiaGeorge The axioms for a "topology" can be found on the Wikipedia page: https://en.wikipedia.org/wiki/General_topology#A_topology_on_a_set -- note also that the link I gave you for the (equivalent) definition of "topology" in terms of neighborhood pointed to something related but not the same, I have edited my answer to reflect this: https://en.wikipedia.org/wiki/Neighbourhood_(mathematics)#Topology_from_neighbourhoods Note however that the definition in terms of neighborhoods is more difficult to understand (I imagine I could understand it well, but I don't bother too, because I'm lazy
Comment: so anyway it's my personal biased opinion that you don't need to know the neighborhood definition of topology -- just know that the simpler definition gives you all of the same power of the neighborhood definition in terms of rigorously describing local behavior, since they are equivalent). Anyway, if you are interested in fractals, maybe you will find these Wikipedia pages interesting -- I can't help you out with that more though, because I am not deeply familiar with the theory and don't know or understand most of the definitions -- I have only heard of some of the
Comment: relevant terms. https://en.wikipedia.org/wiki/Hausdorff_dimension  https://en.wikipedia.org/wiki/List_of_fractals_by_Hausdorff_dimension That being said, the relationship between fractals and manifolds seems to be complicated at best, although I don't know or understand much about fractals. https://math.stackexchange.com/questions/1340973/can-a-fractal-be-a-manifold-if-so-will-its-boundary-if-exists-be-strictly-on  https://math.stackexchange.com/questions/1406058/can-a-fractal-be-a-manifold  One problem is that Hausdorff dimension seems to be defined for spaces more general than manifolds.
Answer: In this context, the term manifold is accurate, but is unnecessarily highfalutin. Technically, a manifold is any space (set of points with a topology) that is sufficiently smooth and continuous (in a way that can, with some effort, be made mathematically well-defined).
Imagine the space of all possible values of your original factors. After a dimensional reduction technique, not all points in that space are attainable. Instead, only points on some embedded sub-space inside in that space will be attainable. That embedded sub-space happens to fulfill the mathematical definition of a manifold. For a linear dimensional reduction technique like PCA, that sub-space is just a linear sub-space (e.g. a hyper-plane), which is a relatively trivial manifold. But for non-linear dimensional reduction technique, that sub-space could be more complicated (e.g. a curved hyper-surface). For data analysis purposes, understanding that these are sub-spaces is much more important than any inference you would draw from knowing that they fulfill the definition of manifold.
Comment: *Mathematically, a manifold is any locally continuous topological space.* I like the idea of trying to explain things in plain language, but this characterization really doesn't work. First off, continuity is always a local property, so I'm not sure what you mean by locally continuous. Also, your definition fails to rule out a lot of things that aren't manifolds, such as the rational number line, or the union of two intersecting lines in the Euclidean plane.
Comment: "Highfalutin"... learned a new word today!
Comment: I agree with Ben, technically it's "locally euclidean".  I'm not sure there is a good way to boil that down to simple english.
Comment: I also have to agree strongly with the two comments above. In fact, the answer I wrote below was originally meant to be a clarifying comment to this answer which became too long. There is no precise notion of a "continuous" topological space (see here: https://math.stackexchange.com/questions/1822769/how-to-axiomize-the-notion-of-continuous-space ). Defining manifolds in terms of non-existent concepts is, in my opinion, in the long-run more likely to be confusing than clarifying. At the very least, I would suggest replacing the word "mathematically" in the first sentence with something else.
Comment: I'll use this comment as an opportunity to ask a little question...I (think) I got the idea of manifolds, but why is it "locally" needed? Isn't a space "locally" continuous...continuous as a whole?
Comment: @BenCrowell and others: I agree with your objections. Since the whole point of my answer is "the details of the mathematical definition aren't important for the usage in data science", I'm not too worried about its admitted deficiencies. But since you are, and since someone who does want the mathematical definition might stumble onto this post, I've edited to improve it a bit. Note that I'm well-aware of the usual attempted to summarize the definition as "locally like ${\mathbb R}^n$; I find it neither mathematically nor colloquially enlightening.
Comment: @DavidWright: Thank you for your answer and the edited updates. In your answer, you have mentioned the example of PCA I have a doubt which is in many cases the original data contains outliers or noise. So, we do data processing before dimension reduction. However, there may be other examples where noise  is not filtered and we do dimension reduction using PCA. Would noise be considered to be part of the same manifold to which the data points belong? Can you provide some insights into this?
Comment: @RiaGeorge: Once you do dimensional reduction, the original points are typically not in the reduced space. But if the reduction is good, they are _close to_ the reduced space. You could regard their displacement from the reduced space as noise; you would essentially be saying that the reduced space is the underlying available space and any displacements from it are small random errors. But whether you call that displacement noise or not, the original points will generally not be exactly on the reduced manifold.
Answer: As Bronstein and others have put it in Geometric deep learning: going beyond Euclidean data (Read the article here)

Roughly, a manifold is a space that is locally Euclidean. One of the
simplest examples is a spherical surface modeling our planet: around a
point, it seems to be planar, which has led generations of people to
believe in the flatness of the Earth. Formally speaking, a
(differentiable) d-dimensional manifold X is a topological space where
each point x has a neighborhood that is topologically equivalent
(homeomorphic) to a d-dimensional Euclidean space, called the tangent
space.
Comment: Their description of a manifold is not accurate as they slip in the notion that it is differentiable. Such a smooth manifold has a tangent space, but is not *just* a manifold. But *all* manifolds are second-countable Hausdorff topological spaces locally homeomorphic to Euclidean space. Riemannian manifolds are smooth manifolds equipped with a Riemannian metric.
Comment: The quotation is self-contradictory. At the outset it describes a Riemannian manifold ("locally Euclidean") but at the end it describes a topological manifold (homeomorphisms do not, by definition, have to respect the differential structure and therefore the concept of tangent space does not apply).

What is a manifold?


Rename all files in a directory based off of columns in another index data frame

Question: I have a large number of csv files in a directory that I need to rename based off of corresponding cols in another index/reference data frame. Here is a three element sample of what I'm dealing with:
dir.create("dir1")

write.csv(mtcars[1:2,], "dir1/20821659.csv", row.names=FALSE)
write.csv(mtcars[3:4,], "dir1/20821654.csv", row.names=FALSE)
write.csv(mtcars[5:6,], "dir1/20821657.csv", row.names=FALSE)

Now I have another data frame with the orignial names of these files in one column, and another column that I would like to use to rename them:
location <- c("SFM01_2", "SFM05_2", "02M08_2")

sn <- c("20821659", "20821654", "20821657")

df<- data.frame(location, sn)

For example, the location name that corresponds to the first file name (20821659) is SFM01_2, and I would like to change that file name to SFM01_2 and so on for all the many files in this folder.
Answer: You could loop over the rows, each time using paste0() to create a mv command, which is then provided to system()
purrr::walk(1:nrow(df),function(i) {
  cmd = paste0("mv dir1/",df[["sn"]][i], ".csv dir1/", df[["location"]][i], ".csv")
  system(command=cmd)
})

Answer: Tested. file.rename returns TRUE on success.
dir1 <- "dir1"
apply(df, 1, \(x) {
  new <- paste0(x[1], ".csv")
  new <- file.path(dir1, new)
  old <- paste0(x[2], ".csv")
  old <- file.path(dir1, old)
  if(file.exists(old)) file.rename(old, new)
})
#[1] TRUE TRUE TRUE

Answer: Here is a solution using mapply. You can create a new dataframe with the full paths of the files. Then, rename the file using the specification of the 2 columns row by row .
dir.create("dir1")

write.csv(mtcars[1:2,], "dir1/20821659.csv", row.names=FALSE)
write.csv(mtcars[3:4,], "dir1/20821654.csv", row.names=FALSE)
write.csv(mtcars[5:6,], "dir1/20821657.csv", row.names=FALSE)
list.files('dir1') # "20821654.csv" "20821657.csv" "20821659.csv"

location <- c("SFM01_2", "SFM05_2", "02M08_2")

sn <- c("20821659", "20821654", "20821657")
df<- data.frame(location, sn)

# Create a new dataframe with the full paths of the files
df2 <- sapply(df, function(i){
  paste0('dir1/', i, '.csv')
})
# rename the file using the specification of the 2 columns row by row
mapply(FUN = file.rename, from = df2[, 2], to = df2[, 1],
       MoreArgs = NULL, SIMPLIFY = TRUE, USE.NAMES = TRUE)

list.files('dir1') # "02M08_2.csv" "SFM01_2.csv" "SFM05_2.csv"


AzCopy a directory by Wildcard pattern Prefix

Question: Current Migration:
I'm currently in the process of migrating a storage account with 100TB + of data from one subscription to another. The legacy storage account has a very distinct folder structure:

ContainerName//{date}/filename.txt

Examples:

ContainerName//20140327/readme1.txt
ContainerName//20140328/readme2.txt

I don't know why they decided to use an empty folder directory // to house this data(before my time). There is a virtual directory on every date from 2014 up until present. Due to the sheer size of this migration, I would like to break it up into multiple azcopy jobs. For example migration 1 would migrate all data from 2014, migration 2 would migrate all data from 2015 and so on....
Problem:
Is there a way to perform an AzCopy based on a blob name prefix, since azure storage does not use real folders, rather it uses virtual folders. For example only migrate data with the following prefix: containerName//2014*
What I have tried:
I see that the most recent version of AzCopy now has --include-pattern command. I have tried multiple ways of doing the following:
<code>azcopy.exe copy $srcContainerUrl $destContainerUrl --recursive --include-pattern "//2014*"
</code>
but it never seems to be able to find the files I am trying to migrate
Comment: Before you start, do two things: 1) Analyze the costs of moving 100 TB. 2) Perform a sample move to estimate time. Moving 100 TB will cost you around $10K and take months (~5 TB per week). That is just a rough guesstimate. I would talk to Azure support first.
Comment: Try using "—include-path" instead of "—include-pattern". Based on the documentation, former is for virtual folders while latter is for blob names. However I don't think using azcopy will solve your "//" problem as the same structure will be copied in your new storage account.
Answer: After going through your post, I recalled that there is a way through which you can migrate all the files and folders from one subscription to another according to the time preferences. You can use '—include-before' or '--include-after' option with the azcopy command. Here you have to specify date and time in ISO-8601 format. Below is an example and reference article from Microsoft: -
Syntax: - azcopy copy '*' 'https://..core.windows.net/' --include-after
Example: - azcopy copy 'C:\myDirectory*' 'https://mystorageaccount.blob.core.windows.net/mycontainer/FileDirectory'  --include-after '2020-08-19T15:04:00Z'
Article: - https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-blobs-upload?toc=/azure/storage/blobs/toc.json
P.S.: - Though, it is not the solution to your current problem, but the scenario as explained by you can be taken care of by this
Note: - Moving 100TB data from one subscription to another is not recommended as it takes more time (for months) and cost too.
Also, you can use the Azure Resource Mover to move the resource from one subscription to another. Please find the below reference links.
https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/move-resource-group-and-subscription
https://learn.microsoft.com/en-us/azure/storage/common/storage-account-move?tabs=azure-portal
Thanking you,


Why can we train calibrators using the original labels?

Question: When using calibrators (e.g. Platt Scaling, Isotonic Regression) to get better calibrated probabilities from our classifiers, we are effectively finding a mapping from the output scores of the classifier to the well-calibrated probability for that score. In other words, we want to find a function approximating the curve on the reliability diagram, like the one below that I took from MathWorks' website.

If I think of this as a regression problem, it would have the classifier scores as the x values, and the actual fractions of positives as the y values.
I understand we cannot actually use these fractions of positives due to not having an infinite amount of data, but instead we fit the calibrator using the original 0/1 training labels (or a separated batch to avoid bias). My question is why does this give us the curve we want, when these labels are seemingly very different from the fractions of positives in the reliability diagram?
Answer: I eventually convinced myself this is okay, so here's my intuition:
Assuming we had infinite data, at any point we could calculate the fraction of positives. We don't have infinite data, but essentially the same thing is happening. If at a certain x value, the true fraction of positives was 0.25, then we'd expect our finite data to reflect that, i.e. there should be roughly 3 times as many 0 labels around that area than 1 labels. This would pull the curve downwards to roughly the right place. The same thing happens in Logistic Regression when we want to train a binary classifier.


ANOVA vs R-Squared (Explained Variance)

Question: My goal is to determine whether for linear regression some predictors uniquely improve the fit beyond that which is already available via all other predictors combined. I have originally tried multi-way ANOVA and partial correlation for this purpose. I have recently learned that multi-way ANOVA performs poorly in the scenario with high multicollinearity. Namely, the reported significances and explained variances of individual predictors are not robust, and thus may misrepresent the true relations within the data.
Here is a solution that came to my mind:

Fit the full model to the data, find the coefficient of determination $r^2_{full}$
Exclude one of the predictors (e.g. $X$) from the model, fit the rest of the predictors, find $r^2_{/X}$
Then the gain of explained variance uniquely due to the excluded predictor $X$ is

$$G(X) = r^2_{full} - r^2_{/X}$$
Naively, this looks like a robust solution for detecting partial effects. My simulations show that it works better than partial correlation on some simple noisy model data, whereas the latter is known to fail to correctly discriminate between a true partial effect and multicollinearity in the presence of noise.
Questions:

Does this approach have a name?
Does it work in practice?
Is there a nice procedure to test $G(X)$ for significance (against null hypothesis that $X$ is random and can only explain variance by chance)? Permutation-testing seems to work for me, I'm just wondering if there is something similar to an F-test.

Note: I am only interested in applying the method for low total number of predictors such as 2 or 3. I am aware of the kitchen sink regression effect, so I want to make it clear that I do not intend to stretch this design to the extreme.
Comment: What you are describing seems to be (semi-) partial $R^2$.  But your original goal will be impossible to achieve when there are many predictors or collinearity, as the sample size is unlikely to be sufficient to disentangle the predictors.
Comment: @FrankHarrell Thanks! I'll look it up.
Answer: If you are willing to assume normally distributed errors, you could simply apply a likelihood ratio test as the models you want to compare are nested.
Edit: Assuming normal errors $\epsilon$ (with variance = 1), the likelihood function of the parameters in the model $$y_i = \theta_1x_{1i} + \theta_2x_{2i} + \theta_3x_{3i} + \epsilon_i$$ is given by $$\mathcal L(\theta) = (2\pi)^{-\frac 32}\exp\left(-\frac 12\Vert y - X\theta\Vert_2^2\right),$$
where $y$ is the n×1 vector of $y_i$'s, $X$ is the n×3 matrix of $(x_{1i}, x_{2i}, x_{3i})'$ and $\theta = (\theta_1,\theta_2,\theta_3)'$. The likelihood ratio test for testing, say, $\theta_3 = 0$ is given by
$$\text{LRT} = \frac{\sup_{\theta\in\mathbb R^2\times\{0\}}\mathcal L(\theta)}{\sup_{\theta\in\mathbb R^3}\mathcal L(\theta)},$$
which is approximately $\chi^2$-distributed with 1 degree of freedom.
In fact, this test does not test whether the improvement in the goodness of fit in terms of the $R^2$ is significant, however, it tests whether the improvement in terms of the likelihood, which could be seen as some measure of goodness of fit (see, for instance, the Nagelkerke $R^2$, McFadden $R^2$, Cox and Snell's $R^2$), is statistically significant.
The $R^2$, in the above example, has a Beta distribution with shape parameters $1$ and $(n - 3)/2$. However, the linear combination of two dependent (!)  Beta distributed random variables is, I guess, no standard distribution. I don't think that following this track will help you achieving what you want. This is why I suggested to test the difference in likelihoods using the likelihood ratio test.
Comment: With all due respect, this neither answers the question I have asked, nor is sufficiently detailed for me to understand what you are suggesting. Consider making a comment instead or rewriting into a full answer
Comment: I am sorry for that. I edited the answer
Comment: Thanks for writing up the post. Now I understand your suggestion
Comment: The problem with Likelihood Ratio is that it requires an assumption about the noise model. I do not know whether the residual distribution is gaussian, and even if it happens to be gaussian, I have no prior knowledge of the variance of residual terms. I work in biology, where effects in question are frequently a combination of known effects and unknown effects that are impossible to control experimentally. The goal is to find metrics that can guide into the right direction without having to make too many prior assumptions
Comment: Actually, it is easy to show that log-likelihood ratio for gaussian noise is just the difference between $R^2$ of the two models, which is exactly the metric that I propose.
Comment: This actually answers my question: I can use $\chi^2_1$ test if I can guarantee residuals to be gaussian, and I (probably) have to resort to permutation-testing if residuals cannot be guaranteed to be normal

ANOVA vs R-Squared (Explained Variance)


In which order are the conditions tested in a Sqlite query?

Question: Let's say I have 1,000,000 rows in a table, and I do this query (using sqlite3 module in Python):
c.execute('SELECT * FROM mytable WHERE editdist3(description, "hello world") < 500
                                   AND price > 3707 AND price < 3800' )

Obviously computing the editdist3 edit distance / Levenshtein distance on 1 million rows will be very costly for CPU!
Question: how to make sure that the filtering condition 3707 < price < 3800 is applied before computing the Levenshtein distance?
Should I do two queries: a primary query to pre-filter this 3707 < price < 3800 condition, and then only, a second query computing editdist3 on the (a lot less) remaining rows? Or is there a way to do this in one single query?
Note: this question is a poor-wording (my bad!) version of this better question.
Answer: If you have index on ID, sql engine will use it to find the description and then apply the distance to only those elements.
The best should be create the index with (ID, description)
create index idx on mytable (ID, description);

You can test it by checking the explain plan.
Comment: I think my question was too specific with "ID" column @JuanCarlosOropeza, I modified to a more general condition `3707 < x < 3800`  (here I gave example with `price` but it could be anything else).
Comment: My answer would be the same, create an index for price. And again you can check using explain analyze
Comment: What do you mean by index on price? Could you give a code example showing your solution with index? Thank you in advance.
Comment: [SQL As Understood By SQLite - CREATE INDEX](https://www.sqlite.org/lang_createindex.html)
Comment: you mean dont know how create index? check edit and this one http://mysql.rjweb.org/doc.php/index_cookbook_mysql


Expectancy of a RV

Question: I have a question. I know that a RV $X$ has the hyperbolic secant distribution (with density function $f_X(x)=\frac{2}{\pi}\frac{1}{e^x+e^{-x}}$). I have a few questions about calculating the expectancy (mean value):
Is $\int^{\infty}_{0}f_{X}(x)dx=\mathbb{E}[X\space{1}_{\{x>0\}}]$? (where $1$ indicates the indicator function)
And why does the fact that $X$ and $-X$ has the same distribution imply that $2\mathbb{E}[X\space{1}_{\{x>0\}}]=\mathbb{E}[X\space{1}_{\{x>0\}}]+\mathbb{E}[-X\space{1}_{\{x<0\}}]=\mathbb{E}[|X|]$?
Thanks in advance
Answer:
No, it should be $\int_0^\infty x f_X(x) \, dx = E[X 1_{X > 0}]$.
If $X$ has the same distribution as $-X$, then $X1_{X > 0}$ has the same distribution as $(-X) 1_{(-X) > 0} = -X1_{X < 0}$, so $E[X1_{X>0}] = E[-X1_{X<0}]$
In general for any real number $x$ we have $|x| = x 1_{x>0} + (-x)1_{x<0}$, so  $E[|X|] = E[X1_{X > 0} + (-X) 1_{X < 0}] = E[X 1_{X > 0}] + E[-X1_{X<0}]$
Answer: Firstly, by definition
$$
\mathbb E(\phi(X)) = \int_{\mathbb R} f_X(t) \phi(t) dt.
$$
(This can be proven from the somehow more basic definition $\mathbb E(x) =\int x f_X(x)$.) So if $\phi(x) =x 1_{\mathbb R^+},$ then
$$
\mathbb E(\phi(X)) = \int_{\mathbb R} f_X(t) \phi(t) dt=\int_{\mathbb R^+} f_X(t)tdt.
$$
For the second question, let me add some details: since $X$ and $-X$ have the same distribution, we have
$$
\mathbb E(\phi(X)) = \mathbb E (\phi(-X)).
$$
Now, what is $\mathbb E (\phi(-X))?$ Well, $\phi(-X) = (-x)1_{\mathbb R^+}(-x) = -x1_{\mathbb R^-} (x)$, which is what you have in the question.

Expectancy of a RV


Node one async call slows another in parallel

Question: For over a year we've seen interesting patterns that don't always rear themselves but on occasion repeat and we've never been able to figure out why and I'm hoping someone can make sense of it. It may be our approach, it may be the environment (node 8.x & koa), it may be a number of things.
We make two async calls in parallel to our dependencies using the request-promise module.
Simplified code of a single api dependency:
const httpRequest = require("request-promise");

module.exports = function (url) {
    const requestOptions = {
        uri: ...,
        json: true,
        resolveWithFullResponse: true
    }

    return httpRequest(requestOptions)
        .then(response => {
            status = response.statusCode;
            tmDiff = moment().diff(tmStart);

            return createResponseObject({
                status,
                payload: response.body,
            })
        })
        .catch(err => { .... };
    });

};

Parallel calls:
const apiResponses = yield {
    api1: foo(ctx, route),
    api2: bar(ctx)
};

Yet we've seen situations in our response time charts where if 1 is slow, latency seems to follow the other separate service. It doesn't matter what services they are, the pattern has been noticed across > 5 services that may be called in parallel. Does anyone have any ideas what could be causing the supposed latency?
Comment: If the latency is caused by a temporarily slowed network connection, then it would be logical to expect both parallel requests to feel that same effect.  `ping` or `tracert` during the slowdown might give you useful diagnostics to see if it's a general transport issue.  If your node.js server (which runs Javascript single threaded) was momentarily busy doing something else with the CPU (serving another request, garbage collecting, etc...), then that would affect the apparent responsiveness of API calls just because it took a moment for node.js to get around to processing the waiting responses.
Comment: While this seems logical, you're suggesting that one monitors the graphs 24/7 until that brief moment of a latency spike to SSH into a remote box and conduct a tracert. Our graphs also show no CPU spike at this time. Not to say your response isn't a potential answer, but is there not a more automated way to make such determinations?
Thank you.
Comment: There are tools that monitor the responsiveness of your own http server on a continual basis (you can set whatever monitoring interval you want).  If you have a CPU-hog somewhere, those tools would show a similar lag in responsiveness.  There are also tools that monitor the health of your network connection which would also show a connectivity glitch.  These are the types of monitoring tools that people whose job it is to maintain a healthy server farm might use.  I don't have a name handy for either one, but you can presumably find such tools by searching.
Comment: Thank you. I'd mark your first comment as the answer but it wasn't posted as such for me to mark as the answer.
Comment: OK, I put my comments into an answer.
Answer: If the latency is caused by a temporarily slowed network connection, then it would be logical to expect both parallel requests to feel that same effect. ping or tracert during the slowdown might give you useful diagnostics to see if it's a general transport issue. If your node.js server (which runs Javascript single threaded) was momentarily busy doing something else with the CPU (serving another request, garbage collecting, etc...), then that would affect the apparent responsiveness of API calls just because it took a moment for node.js to get around to processing the waiting responses.
There are tools that monitor the responsiveness of your own http server on a continual basis (you can set whatever monitoring interval you want). If you have a CPU-hog somewhere, those tools would show a similar lag in responsiveness. There are also tools that monitor the health of your network connection which would also show a connectivity glitch. These are the types of monitoring tools that people whose job it is to maintain a healthy server farm might use. I don't have a name handy for either one, but you can presumably find such tools by searching.