Balancing audio frequency bands and instrument volumes for consistent audio quality in highly variable environments, software, and hardware?

Question: I use GarageBand to write music but in theory this question could apply to any composition and recording workflow.
A problem I often run into is that different audio players seem to have their own "flavor" of output. For example, whatever I write obviously sounds like what I intended in GarageBand. When I export it to MP3 format and listen in QuickTime Player, same result.
When I import the same file in iTunes, it seems to apply some sort of treble booster that makes the whole track sound tinny. Ditto on my iPhone (where the default music player is basically a subset of iTunes). In practice, I "fix" this with a bass booster or treble reducer in the Music -> Equalizer menu. I'd expect the default media players on Windows, Android, and Linux to have their own specific quirks.
My current approach is to test on a small selection of hardware and software that I have at home, although obviously it's impossible for me to test on every possible piece of equipment my music could potentially run on.
It gets even more difficult if I have to account for different types of headphones and audio equipment. Some headphones and speakers are well-balanced, some might be bass-boosted, and others might have low-quality tinny audio. The amount and type of padding between the ears and speaker holes on a pair of headphones will obviously cause some sort of frequency-based distortion or bias. I'd like to be able to hear bass frequencies on my music loud and clear on my headphones without throwing an equalizer into the mix. I'd also like other people, some of whom might use heavy bass-enhanced or noise-canceling headphones, to be able to listen to the same music without voiding the warranty on their eardrums in the 20-100Hz range. It's not as simple as "make all the bass instruments louder".
Different headphones have different audio leakage profiles as well - for example, the ones I use leak a lot of noise through from the outside. I consider this to be a feature rather than a disadvantage since it keeps me aware of my surroundings. Meanwhile, someone else might be using a noise-cancelling headset that blocks everything except the music.
On top of that, I'd like to build my music so it can be reasonably listened to in highly variable environments. Indoor or outdoor, and also on mass transit with a lot of possible engine noise polluting the lower audible frequency spectrum. Do NOT assume the listener is using a noise-canceler. My experience so far is that a bass booster or treble reducer is a must for listening on a noisy train, for example. Any way to fix this without causing the reverse problem when there isn't a noisy train engine humming loudly in the background?
The main question: Is there an industry-standard way to calculate frequency distribution and instrument volumes for the optimized audio quality across a wide range of hardware, software, and environments? Preferably a workflow built for a DAW but anything reasonable should be adaptable.
EDIT: Writing music is a hobby for me. My production budget is only the time I spend on it, the cost of my computer and associated hardware (to be fair, I use that computer for a ton of other stuff; GarageBand comes free with a Mac), and the electricity it runs on. I definitely don't have the budget allocation to bring a recording engineer on board, except as a volunteer.
Comment: You're talking about mixing and mastering, both of which are non-trivial arts really. :) It's HARD and it takes SKILL and EXPERIENCE. :D I'd be greatly surprised if someone could tell you how to do all that in a Stack Exchange answer, but let's see!
Comment: Interesting question! Looking forward to see the answers! Maybe you'll get more stuff on [sound design](https://sound.stackexchange.com/)…
Comment: That's another good question!
Comment: @user1258361: "I didn't know that was a thing on Stack Exchange. Did it just pop up recently?" – The [sounddesign.se] Stack Exchange started in the Spring of 2010.
Comment: I didn't know that was a thing on Stack Exchange. Did it just pop up recently?
Answer: iTunes has a feature called "Sound Check" which scans all newly imported music and does some auto-EQ and volume adjustment so all the songs in your library play at roughly the same volume and have similar EQ profiles. This may be your issue. If you don't want it to do that, it's a check box in the app preferences.
Beyond that, I agree with the other answers regarding a good studio monitor pair or other high quality flat response speakers to ensure that what you hear matches what you're recording.
You may also want to check your iTunes preferences to make sure that it is not automatically converting imported audio to a default lower quality MP3 to save space.
Comment: I already use Sound Check for exporting my projects. I've also tested using varying quality of MP3 output and I hardly notice any difference except if I'm using the 64 kbps encoding. A bit of advice: the 64 kbps encoding produces horrid results, only reason to use it is when you need to save every last bit of disk space.
Comment: This is definitely the problem for the itunes case: you are allowing itunes to "remaster" the track. The tinny nature is almost assuredly a bitrate and quality change (for the worse) where the high frequencies start to "sparkle." It is editing the files.
Answer:
The main question: Is there an industry-standard way to calculate frequency distribution and instrument volumes for the optimized audio quality across a wide range of hardware, software, and environments?

Yes: hire capable and experienced mixing and mastering engineers. That's the industry standard.
How do those engineers ensure mixes will translate well? Mainly, they know their business. They have worked for years developing their ears and their knowledge of how a mix should sound if it's going to translate well. And they also know how to test their mixes and masters in their custom listening environments and a limited number of other environments to correlate the data gathered by their ears and make the necessary adjustments.
So far, no algorithm, software, hardware, or simple process has been invented that can replace the work of good engineers.
As in pretty much every area of human endeavor, the hobbyist should not expect to operate at the same level as the professional. Having been a semi-pro/low level pro engineer for several years in the past, I would expect to take ten years or so of daily part time recording, mixing, and mastering to reach the level I was at in my prime, when I was able to get hired locally to record and mix. If you primarily prefer to be a musician and/or composer, I suggest that you focus your time on that and do your best on the engineering side while looking for people who either value your music highly enough to pay for professional engineering or who are engineers themselves and are happy to work with you on spec.
Comment: @user1258361, you are misquoting Todd's answer.  He stated that so far none has been invented.  Secondly, there does exist a "standard" way to cal f-dist, and dB blah...  That uses the GOF-FFT.  But a human in the loop needs to make a choice as to what is important in that spectrum.  Perhaps using samples of known sounds or physics based models could be used with AI (e.g. ANN) to make the choices but this has not occurred yet.  And this would need training which requires a human to make a choice.
Comment: @user1258361 I would tend to agree with except that, IMO, mixing and mastering are part of the art of music making and, in certain bands the sound engineer, plays a significant role in the final production, which would be different with another SE. An algorithm might product good mixing, but everything might end up too "standard"…
Comment: Kind of an echo or expansion on what I think @ggcg is getting at: mixing and mastering are (partly to mostly) *artistic* processes, by which I mean trying to remove the human element would not lead to the most pleasing results. They are not mechanical process (despite the use of the word "engineering" which I think was an unfortunate choice made long ago). Just as we don't currently have algorithms or mechanical devices that can write songs or play emotional guitar solos, for the same reasons we don't have algorithms or mechanical devices that can produce effective mixes or masters.
Comment: I tend to disagree with your assessment that "the algorithm" can't replace mixing and mastering engineers; the only thing preventing that is a lack of research and application. I'm sure "the algorithm" could produce much better (and automated) mixing if Big Tech productively dumped a few billion bucks into the research effort.
Comment: @user1258361, All that is needed to screw that up is for different groups of people to make different choices about what is an important signal feature.
Comment: Update: It definitely is something to do with the headphones. I recently replaced the padding on them because the old ones were wearing through and now everything sounds tinny. I also tested equalizer off vs. treble reducer on a variety of songs made by professional musicians and about a third of those songs got absolutely wrecked through a tin can on these headphones with the equalizer off. Guess it isn't a problem with my workflow if professional productions are having the same issue.
Answer:
If you export sound from a program and it sounds clearly differently in another program then there's something wrong. Perhaps you have an equalizer running all the time in one of your programs? Of course there is some flavour added by the MP3 format itself, but at high bitrates it should be rather small. If you observe a big effect, then there is an issue elsewhere. Normally it shouldn't happen.
An "industry standard" for listening in audio production is called studio monitors. These are speakers intending to have flat frequency response and reproduce sound neutrally, without coloring the sound, as consumer equipment typically does. But to make it work you need you need to make your software work properly as well (point 1.)
It is a good exercise to check how your mix will sound on various consumer type of equipment, headphones, laptop speakers, mono radio, car speakers – but this can only supplement listening on good monitors.
Comment: I'm almost convinced that different MP3 player apps on various platforms include their own flavor of subtle post-processing. That's why I test on a small selection of platforms that I have at home for purposes of "cross-platform compatibility".
Answer: Know your equipment.
Buy a decent set of reference monitors, and learn how they sound in relation to other environments and equipment. Use them consistently and regularly, so you are aware of how they will work regarding the music you record.
Remember that people who like a lot of bass will already have the bass eq turned up to silly levels on their equipment, so you don't need to worry about that. If it sounds good on reference monitors, in-ear headphones and a car stereo, you're as close as you'll get without paying an experienced mix engineer to redo everything.
Answer: Apparently, AI-powered auto-mixing has been a thing for a few years now. As you probably expected, it's a tradeoff. If you "outsource to the robot", you save money while giving up customization and flexibility you would otherwise get with a professional recording engineer. I don't expect "the robot" to displace professional recording engineers out of their jobs anytime soon, but auto-mixing definitely provides a "much better than nothing" option for independent musicians on an extremely low budget.
https://www.theverge.com/2019/1/30/18201163/ai-mastering-engineers-algorithm-replace-human-music-production
Landr is one of the most popular such services, hosted as a web service. You can upload the song you want mastered, let Landr's algorithm analyze it, choose between 3 options for how strongly you want effects applied, and then export the result. It's a catchall approach, and it's not exactly flexible.
Software company iZotope approached AI with an educational lens. The company already makes a popular suite of plug-ins called Ozone, and added in an intelligent "Master Assistant" in 2017. The assistant doesn't do all the work for you. Instead, it gives you a starting point you can tweak. That way, producers can make informed decisions based upon the choices the AI has made.


Searching for time series inside another time series

Question: I have a time series "A" and another one "B". I would like to find occurrences of "B" inside "A". Typically, "A" is much bigger (magnitude: millions of points) than "B" (magnitude: hundreds of points)

I don't necessarily want exact matching (obviously, since the values we deal are real numbers not, say, text) The values are preferably close, though.
Trends/shapes are more important than exact time matching (i.e. somehow Dynamic Time Warping could be useful)
With regards to the first two points, what happens if I don't care about the absolute values at all but just the shape (maybe taking a log of the values?)
What happens if I would like to scale the number of points to billions from millions? How to scale the system? How do I index a time series? (maybe use Mahout, Lucene, somehow?)

What's inside above parentheses are the results of my short research, not necessarilly correct. All in all, what kind of an approach should I take? What are your general suggestions/tips? I'd like to develop a system, so any practical examples/suggestions are even more welcome.
Thanks!
Comment: Can you provide a simple example w/ data for time-series A & B that would demonstrate what you mean?
Comment: A clear and quantitative description of what it would mean for $A$ to be "close" to a subsequence of $B$ is essential. For some meanings--for instance, where "close" means strongly correlated--there are extremely efficient and effective solutions. For other meanings the problem will be impossible to solve without an exhaustive search.
Comment: That does not clarify what it means for two *time series* of values to be "close".  Although the question refers to "the shape," this could mean many things, especially in light of the mysterious reference to "absolute values" (which, although that has a well-defined mathematical meaning, does not seem to be what you wanted to say).
Comment: Let's assume we're talking about temperature values: from -100 to 1000 celcius, real numbers.
Answer: Such a shorter series is known as motif. Think of music: a motif is a subsequence that appears at multiple times, and even in different compositions.
With Time Series Motifs you will find plenty of literature.


Can't save location when posting node through services module

Question: I am using services module to post nodes using rest/json. The nodes also contain images and location. So far I can upload the image and save the node, but the location is not saved. I receive no errors in logs.
Any ideas? Is there something I have to add,maybe permissions, in order to be saved?
This is the json that creates the article but doens not save the location.
<code>{
    "title": "some title",
    "type": "article",
    "language": "el",
    "body" : {
          "und": [
              {
                  "value": "some text"
              } ]
    },
    "field_tags": { "und": "9" },
    "field_image": {
        "und":
          [{
            "fid": "1364"
          }]

    },
    "locations":{
        "und":[
                {
                  "country":"gr",
                  "latitude":"55.7046601",
                  "longitude":"13.191007300000024",
                  "name":"TEST LOCATION",
                }
            ]
    }

}
</code>
Answer: How are you storing the location? Node location? Location CCK field?
take a look at:
https://www.drupal.org/node/906968
If you are using a CCK location field you should disable node location module.

Or if you aren't you could switch to a location field.

As a last option you could put latitude and longitude in their own text fields (like field_latitude and field_longitude) since gmaps module can take lat and lng from text fields.

you can use CCK location which allows you a geolocation field, it
  comes with location module:
https://www.drupal.org/project/location
Comment: I use drupal 7 and when I enabled the location/gmap module I see a default 'locations' field in the node. I also tried to create a new 'field_location' but no luck so far.
The links that you sent is about drupal 6.
Comment: Yep, but I found myself with the same problem in D7.  If you want so use gmap and need a location field you can try: Location CCK which allows you to use a geolocation field and select this field as a coordinates source when you create a gmap view.
Comment: There was some progress. I disabled 'node locations' and I posted to the new field_location the longtitude and latitude. But in table 'location' i see that source is 0 and the longtitude and latitude are both 0.000000.
Comment: Make sure you pass the values in the right format  with decimals after a dot
Comment: I sent the following but still i get 0.00000...
"field_location":{
       "und":[
            {
                  "country":"gr",
                  "latitude":"55.7046601",
                  "longitude":"13.7046601",
                  "name":"TEST LOCATION",
                  "city":"Tripoli"
                }
         ]
    }
Comment: Also, locations may be trying to geocode the coordinates using the address so you may try to pass only the coordinates
Comment: If I only pass coordinates then nothing is saved. I just sent "field_location":{
       "und":[
            {
                  "latitude":"55.7046601",
                  "longitude":"13.7046601"
                }
         ]
    }, with no luck.
Comment: did you configure the cck location field properly? You can configure it to accept just the coordinates and make all other 'subfields' optionals.
Comment: The cck location field is set to allow only location name, country and coordinate chooser. Same in display settings. Is there a problem with coordinate chooser?
Comment: I just uploaded 2 screenshots in case you see something wrong. Thank you for you time.
https://dl.dropboxusercontent.com/u/2534039/Screenshot_2.png
https://dl.dropboxusercontent.com/u/2534039/Screenshot_3.png
Comment: It's more or less how I have It. If you put the latitude and longitude from the node itself (not programmatically) ¿do you get the same error? I had a lot of trouble with coordinates too.
Comment: If I put long/lat from the broswer, either by coordinate chooser, either by by setting them into the form, they are saved without any errors. When posting from rest/json and they are not saved, there are no errors in logs.
Comment: In content type settings there is on default locations field and one field_location that i created. Is there a chance for a conflict? How do I remove the default?
https://dl.dropboxusercontent.com/u/2534039/Screenshot_1.png
Comment: the default should not affect the field. I you can't save the values via json may be there's a problem on how ypou're passing the values. I use the same field type and I only fill latitude and longitude programmatically without any problem. with entity api i pass something like '$entity->$clave->set(array(
            'latitude' => $campo['value']['latitude'],
            'longitude' => $campo['value']['longitude'],
            //other elements added here (province, postal_code, country [es, en,...],additional, ...)
          ));'
Comment: I didn't add any php code so far. I thought it would work with default services. Is there a guide, do I need a custom module to do in order to save the location?
Comment: No, It's just an example of the parameters i pass, which are only latitude and longitude. You can try to put them in separate text field as i suggested in an edit. As a last option It can save you some time.
Comment: Ι don't understant where to add the sample code. In node-article.tpl.php? Or other file?
Comment: It's a little complicated to explain in the comments, and this answer would be more related on how to programmatically create nodes rather than with your actual problem.  If you feel comfortable with php coding you could use this solution, otherwise I'd suggest you to put latitude and longitude in separate text fields until a better solution is found. If you want to know how to create nodes using php I can give you some directions.
Comment: The devel module shows the following structure:
https://dl.dropboxusercontent.com/u/2534039/devel.jpg

Do you think the problem is with json or I need to add php code to handle longtitude / latitude?


linked list implementation fail without malloc

Question: The first example is the example of a linked list without dynamic memory allocation. But in this example when we run it, it will only display 10 and then will provide runtime error. Please help me finding what is wrong.
#include<bits/stdc++.h>

using namespace std;

struct Node{
    int data;
    struct Node* next;
}Node;

struct Node* head;

void Insert(int x)
{
    struct Node* temp;
    temp=head;

    struct Node* newnode;
    newnode->data=x;
    newnode->next=NULL;

    if(head==NULL)
    {
        head=newnode;
        return;
    }

    while(temp->next!=NULL)
    {
        temp=temp->next;
    }

    temp->next=newnode;

}

void print()
{
    struct Node* temp;
    temp=head;

    while(temp!=NULL)
    {
        cout<<temp->data<<" ";
        temp=temp->next;
    }
    cout<<endl;
}

int main()
{
    head=NULL;
    Insert(10);
    Insert(20);
    Insert(30);
    print();
}

In the below example, I have just used malloc to provide dynamic memory to the variables. It works perfectly as expected. Why is there such a difference between the two?
#include<bits/stdc++.h>

using namespace std;

struct Node{
    int data;
    struct Node* next;
}Node;

struct Node* head;

void Insert(int x)
{
    struct Node* temp=(struct Node*)malloc(sizeof(struct Node));
    temp=head;

    struct Node* newnode=(struct Node*)malloc(sizeof(struct Node));
    newnode->data=x;
    newnode->next=NULL;
    cout<<"Reached here"<<endl;

    if(head==NULL)
    {
        head=newnode;
        return;
    }

    while(temp->next!=NULL)
    {
        temp=temp->next;
    }

    temp->next=newnode;

}

void print()
{
    struct Node* temp=(struct Node*)malloc(sizeof(struct Node));
    temp=head;

    while(temp!=NULL)
    {
        cout<<temp->data<<" ";
        temp=temp->next;
    }
    cout<<endl;
}

int main()
{
    head=NULL;
    Insert(10);
    cout<<"Inserted 10"<<endl;
    Insert(20);
    cout<<"2nd"<<endl;
    Insert(30);
    cout<<"3rd"<<endl;
    print();
}

Comment: you are creating a pointer and trying to access its fields without initializing it. what else did you expect?
Comment: Check your reading materials. It looks like they're teaching an uncomfortable combination of C and C++ that you'll have to train yourself out of. I'm not hopping on the"Just use `std::list` bandwagon here (but seriously, use `std::list` when it comes time for real work in C++), but each language does things differently and you should take advantage of each one's strengths.
Comment: [Why should I not `#include `?](https://stackoverflow.com/questions/31816095/why-should-i-not-include-bits-stdc-h), [Why is `using namespace std;` considered bad practice?](https://stackoverflow.com/questions/1452721/why-is-using-namespace-std-considered-bad-practice)
Answer: In your example without malloc this is simply illegal code:
struct Node* newnode;
newnode->data=x;
newnode->next=NULL;

newnode is not pointing to any legal memory so writing to memory using newnode is undefined behavior.
If you want a linked list without using malloc you must reserve memory for the nodes up front. For instance, you can make a pool of 1000 nodes at program start and then use these node during program execution.
Like:
 static struct node nodePool[1000];

 struct node* getNode()
 {
     // Add code to find an unused node and return a pointer to the node
     // This will require some extra data object to track which nodes are free
 }

 void freeNode(struct node* p)
 {
     // Add code to mark the node as unused
     // This will require some extra data object to track which nodes are free
 }

void Insert(int x)
{
    ...

    struct Node* newnode = getNode();
    if (newnode == NULL)
    {
        // No more nodes
        add error handling
        return something or exit(1);
    }
    newnode->data=x;
    newnode->next=NULL;

    ...
}

Note: Using a flat array for a node-pool is doable but it's not good for performance. Get/fre node will be O(n) which is bad for program performance. Programs using pools have more advanced pool implementations that allows for O(1) operations. One (pretty simple) method is to implement the pools control-data as a linked list with both head and tail pointers. That will allow for O(1) complexicity for both getNode and freeNode.
Note: I just noticed that the question is marked with both C++ and C. This answer was written with C in mind. For C++ code there is no (aka seldom any) reason to implement your own linked list - simply use std::list
Answer: Both your examples are wrong but for different reasons. Pointer variables need to be given values, just like regular variables. So your first example is wrong because the pointer variable newnode is uninitialised. See the coments I've added to your code
struct Node* temp; // temp has no value
temp=head;         // now temp is given a value

struct Node* newnode; // newnode has no value
newnode->data=x;      // ERROR, newnode is used before it's been given a value
newnode->next=NULL;

You wouldn't use a regular variable before you'd given it a value. It's no different with pointer variables.
Your second example has the opposite error, you give the temp pointer a value twice! Again see the comments added
struct Node* temp=(struct Node*)malloc(sizeof(struct Node)); // temp is given a value
temp=head;                         // now temp is given a different value,
                                   // the first value is discarded

struct Node* newnode=(struct Node*)malloc(sizeof(struct Node)); // newnode is given a value
newnode->data=x;                                       // newnode is being used correctly
newnode->next=NULL;

You can give a pointer a different value in two ways, you can create a new node for it to point to, that's what temp=(struct Node*)malloc(sizeof(struct Node)); does, or you can make it point at an already existing node, that's what temp=head; does. There's not much point in doing both. In your second example the node you create with temp=(struct Node*)malloc(sizeof(struct Node)); is never used, it is created and then discarded when you make temp point at the head node instead. This is called a memory leak. If the leak gets large enough your program will run out of memory. At best it's wasteful.
Again think about how you would use regular variables, you'd never write code like this int x = 10; x = 20; it doesn't make sense to give a variable one value, and then on the very next line throw that value away and give the variable a different value. But that's exactly what you did with the temp variable. Again pointers aren't special, the same rules that apply to regular variables also apply to
pointer variables.


how can so called "unallocated" IP adresses appear in web logs?

Question: My server is currently experiencing some web app attack to which miscellaneous IPv6 addresses are participating.
Several of these addresses are said belonging to so called unallocated networks
None of these addresses are currently being report on the major lists of dirty IPs
As an example of these : 2002:d15a:e1c3::d15a:e1c3
The remarks section of the whois notes : This object is here for Database consistency.
How can this be possible ? Is it worth blacklisting ?
Answer: This is a deprecated 6to4 address, as is any address beginning with 2002:.
6to4 is an IPv6 transition technology that creates an IPv6 tunnel. In its public anycast form, the 6to4 IPv6 address has a 1:1 correspondence with the endpoint's IPv4 address. In this case, the 6to4 address 2002:d15a:e1c3::d15a:e1c3 corresponds to 188.8.131.52.
That machine didn't use native IPv6 connectivity, but instead used a 6to4 tunnel through a public 6to4 anycast relay, possibly to attempt to obscure its origin. It could possibly also be misconfiguration; some operating systems (such as Windows) enable public 6to4 by default.
Comment: That does make sense. Among other IPv4 addresses involved in the attack, several belong to the WOW network. Thanks Michael for that piece of valuable information.
Comment: I can't wait for the public 6to4 relays to finally shut down. In the meantime, web sites can generally get away with blocking `2002::/16` as actual web browsers will retry on IPv4 (due to Happy Eyeballs).
Comment: The RFC that you link explicitly says that 2002::/16 is **not** deprecated. Only 126.96.36.199 is deprecated. And you cannot tell from the log if the client actually used a public 6to4 relay. It's likely that the client used a public 6to4 relay, but there are other ways the packets could have been delivered from the client to the server.
Comment: @kasperd The prefix itself is not deprecated, just the service that uses it. As a practical matter this makes no difference in this situation. As the RFC also mentions, private 6to4 is called 6rd and uses ISP-owned prefixes rather than 2002::/16. I suppose it's possible someone put up a public 6to4 anycast relay on some address other than 22.214.171.124, but I've never heard of it.
Comment: @MichaelHampton Maybe you've never heard of it, but that is actually how 6to4 was defined when the RFC was first published. Though at the time the addresses were more likely unicast than anycast. And there was even a site with [a list](https://web.archive.org/web/20010604065249/http://www.kfu.com:80/~nsayer/6to4/) of relays to choose from. The use of the 184.108.40.206 anycast address was a later addition, and it's only that later addition to 6to4 which has been deprecated.
Comment: @kasperd Yeah, I missed that. I only started using IPv6 in 2006, so I did miss a bunch of the very early stuff if it didn't make it into an RFC (and maybe some stuff that did).


Can I remove some of the chords from a song?

Question: I have a song and found out the chords using RiffStation. The software detects more than 2 or 3 chords in some measures which means I should rapidly change chords while playing.
My question is can I remove some short chords and use only 1 or 2 chord in the measures that I have more than 2? Does it hurt song quality?
Comment: Do you have any theoretical or experimental reason for your answer? I think human ear can't detect differences between 2 chords or 4 chords that played in a couple of seconds. I'm not sure though
Comment: So you are saying as long as  I feel the song sounds good it is okay, right?
Comment: Anything is "ok". There are no "sounds ok" police. No one is going to come to you and fine you for doing what you want. Do what you want. If you leave out some chords, it **will** *change* the song, but that's ok. People have been changing songs as long as there have been songs. When I was teaching I would teach the youngest students simplified versions of songs that left out some chords and made other chords easier to play. It still has the same vibe as the full song, just not as much character.
Comment: "I think human ear can't detect differences between 2 chords or 4 chords that played in a couple of seconds" There is plenty of classical piano music where the chords change a lot faster than once or twice per second! Of course it's easier to *play* fast chord changes on piano than on guitar, but there is no problem *hearing* the changes.
Comment: You are Right. I should mention normal people. Obviously musicians have amazing ears :)
Answer: The question is, whether it is acceptable to leave some chords out of a piece of music when you play it. The answer will depend on the context. For a piece of classical music, the answer may be no. So let's focus on other contexts.
I would start by saying that it is best to transcribe tunes from recordings yourself, rather than to rely on others to do this work. So many transcriptions that you find in print or online contain errors. And different transcriptions may be based on different recordings; who is to say which is authoritative? In any case, you should compare the transcription with at least one recording to check its accuracy.
Setting this aside, and focusing on chords, suppose that you have a set of chord changes that is too dense to comfortably play. You might work to achieve a higher level of technical proficiency, but you might also consider leaving some chords out of your performance. This is fine, so long as what you play sounds right. This is the golden rule of music: no choice is bad if it sounds right. Of course, this allows for much subjectivity, but it does not let players off the hook for making poor musical choices. Part of growing as a musician is developing an ability to be self-critical (in a positive sense).
Note that it is not at all uncommon for chord progressions to move through four chords in one second. For a concrete example, consider the Duke Ellington tune Don't Get Around Much Anymore. This song is often played with four chords in one bar:

CMaj7 - Dmin7 - D♯° - CMaj/E.

I don't know what the original changes were, but I suspect that the variations on the above that you hear, and see in transcriptions, are just substitutions made to increase harmonic motion. You can see that the main harmony here is CMaj, with the Dmin7 and D♯° connecting two CMaj chords. You could easily leave these two chords out and just play on CMaj7 for the whole bar. It would sound different, with less harmonic activity, but not necessarily bad or wrong.
Apart from this, you may want to play with fewer chords because you think the original changes are too busy, or because you want a more relaxed sound. Or you may want more chords. All such decisions are up to the player (or the bandleader, or the conductor, or maybe someone else... let's just say the player).
Working within limitations is a great way to improve as a musician. By transposing parts played on other instruments, for example, you will be forced to make choices about what to play or not play. A guitarist might hear something in a piano part that is impossible to play on the fretboard; how to get the essential sound? Or both pianist and guitarist might hear something in a horn section not reproducible on either instrument. The challenge of interpreting what you can't play leads to growth.
Comment: excellent detailed and very accurate answer.
Comment: The process of working out guitar chords for a song arranged on piano is quite educational.  I've done it a few times.  It's quite a project and leads to learning some interesting less common chords and voicings.
Answer: I checked reviews for the RiffStation software and the consensus seems to be that the chord detection ability is only useful for songs with very basic major or minor chords using the most common voicings with no inversions.  In other words, more often than not, the chords it suggests are not all correct.  So I would avoid using the program as a primary means of determining the chords to most songs.
But to your actual question

can I remove some short chords and use only 1 or 2 chord in the measures that I have more than 2? Does it hurt song quality?

The answer is - it depends.
If you are unable to play the actual full chords in the quick succession that are called for in a particular arrangement of a song, you can sometimes get away with omitting a passing chord - particularly if that passing chord is not a prominent feature of the song.  Just because a particular chord would work with the underlying melody in a given measure, does not necessarily make it essential.   If the chord you are holding (instead of playing the indicated transition chord) still works with the underlying melody - you can often skip the passing chord altogether.
Popular recorded versions of most songs have a certain sound, feel, and vibe that partially arises out of the chord progression used.  Of course other elements such as rhythm, tempo, time signature, and the instruments used in the mix will also contribute to the unique signature of a given version of a given song.
As a musician who plays primarily for the fun of entertaining others - when I learn to cover another artist's song using the guitar as my accompaniment instrument, I attempt to replicate to the extent I am capable of (using the guitar) - a musical rendition that captures some semblance of the sound and feel and mood of the original.  I want my cover to sound somewhat authentic and recognizable.
But there are songs that have fast moving chord changes that are difficult for me to pull off in real time - especially in front of a live audience.  So what I often do in that situation, is simply substitute the root base note for a quick 2nd or 3rd chord in a measure,  as I transition to the final chord of that measure.  Played with proper timing and rhythm, this method often lends a surprising degree of authenticity to the passage.
Other songs might have fast changes that include chords that are difficult to transition between with the speed demanded by the song.  Sometimes instead of using bass notes, I will use a substitute chord or alternate voicing of the chord used in the song.  Or sometime a power chord (using two of the fatter strings) will make a good substitute for a full chord.
Ultimately you want to create a rendition that sounds recognizable and maintains most of the signature distinctive elements of the song that define that song.  Without the full band using the same instruments with the same talent as the original band, no matter what you play - your are not going to sound like the original. So how you pull off the illusion is not as important as how the overall performance sounds to the audience.
If you can't play it exactly like the original artist or band - use whatever tricks, shortcuts, or substitutes make your performance sound as close as possible to the version you are attempting to emulate (in accordance with your ability). Don't allow an unrealistic demand for perfection keep you from playing the music you enjoy!
Have fun.
Answer: Here's a simple example. Apparently the original Happy Birthday to You was based on the melody of Good Morning to All. If you look at the third last bar here: https://en.wikipedia.org/wiki/Happy_Birthday_to_You  you will notice two chords that are not always played at birthday parties. In fact most versions of Happy Birthday to You are probably played simply with chords I, IV and V without spoiling anybody's party. Another example, in reverse, would be Bobby McFerrin's version of Van Morrison's Moondance. It has more chords than the original but  it sends the Seduct-O-Meter (technical term) off the scale and makes Van's version sound like bubblegum music. There is no doubt that the original is a successful piece of music, but if it was released after McFerrin's version would people be complaining, 'oh he's left chords out'? Fewer chords just means less detail and subtlety. It has to be said that leaving out a chord altogether could be preferable to chugging on with the previous chord for the rest of the bar and hoping there'll be no graunching (technical term) dissonance.


apache location directive spanning virtual hosts

Question: I have a simple question.
In my environment there is a apache configured like this.
<VirtualHost 188.8.131.52:80>
        ServerAdmin firstname.lastname@example.com
        DocumentRoot /apps/docroot/
        ServerName server.mydomain.tld
        <Location /report >
                ProxyPass http://report.mydomain.tld/
                ProxyPassReverse http://report.mydomain.tld/
        </Location>
</VirtualHost>

<VirtualHost 1.1.11:443>
         ServerAdmin email@example.com
        DocumentRoot /apps/docroot/
        ServerName server.mydomain.tld
        SSLEngine on
        SSLCipherSuite !EXP1024-RC4-SHA:!EXP1024-DES-CBC-SHA:ALL:!ADH:!EXP56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
        SSLCertificateFile /path/to/crt.crt
        SSLCertificateKeyFile /path/to/crt.key
        SSLCACertificateFile /path/to/CA.crt
</VirtualHost>

The question is, will the proxy settings apply to the SSL vhost? (I.E If i access /report using https, Will the request still get proxied).. I know it will get proxied if i use http..
-Thanks
Vijay
Answer: No, it won't.  The directives within one virtual host only apply to that host.
If you have a set of directives that you want to apply repeatedly in multiple vhost/location/directory contexts, the best way to do that is to put them into a separate file, and Include them in each context where they're needed.


Can't Deploy - VM instance field blank in deployment wizard

Question: I am new - something just must be missing in my configuration...
Can't Deploy - VM instance field blank in deployment wizard:
The wizard is shown where I can't select from the drop-down menu.  I expected it to have the instance "instance-1" which I have created and activated on Google Cloud Platform
Answer: Seems that the Wizard is unable to reach out to the list of VMs. This might be a transient issue that could be resolved by restarting your Visual Studio. If this does not work, check the permissions over the VM in the Cloud Console or reinstall the  Visual Studio extension.
Could note that your project is an APS.NET site. Another way to perform the upload, is the web deploy manager, that fits perfectly with a .NET site and a windows VM.
https://docs.microsoft.com/en-us/aspnet/web-forms/overview/deployment/configuring-server-environments-for-web-deployment/configuring-a-web-server-for-web-deploy-publishing-web-deploy-handler
I am not pretty sure about performing this deploy on command line, since that the deploy-manager is more related to Docker images but maybe it could be useful for you.
https://cloud.google.com/deployment-manager/docs/quickstart
Good Luck!


Module does not show up in blocks (6.x)

Question: I have a module that I've activated, but it does not show up in the disabled blocks list.
The module is named "My Module."
Inside the my_module folder I have the following files:

my_module.module
my_module.info

my_module.info
name = My Module
description = My module description.
core = 6.x
package = My Modules

my_module.module
<?php
function hook_block($op = 'list', $delta = 0, $edit = array()){
  switch ($op) {
    case 'list':
      $block = array();
      //List out all blocks you want to create here
      $block[0]["info"] = t('Display info');
      break;
    case 'view':
      switch ($delta) {
        case 0:
          global $user;
          $block['subject'] = "ADMIN Header of the block";

          if (in_array('Site admin', array_values($user->roles) || $user->uid == 1)) {
            $block['content'] = "input form";
            $block['subject'] = "Header of the block";
          }
          break;
      }
  }
}

Answer: Change function hook_block to function my_module_block, and add return $block; to the end of the 'list' and  'view' cases.
Comment: Thank you! Also had to add `return $block;` to the list case for it to show up.

Module does not show up in blocks (6.x)


Heroku file upload

Question: I'm new to Heroku and I'm looking for a way to upload a bunch of images to my application's directory. Is there a proper way of doing this, like a ssh, or a filesystem access, or something?
Thanks in advance
Answer: I'm not sure what you mean, but I would just add them to the git and upload them like all other documents. Like so:
<code>git push heroku master
</code>
Hope it helps!
Comment: That would do it.
Comment: Glad I could help!

Heroku file upload


Multi-level / Hierarchical Machine Learning

Question: I am trying to tackle a problem that involves binary classification. However, my data is multi-level or hierarchical in its structure. This example illustrates its structure:
  Time Group Response   F1   F2 F3 F4
1    1     a        0 0.89 0.01 10  2
2    2     a        1 0.82 0.58 21  4
3    3     a        1 0.34 0.39 37  6
4    1     b        0 0.32 0.26 10  2
5    2     b        0 0.32 0.04 21  4
6    3     b        0 0.00 0.20 37  6
7    1     c        1 0.80 0.18 10  2
8    2     c        0 0.34 0.71 21  4
9    3     c        1 0.55 0.52 37  6

We have multiple groups which have a binary response (0, 1) that is measured over time. The are an array of features associated with each response in each group. Some of these features take unique values for each time point in each group (F1, F2), while others (F3, F4) represent the environmental "state" that was common across groups at each time point. All feature variables are continuous and numeric and I also do not tell the model that they belong to different groups as I want the model to predict the response based on the continuous features. This is important because over time some groups go extinct and disappear from the data set and other new groups come in so I want the model to be agnostic to group ID.
I use a rolling window walk-forward validation scheme to account for the temporal nature of the data. Until now, I have been running this data through a random forest model via the h2o package in R with y = Response and x = c(F1:F4). The results have been reasonable, however I am noticing that the environmental features (F3, F4) have the most importance and the model is good at predicting when the environment is right – e.g. conditions of F3 and F4 that enable discrimination between 0 and 1 in the Response. However the model is less good at distinguishing between groups on any given day given the environment.
This led me to think that this outcome may be due to not fully accounting for the hierarchical nature of the data. Instead I had the idea to fit random forest models in a 2 stage process. In stage 1, I train and validate the model using only the environmental features (F3, F4). I then use this model to generate predictions for the response variable and then keep only the positive responses ('1s') and throw out all negative responses ('0s'). In stage 2 I then train and validate a new random forest model on this new dataset which includes only the group specific features (F1, F2) to make more granular predictions of when each group will exhibit a positive or negative response. I then finish in stage 3 by testing the two random forest models on the hold out test data in the same 2 stage process – i.e. test with model 1 on F3 and F4, keep positive predictions and then test with model 2 using F1 and F2 to generate the final set of granular predictions.
Does this approach seem reasonable and sensible? Or are there other/better ways to deal with the data structure I have?
Answer: With the data set out how it is, I'm not sure it accounts for the non-independence of your data points. I would consider modelling each time point separately. This has the advantage that you could include previous state as a predictor - it depends what exactly you're modelling, but shouldn't the likelihood of being 1 or 0 at time point 3 depend strongly on whether that group was 1 or 0 at time point 2?
So for example, I would split your example data into three training data sets, split by the Time variable.
Response at Time point 1 would be modelled like this:
Response_at_time_point_1 ~ F1 + F2 + F3 + F4

Response at Time point 2 would be modelled like this:
Response_at_time_point_2 ~ F1 + F2 + F3 + F4 + Response_at_time_point_1

Response at Time point 3 would be modelled like this:
Response_at_time_point_3 ~ F1 + F2 + F3 + F4 + Response_at_time_point_1 + Response_at_time_point_2

You could add a helper column to determine which sub-model each row should go to, based on what time point they are at.
Comment: Thanks for the suggestion. I'm not sure it would be feasible to model each time point separately. The full data set has 200 groups each with 800 time points and over 1000 features. It's not clear that there should a priori be strong temporal auto correlation in the data though I can test that. The key level that non independence exists is at the level of group and the way in which some features are uniquely defined for each time point in each group whilst others are common across groups at any given time point.


Tree-based models for joint Bernoulli/Log-normal distribution

Question: I'm working on trying to model the profitability of a set of customers given a vector of behaviors.
This data includes both customers who did and did not make a purchase. As a result, the DV (profit) is heavily zero-inflated. I've used hurdle models to capture the joint distribution of this outcomes, but in my current problem, I'm currently restricted to only using a tree based model. To that extent, I can only use a single tree.
Are there any tree based models that can accommodate this type of joint distribution?
One possible solution I thought of is to change all of the zeros to an extreme negative number (e.g., -9999). This will max the difference between any observation with a positive value and any zero to try and manipulate the distance. My hope is that I can artificially force a majority of the 0's into a single node and have the algorithm do it's best to discriminate profit values within the non-zero node.
Answer: If I understand you correctly, you want to force one node to lead to predicted value of 0. What you can do is change your dependent variable into a binary value, did they purchase, and build a classification tree predicting whether or not a person purchased. You can then subset your data to only those who purchased and build a tree that predicts the amount purchased assuming they've purchased. You can then paste these two trees together (i.e, the final node in the classification tree is the beginning of the "amount purchased" tree). Basically, a two-step tree procedure.
Comment: I've thought about that, but, due to lack of experience with decision trees, I'm not sure exactly how the estimates would work. With a classification tree, I don't get a node with only a certain group of discrete values, Instead I get a ratio of 1 to 0s. Is multiplying this value through the subtree you suggested different than estimating the two groups independently and multplying the outcome?
Comment: The two-part tree procedure suggested by @TrynnaDoStat is very close to the tree-based version of a hurdle model. One part would be the binary zero hurdle (purchase yes/no) and the second part would be the model for the log(amount) of those who did purchase. The first part would be a binary classification tree (yielding predicted probabilities) and the second part would be a regression tree (yielding a predicted log-amount). In principle, any classification and regression tree procedure could be used that you feel comfortable with, e.g., in R `rpart` or `partykit` etc.


Likelihood of an algorithm-based function

Question: Forgive me if this is a dumb question, but it is way outside of my area of expertise.
Suppose I have an algorithm that produces some sort of a fit function, but that fit is not expressed in closed form (e.g., computational models in biology or cognitive psychology). In other words, to find the optimal parameter, I have to simulate data based on the parameters, do a bunch of steps, simulate some sort of psychological or biological process, do a bunch of mathematical gymnastics, then finally I have a simulated behavior based on the $\theta$ values, which I then compare to the actual behavior. I then compute some sort of fit function that compares the simulated behavior to the actual behavior (e.g., sum of squared errors or something), then modify the $\theta$ values until the discrepancy is very small.
What I want to do is incorporate some bayesian statistics into this computational model (so I can include prior information). I know that a poster $\propto$ likelihood $\times$ prior. I can easily get the priors (based on the literature), but how do I convert the results of the algorithm into a likelihood?
My best guess is to do the following:

Run the computational model algorithm with a bunch of different values for the parameter of interest (e.g., $\theta_1$), holding other parameters constant.
For each proposed $\theta_1$ value, compute the $sse$.
Invert $sse$ (to come up with a crude likelihood measure that increases as fit increases).
Scale all inverted $sse$'s such that they sum to one to compute my pdf.
Repeat for all $\theta_2:\theta_k$.

Will this work to convert the results of an algorithm into a likelihood? If not, what else can I do?
Edit
Here's a bit more details. I'm trying to make an existing model of facial recognition under a criminal lineup (witness) bayesian. The basic idea is this:

Simulate a vector of "features" from a uniform distribution for the criminal
Simulate encoding of said features into one's memory by creating a new vector that is identical to the criminal vector for $\theta_1$ of features. We will call this vector the memory vector.
Simulate "foils" that are used in the same lineup by creating 5 additional vectors that share $\theta_2$ of the features from the criminal vector.
Simulate picking the criminal out of a lineup by computing the dot product between the memory vector and each of the foil vectors and the criminal vector. If the dot product exceeds some value ($\theta_3$), a decision is made as to who the witness thinks committed the crime. Otherwise, the person makes no decision.
Record the proportion of times the simulated individuals picked the right person/wrong person/no person and compare it to the actual number of times experimental subjects picked the right/wrong/no person. This is typically done using $sse$.
We adjust $\theta_1-\theta_3$ until the simulated data matches closely the experimental data (typically done through genetic algorithm).

So, how do I convert $sse|\theta_1, \theta_2, \theta_3$ into a likelihood so I can allow bayesian estimation?
Answer: So you can simulate pseudo observations given the parameters.
In this case it might be helpful to use Approximate Bayesian Computation (ABC). In its crudest form you: define a distance metric between the observed data and some pseudo data; simulate parameter values from your prior and in each simulation generate a pseudo data set; retain all parameter values where the distance between the observed and pseudo data are small. This produces an approximate sample from the posterior.
If you gave a bit more info we might be able to suggest some appropriate metric.
Response to Edit
Steps 1-6 are your means of generating a pseudo sample given the parameters. sse is your distance metric.
So you could simulate steps 1-6 many times and only retain simulations where sse is less than some value, k say. As k goes to zero you will get a more accurate estimate of the posterior.
If you had all the time in the world, you could only retain simulations in which the pseudo data exactly corresponded to the actual data--this is possible in your case as you have discrete outcomes (0 or 1 for each lineup) although probably infeasible if the sample size is moderate. Even if you don't, however, any distance metric you use is just an approximation of this "exact" rejection criteria.
Comment: Thanks for the answer. This may be exactly what I need. See my updated response with more details. Would I just do 1/sse to generate a likelihood?
Comment: Sorry, I mean to say 1/sse to generate a *posterior*
Comment: And if I wanted to construct a credible interval, could I just use 1/sse, divide by the sum of 1/sse to obtain densities?
Comment: The approach I describe gives a sample from the posterior (approximately). Hence you can read off credible intervals from the ECDF of this sample.


Use of .on method over the .delegate method

Question: I gone through the different description provided for the .on() method or .delegate() method. But unable to clarify the actual need of .on() over .delegate().
I just came out with only one difference which of syntax nothing else.
.on() -- function( types, selector, data, fn )
.delegate -- function( selector, types, data, fn )
Any other difference?
Answer: Actually, delegate() uses on(), internally.
The jQuery delegate() method's source code:
function (selector, types, data, fn) {
    return this.on(types, selector, data, fn);
}

So no, other than calling one less function, there is no difference. delegate() is there purely to better differentiate between dynamically and statically added event listeners, should you want to.
Comment: From the [`.delegate()`](https://api.jquery.com/delegate/) API docs: `As of jQuery 1.7, .delegate() has been superseded by the .on() method.`. So yeah, looks like backwards compatibility to me.
Comment: delegate is kept for backward compatibility i guess
Comment: yup, it is. But which difference made them to deprecate the delegate and introduce the new .on method. they might have used the same.
Comment: Sorry, previously I thought superceeded means deprecated. Is there any major reason which made them to supeceeded the delegate and introduce the new .on method.


The Continuity of the Discrete Time Fourier Transform of Absolutely Summable Series

Question: I saw on a book to following claim:
Given an Absolutely Summable Series $ \sum_{n = -\infty }^{\infty}\left | x\left [ n \right ] \right | \leqslant \infty $, Namely, $ l_1 $ series it is possible to show its DTFT (Discrete Time Fourier Transform) is continuous.
Where the DTFT is given by:
$$ DTFT\left \{ x\left [ n \right ] \right \} = X\left ( {e}^{j \omega} \right ) = \sum_{n = -\infty}^{\infty} x\left [ n \right ] {e}^{-j \omega n} $$
My question is, how could that be proven?
Is it also differentiable?
Answer: Defining $ f(n) = x \left [ n \right ] {e}^{-j \omega n} $ and rewriting the DTFT:
$$ DTFT \left \{ x \left [ n \right ] \right \} = X \left ( {e}^{j \omega} \right ) = \sum_{n = -\infty}^{\infty} x \left [ n \right ] {e}^{-j \omega n} = \sum_{n = -\infty}^{\infty} f(n) $$
Since $ f \left ( n \right ) $ is continuous for every $ n $ by the Uniform Limit Theorem the limit function is also continuous if the sequence converges uniformly.
$$\begin{align*}
\sum_{n = -\infty}^{\infty} x \left [ n \right ] {e}^{-j \omega n} & \leq \left | \sum_{n = -\infty}^{\infty} x \left [ n \right ] {e}^{-j \omega n} \right | \\
& \leq \sum_{n = -\infty}^{\infty} \left | x \left [ n \right ] {e}^{-j \omega n} \right | \\
& = \sum_{n = -\infty}^{\infty} \left | x \left [ n \right ] \right | \left | {e}^{-j \omega n} \right | \\
& = \sum_{n = -\infty}^{\infty} \left | x \left [ n \right ] \right | \\
& \leq \infty
\end{align*}$$
The last equality $ \sum_{n = -\infty}^{\infty} \left | x \left [ n \right ] \right | \leq \infty $ come from the definition of the sequence.
Since the convergence doesn't depend on $ \omega $ this is a Uniform Convergence and hence the limit function is continuous.


Git says a file is unmerged and I can't commit, but the file seems to be merged

Question: I had about 12 file conflicts when I merged branches. I manually fixed all the conflicts and staged the files. But now when I try to commit, it says one of the files is unmerged.
U      app/models/disclosure_event.rb
fatal: 'commit' is not possible because you have unmerged files.
Please, fix them up in the work tree, and then use 'git add/rm <file>' as
appropriate to mark resolution and make a commit, or use 'git commit -a'.

When I type git status, it shows all the changed files, including "disclosure_event.rb", under Changes to be committed. It shows no files as unstaged. I've repeatedly tried to add the file, but it seems to do nothing. If I open the file, everything looks great.
Any ideas? I'm completely stuck.
Comment: Sounds like yet another Git bug.....
Comment: Possible duplicate of [Git: can't undo local changes (error: path ... is unmerged)](http://stackoverflow.com/questions/3021161/git-cant-undo-local-changes-error-path-is-unmerged)
Answer: I can't tell you what's wrong, but you might try
git reset app/models/disclosure_even.rb
git add app/models/disclosure_even.rb

and then commit.
Comment: I had to specify `git reset HEAD app/models/disclosure_even.rb`. Otherwise, I'd get `git reset app/models/disclosure_even.rb =>
fatal: ambiguous argument 'app/models/disclosure_even.rb': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions`
Comment: I had to use "git reset -- lib/OaisAip.php", but that got me going!
Answer: I had exactly the same situation today on branches merging (not rebasing).
I spot, that file it errors on, in your case:
U      app/models/disclosure_event.rb

was absent in git status. But is could be seen as unmerged in gitk GUI. It also was absent in working copy (moved to another place in my case).
I tried git rm, and it solved the problem:
git rm app/models/disclosure_event.rb

Comment: Didn't work here... `git rm backend/server/main.js =>
backend/server/main.js: needs merge`
Answer: Set the remote is origin and the branch is master, and say you already have master checked out, might try the following:

git fetch origin
git reset --hard origin/master

It will set the current branch and points it to the HEAD of the remote branch.
Answer: I encountered such situation before with rebase. Git determines that you leave some files as they were before merge and thinks you still didn't merge them. That time I googled and find out many suggestions to do just git rebase --skip. It worked for me.
More detailed description of such case: http://git.661346.n2.nabble.com/BUG-git-rebase-is-confuse-if-conflict-resolution-doesn-t-produce-diff-td726597.html
Similar problem: Git rebase: conflicts keep blocking progress
I'm not sure it is your case if you have merge, though.
Answer: I tried the 2 suggestions without any luck. And I couldn't get it to reset back to HEAD, so I just re-cloned the repository and did the entire merge over and didn't have any issues. For what it's worth, I now highly recommend p4merge over vimdiff...
Comment: I also recommend `mcdiff` (from the `mc` package) instead of the terrible vimdiff.


Classificar resultados com uma condicional e agrupar numa coluna no R

Question: tenho uma coluna com a variavel idade
age =c( 57, 17, 10, 21, 18, 21, 21, 23, 57, 13, 21, 14,  9, 40, 23, 25, 25, 44, 29, 28, 53, 55, 48, 12, 43, 13)

e gostaria de criar uma nova coluna classificando a idade em categorias <12 anos coloque 1, <=12 e>=30 coloque 2 e assim por diante ,
Comment: A principal dificuldade que eu estou a ver é saber quais são os limites das categorias, *"e assim por diante"* não é lá muito util.
Comment: Está difícil de entender a pergunta pois os classificadores em R são usados para delinear informações em conjuntos de dados de modelos preditivos, outra coisa que me chama atenção é fato de você querer adicionar uma nova coluna em um vetor, não faz muito sentido nova coluna em um vetor. Eu estou a mais de 45 min tentando entender sua pergunta e minha interpretação é a seguinte, você gostaria de aplicar uma função predicado a cada elemento de um vetor e colocar o resultado num data frame? Algo parecido com isso https://replit.com/@Augusto-Vasques/SillyFirstShell#main.r
Comment: Tente `cut(age, breaks = c(0, 12, 30, Inf)) |> as.integer()`.
Comment: No exemplo 0-12;13-30;30;ou mais ... Indexado 1,2,3. ...
Comment: Olá, agradeço muito sua ajuda, era exatamente o que eu precisava, editei a pergunta ver se tornei ela mais clara.
Answer: Voce pode usar o comando mutate() para criar uma nova coluna new_age, por exemplo, juntamente com o comando case_when() para criar as condicionais dentro do mutate().
As suas regras nao ficaram claras eu partir do principio que você gostaria:
age<12 = 1
age >=12 e age <30 = 2
age >=30 = 3
```{r}
library(dplyr)
```

```{r}

#vetor
age=c( 57, 17, 10, 21, 18, 21, 21, 23, 57, 13, 21, 14,  9, 40, 23, 25, 25, 44, 29, 28, 53, 55, 48, 12, 43, 13)

#converte esse vetor para um data.frame
df<-data.frame(age)%>%
  #usa a função mutate para criar uma nova coluna new_age
  #usa a função case_when para determinar as condicionais
  mutate(new_age=case_when(
    age<12~1,
    age >=12 & age <=30 ~2,
    TRUE~3))
df
```

age new_age
57  0
17  2
10  1
21  2
18  2
21  2
21  2
23  2
57  0
13  2
Answer: rm(list = ls())
library(dplyr)

df <- data.frame( age= c(57, 17, 10, 21, 18, 21, 21, 23, 57, 13, 21, 14,9, 40, 23, 25, 25, 44, 29, 28, 53, 55, 48, 12, 43, 13))|>
  arrange(age)|>
  mutate(faixa= if_else(age<=30, 1, 2))

Dividi entre menores e maiores de 30, pode ser adicionada outras idades seguindo o modelo.


Combine nested loop queries to parent array result - pg-promise

Question: I'm new to node(express) and pg-promise, and have not been able to figure out how to add the result of each nested query(loop) into the main json array result query.
I have two tables: Posts and comments.
CREATE TABLE post(
id serial,
content text not null,
linkExterno text,
usuario VARCHAR(50) NOT NULL REFERENCES usuarios(alias) ON UPDATE cascade ON DELETE cascade,
multimedia text,
ubicacation VARCHAR(100),
likes integer default 0,
time VARCHAR default now(),
reported boolean default false,
PRIMARY KEY (id)  );

CREATE TABLE comment(
id serial,
idPost integer NOT NULL REFERENCES post(id) ON UPDATE cascade ON DELETE cascade,
acount VARCHAR(50) NOT NULL REFERENCES users(alias) ON UPDATE cascade ON DELETE cascade,
content text NOT NULL,
date date default now(),
PRIMARY KEY (id));

So I want to add the result of each comments to each post and return the posts.
I have this, but doesn't work:
con.task(t => {
    return t.any('select *, avatar from post, users where user= $1 and user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, pos])
    .then(posts => {
        if(posts.length > 0){
            for (var post of posts){
                post.coments = t.any('select * from comment where idPost = $1 ', post.id);
            }
        }
    });
}).then(posts => {
    res.send(posts);
}).catch(error => {
    console.log(error);
});

Any suggestions?
PD: I think my question is kind of similar to this one:
get JOIN table as array of results with PostgreSQL/NodeJS

ANSWERS:
Option 1 (best choice):
Making a single query through JSON to psql (JSON query)

See answer by @vitaly-t

OR

Getting the nested data asynchronously using ajax.

Option 2:
function buildTree(t) {
        return t.map("select *, avatar from publicacion, usuarios where usuario = $1 and usuario = alias ORDER BY hora DESC LIMIT 10 OFFSET $2", [username, cantidad], posts => {
                return t.any('select * from comentario where idPublicacion = $1', posts.id)
                    .then(coments => {
                        posts.coments = coments;
                        console.log(posts.coments);
                        return posts;
                    });
        }).then(t.batch); // settles the array of generated promises
    }

    router.get('/publicaciones', function (req, res) {
        cantidad = req.query.cantidad || 0; //num de publicaciones que hay
        username = req.session.user.alias;

        con.task(buildTree)
        .then(data => {
            res.send(data);
        })
        .catch(error => {
            console.log(error);
        });
    });

Option 3(async):
try{
    var posts = await con.any('select *, avatar from post, users where user = $1 and user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, q])
    for (var post of posts){
        post.coments = await con.any('select * from comment where idPublictcion = $1', post.id);
    }
}catch(e){
    console.log(e);
}

Comment: Actually, the best answer is the second option through a JSON query here: https://stackoverflow.com/questions/39805736/get-join-table-as-array-of-results-with-postgresql-nodejs, because it will be hugely faster than anything else, and will scale well. And that `await` code will be the worst in terms of scaling, as it is synchronous ;)
Comment: @vitaly-t Your are rigth! Thanks for your time! :)
Comment: See the update, you might like it better ;)
Answer: I'm the author of pg-promise ;)

con.task(t => {
    const a = post => t.any('SELECT * FROM comment WHERE idPost = $1', post.id)
        .then(comments => {
            post.comments = comments;
            return post;
        });
    return t.map('SELECT *, avatar FROM post, users WHERE user = $1 AND user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, pos], a)
        .then(t.batch);
})
    .then(posts => {
        res.send(posts);
    })
    .catch(error => {
        console.log(error);
    });

Also see this question: get JOIN table as array of results with PostgreSQL/NodeJS.
UPDATE
In case you do not want to go all the way with the JSON query approach, then the following will scale much better than the original solution, as we concatenate all child queries, and then execute them as one query:
con.task(async t => {
    const posts = await t.any('SELECT *, avatar FROM post, users WHERE user = $1 AND user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, pos]);
    const a = post => ({query: 'SELECT * FROM comment WHERE idPost = ${id}', values: post});
    const queries = pgp.helpers.concat(posts.map(a));
    await t.multi(queries)
        .then(comments => {
            posts.forEach((p, index) => {
                p.comments = comments[index];
            });
        });
    return posts;
})
    .then(posts => {
        res.send(posts);
    })
    .catch(error => {
        console.log(error);
    });

See API:

helpers.concat
Database.multi
Comment: Nice update! The problem is that I don't have the _pgp_ object in my route file, just the conection instance but anyway good contribution.
Comment: @SergioRey That doesn't sound right. If you organize the library [the way you should](https://stackoverflow.com/questions/34382796/where-should-i-initialize-pg-promise), you will not have that problem.
Comment: How can this be implemented when you have to combine more than two queries?
Comment: @NaveenGeorgeThoppan That depends entirely on the query logic/relationship. Cannot answer it in a generic way.
Comment: In the same case as above, a post can have votes associated with it. How can I add a query for getting the votes as well for the post? So the there will be two queries to get comments and votes. Can you help me with the syntax to add another query? In my case I've 5 or 6 such queries to be added.
Comment: @NaveenGeorgeThoppan You can either use the same `map` + `batch` logic as in the original answer, or use the same approach (in update). I cant post a solution in the comments.
Comment: @vitaly-t I'll post a question. Hope you'll be able to help me.
Comment: @vitaly-t Please see question https://stackoverflow.com/questions/51476472/pg-promise-combine-more-than-two-queries
Comment: @vitaly-t you my friend are a legend, this example is exactly what I was looking for (found it in half a day), can you please also put this one into. your section of Learn By Example for lesser mortals :) , that one is good for single value return for parent query, but i had to scourge the net for an example where multiple were returned and then populated.
Answer: If you want structured (nested) data, without having to
A) re-write your sql using json function, or split it out into multiple task queries, or
B) refactor your code to use the API of a heavy ORM
you could check out sql-toolkit. It's a node library built for pg-promise which allows you to write regular native SQL and receive back properly structured (nested) pure business objects. It's strictly an enhancement toolkit on top of pg-promise, and does not seek to abstract out pg-promise (you still set up pg-promise and can use it directly).
For example:
class Article extends BaseDAO {
  getBySlug(slug) {
    const query = `
      SELECT
        ${Article.getSQLSelectClause()},
        ${Person.getSQLSelectClause()},
        ${ArticleTag.getSQLSelectClause()},
        ${Tag.getSQLSelectClause()}
      FROM article
      JOIN person
        ON article.author_id = person.id
      LEFT JOIN article_tags
        ON article.id = article_tags.article_id
      LEFT JOIN tag
        ON article_tags.tag_id = tag.id
      WHERE article.slug = $(slug);
  `;
  return this.one(query, { slug });
  // OUTPUT: Article {person: Person, tags: Tags[Tag, Tag, Tag]}
}

The select clause uses the business object "getSQLSelectClause" methods to save tedium in typing the columns, as well as ensure no collisions of names (nothing magical going on, and could just be written out instead).
The this.one is a call into sql-toolkits base DAO class. It is responsible for structuring the flat result records into a nice nested structure.
(Also notice that it is "one" which matches our mental model for the SQL. The DAO methods for one, oneOrNone, many, and any ensure their count against the number of generated top level business objects - not the number of rows the sql expression returns!)
Check out the repository for details on how to set it up on top of pg-promise. (Disclamer, I am the author of sql-toolkit.)
Answer: You can use await but it will work sync.
return t.any('select *, avatar from post, users where user= $1 and user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, pos])
    .then(posts => {
        if(posts.length > 0){
            for (var post of posts){
                post.coments = await t.any('select * from comment where idPost = ', post.id);
            }
        }
        return posts;
    });

Actually i recommend you to use orm tools like bookshelf, knex, typeorm
Comment: The solution with pg-promise will outperform any of the mentioned ORM-s by a large margin, so it is not a good recommendation.
Comment: Thanks for your great module vitaly-t ;)


Find second customer visit based on custom conditions in sql

Question: I have a customer visit table in postgre sql with the following fields customer_id, visit_date, purchase_amount. For each unique customer, I want to find their first and second visit.
I find their first treatment by
min(d.visit_date) OVER (PARTITION BY d.customer_id) as index_visit_date

I am having trouble figuring out how to find the second treatment because I want to define the second visit as a visit at least 60 days after the index_visit_date.
So if a customer has the following visit history:
 id|visit_date|purchase_amount
 1 | 10-19-19 |1000
 1 | 10-24-19 |100
 1 | 12-15-19 |1000

The index_visit_date = 10-19-19 and the second_treatment = 12-15-19
I am not sure how to calculate the difference in time from index_visit_date and then select the next visit a minimum of 60 days after as the second treatment.
Answer: One method is a lateral join:
select vfirst.*, vnext.*
from (select distinct on (id) v.*
      from visits v
      order by id, visit_date
     ) vfirst left join lateral
     (select vnext.*
      from visits vnext
      where vnext.id = vfirst.id and
            vnext.visit_date >= vfirst.visit_date + interval '60 day'
      order by vnext.visit_date
      limit 1
     ) vnext
     on true;

Actually, if you just want the date, there are simpler methods, such as:
select v.*
from (select v.*,
             row_number() over (partition by id order by visit_date) as seqnum,
             min(visit_date) over (partition by id
                                   order by visit_date
                                   range between '60 day' following and unbounded following
                                  ) as next_visit_date
      from visits v
     ) v
where seqnum = 1

Here is a db<>fiddle.
Comment: Hmm neither solution is working for me. The first method is throwing the error that distinct on is not supported. The second method is throwing a syntax error for the 60 days. I am running this in redshift postgre if that helps
Comment: @dsmess . . . They work fine.  Your sample data has no instance with a visit more than 60 days after the first visit.


What is the name of a circular clock-like diagram in which you can place a triad shape and rotate it to get the notes in a chord of a particular type?

Question: Not sure where I got this idea, but if you take your chromatic scale and wrap it in a circle, then draw a triangle, for example from A to C to E back to A, and rotate that triangle, it will show you the notes of another chord of the same type.  It also works with extended chords and entire scales, like if you draw a star from each point ofthe major scale or one of the modes, you can visually see what notes belong to a scale or mode with a different root simply by rotating the shape.
Is there a name for this?
Comment: Could that be just a circle of fifths? Like this one: https://en.wikipedia.org/wiki/File:Circle_of_fifths_deluxe_4.svg ? (It's not the same thing exactly but you can do all the things with inscribed shapes that you describe and it will work too. And it's useful in other ways too.)
Comment: Well.. almost, but a Circle of Fifths unwound.  If you trace the whole circle of fifths around it you get a dodecagon (12-pointed star).  If you just label the natural notes, and/or add white and black circles (or whatever mark you chose) to mark the white keys on a piano, it also makes it real easy to relate or map the chord and scale shapes to a keyboard.  I've only met one person who used this, and he didn't seem to know where he got the idea either.  I've never heard or seen it mentioned by anyone with formal music training.
Answer: Sounds like you are referring to the Tone Clock:

Peter Schat's 'Zodiac of the Hours', which graphically represents the
  tone-clock steerings of the twelve hours. Note that X can only be
  steered as a diminished seventh tetrachord (hence, the only
  non-triangular shape). Each point of a shape represents a pitch-class
  on the chromatic circle, and each shape represents one transposition
  or inversion of an hour.
Comment: Awesome! @Stinkfoot, I've been trying to find this forever!!  So cool.  Besides mapping triads, you can also map extended chords like 7ths etc.  I found a lot of things out this way long before being told or reading about it somewhere.  Like the various 7th chords are basically two triads overlaid, that share 2 of their 3 notes, and that many chords with complicated names are just inversions of simpler, more common chords.  You can even map out scales or whole songs geometrically within a key, or continuing outside.  Made sense of a lot of things where the theory was verbally confusing.
Comment: @jdmayfield - I just found it a few days ago, so I guess it was fate. I was doing some research about jazz harmonica and came across that site where it's mentioned. I haven't spent much time with it yet, but the last tube I put up there is a lecture about its use.
Comment: That's pretty awesome.  I've found it helps a lot trying to understand how different instruments relate tk each other as well.  Really, it's kind of an awesome map of the entire Chromatic Scale, where you can plot intrval paths between the notes for chords, polychords, scales, even entire songs that don't always fit into a little scale box or single key. So cool how you can transpose by taking a particular shape and rotate it and immediately see what notes in the transposition without doing any mental gymnastics.  It's like a musical calculator, only it's just a diagram on a piece of paper!
Answer:
Oh hey, I made one of those not too long ago. I even prettied it up. I call it the Major Heptagon. I suppose I should go back and make a 12 tone version of it too.
Red is the tonic (is Major) and moving clockwise goes up the scale while counter-clockwise goes down. The green lines are 2nds/7ths, the gold lines are 3rds/6ths, the blue lines are 4ths and the red lines are 5ths. The spiked blue line is an augmented 4th while the spiked red line is a diminished 5th.
Comment: I don't really associate the colors with a particular key, ie red could be any note. I tend to think of natural minor as the major scale starting on 6. On my chromatic diagram I did make C red just because most theory is based on C. I also prefer the use of RGB/CYM since they're the purest forms of color; orange being the odd color out. It also doesn't matter much to me for it to be symmetrical since the scale itself is not built symmetrically. In short, this diagram is mostly for me to remember the intervals between all the notes.
Comment: Nice, @Tama.  I have an easier time relating everything to Natural Minor, probably because I'm dyslexic and it's easier for me to think starting at A.  I also use colors associated with the notes, so for me A is red.  So the spectrum ROYGBIV is ABCDEFG, making the black keys in-between something like reddish-orange, lime-green, greenish-blue, purple, and reddish purple.  If you make your 12-note clock with D at the top, and trace the diatonic scale skipping every other note (ie min and maj 3rds), you get a nice symetrical, rotatable view of all the Church Modes.
Comment: Ah, yes.  I like to use the modes a lot, so more specifically, red for me is the Minor/Aeolian root, orange Locrian root, yellow Major/Ionian root and so forth.  The in-between colors for blue notes or adjusted scale roots, like the 7th mode of Harmonic Minor (I forget the name).  Actually Natural Minor is symetrical from beginning on the left to the right, the Dorian mode is symetrical from the root going left or right outward.  If you play the modes using the same root note, changing to the next mode by order of fifths, you get a symetrical order of feeling from darkest to lightest...
Comment: e.g. keeping, let's say A as root, play Lydian, then Ionian, Mixolydian, Dorian, Aeolian, Phrygian, and Locrian, or the exact reverse.  You will see that in that order you go from lightest feeling (Lydian) to the darkest (Locrian).  Coincidentally, in conventional tonal music theory you are just going through the keys according to the Circle of Fifths/Fourths.  But if you have a drone note in the background to keep your attention on the A (or whatever note you choose as long as it's the same) as root for all 7 modes, it's quite noticable.  Also only one note changes at each modulation.


SVD for PCA: Why would one standardize the data matrix?

Question: As explained in amoeba's beautiful answer here one can use a singular value decomposition of the data matrix, $\mathbf{X} = \mathbf{USV}^\top$, to do a principal component analysis, if it is assumed that the data matrix $\mathbf{X}$ is centered. The principal directions will then be given as the columns of $\mathbf{V}$.
Now, I have come across some statistical analysis where the data matrix was centered and standardized for the data to have unit variance before being used in a singular value decomposition. The analysis then proceeded in using the columns of $\mathbf{V}$ as principal directions.
Can any of you think of a reason to do this standardization step before running the singular value decomposition? Is the result of such a procedure in terms of what PCA should tell you even meaningful?
Comment: This is indeed a duplicate but +1 for a clear and well-formulated question.
Answer: In the context of PCA, the reason for making all the variables have the same variance (by standardizing them) is simple: to make them all be treated as equally important. PCA will treat variables with greater variance as more important, since it tries to account for the greatest possible total variance with each successive component.
Comment: +1. Related meta post: http://meta.stats.stackexchange.com/questions/4332.


SignalR with Redis Backplane Behind F5 - StatusCode: 400, ReasonPhrase: 'Bad Request'

Question: I'm using SignalR version 2.1.2 with SignalR.Redis 2.1.2 on Server 2012 R2, IIS 8.5 with WebSockets enabled.
All is running perfectly in my development environment. I can even stand up copies on different servers (e.g. http machine1/myapp/signalr, http machine2/myapp/signalr) of the site configured to use the same backplane, and both UI's get messages pubb'd to them perfectly.
I then moved "myapp" to our next environment, which is a cluster of 2 machines sitting behind an F5 load balancer, with a dns alias setup to route to the F5, and then round robin "myapp". The website itself can connect to signalr just fine, and can receive published messages it subscribes to, BUT when I try to publish to the site via the alias (e.g. http myappalias/signalr), I get a 400, Bad Request error response. Here is an example of the error.
<code>  InnerException: Microsoft.AspNet.SignalR.Client.Infrastructure.StartException
       _HResult=-2146233088
       _message=Error during start request. Stopping the connection.
       HResult=-2146233088
       IsTransient=false
       Message=Error during start request. Stopping the connection.
       InnerException: System.AggregateException
            _HResult=-2146233088
            _message=One or more errors occurred.
            HResult=-2146233088
            IsTransient=false
            Message=One or more errors occurred.
            InnerException: Microsoft.AspNet.SignalR.Client.HttpClientException
                 _HResult=-2146233088
                 _message=StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Pragma: no-cache
  Transfer-Encoding: chunked
  X-Content-Type-Options: nosniff
  Persistent-Auth: true
  Cache-Control: no-cache
  Date: Thu, 13 Nov 2014 22:30:22 GMT
  Server: Microsoft-IIS/8.5
  X-AspNet-Version: 4.0.30319
  X-Powered-By: ASP.NET
  Content-Type: text/html
  Expires: -1
}
</code>
Here is some test code I'm using to publish test messages to each environment, where it fails on "connection.Start().Wait()"
<code>class Program
{
    static void Main(string[] args)
    {
        var connection = new HubConnection("http://myappalias/signalr");

        connection.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;

        var proxy = connection.CreateHubProxy("MyAppHub");

        connection.Start().Wait();

        ConsoleKeyInfo key = Console.ReadKey();

        do
        {

            proxy.Invoke("NewMessage", new Message() { Payload = "Hello" });

            Console.WriteLine("Message fired.");

            key = Console.ReadKey();

        } while (key.Key != ConsoleKey.Escape);
    }
}
</code>
Now, if I don't use the "myappalias", and instead hit the server head on, it works perfectly. It appears either the F5 is the problem, the client needs to be configured differently for this scenario or I have to do something different when setting up signlar's startup class. Here is an example of the startup class I'm using.
<code>[assembly: OwinStartup(typeof(MyApp.Startup))]
namespace MyApp
{
    public class Startup
    {
        private static readonly ILog log = LogManager.GetLogger
        (System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

        public void Configuration(IAppBuilder app)
        {
            try
            {
                log.Debug(LoggingConstants.Begin);

                string redisServer = ConfigurationManager.AppSettings["redis:server"];

                int redisPort = Convert.ToInt32(ConfigurationManager.AppSettings["redis:port"]);

                HubConfiguration configuration = new HubConfiguration();
                configuration.EnableDetailedErrors = true;
                configuration.EnableJavaScriptProxies = false;
                configuration.Resolver = GlobalHost.DependencyResolver.UseRedis(redisServer, redisPort, string.Empty, "MyApp");

                app.MapSignalR("/signalr", configuration);

                log.Info("SIGNALR - Startup Complete");
            }
            finally
            {
                log.Debug(LoggingConstants.End);
            }
        }

    }

}
</code>
I download the client source code, and wired that in directly instead of the nuget package, so I could step through everything. I seems it successfully negotiates, and then attempt to "connect" with SSE's and then LongPolling transports, but fails at both.
Question 1.1
Anyone know of an alternative to Signalr for .NET that supports scaling with load balancing in a less "I want to pull my hair out" kind of way?
Answer: It should not be necessary to configure source address affinity to use SignalR behind a load balancer. It's certainly not wrong to set up session affinity, but that doesn't fix your underlying problem.
If you look closely at the content of the 400 response, you probably see a message similar to "The ConnectionId is in the incorrect format."
SignalR uses the server's machine key to create an anti-CSRF token, but this requires that all the servers in your farm share a machine key for the token to be properly decrypted in when SignalR requests hop servers. The /negotiate request that you see succeed is the request that retrieves the anti-CSRF token. When the SignalR client then uses the anti-CSRF token to make a /connect request, it failed because the /connect request was processed by a different server that didn't create the token and is unable to decrypt it.
This explains why setting up session affinity fixed your problem, but sharing a machine key will help you avoid this problem even if something goes wrong with session affinity.
Here is an issue that filed on GitHub by someone who experienced a similar issue: https://github.com/SignalR/SignalR/issues/2292.
Comment: Interesting, thanks for the link and explanation. I did not see "Then ConnectionId is in the incorrect format" anywhere in the response or it's nested responses, but it certainly makes sense that your example could be what was actually happening.
Comment: It can be a little tricky to read the response body when inspecting an HttpClientException/WebException since the response body is still a stream. The easiest way to get at it is probably by accessing `ex.GetError().ResponseBody` where [GetError()](http://msdn.microsoft.com/en-us/library/microsoft.aspnet.signalr.client.errorextensions.geterror(v=vs.118).aspx) is an extension method for Exception defined by SignalR in the Microsoft.AspNet.SignalR.Client namespace.
Answer: The problem was fixed by switching the profile for "MyApp" in the F5, to using the "source_addr" profile built into the F5 as a parent profile with a timeout of 1 hour. Here is a description of what that profile does:

Source address affinity persistence Also known as simple persistence,
  source address affinity persistence supports TCP and UDP protocols,
  and directs session requests to the same server based solely on the
  source IP address of a packet.

EDIT
This ended up "Working" for a while, but if I deploy a publisher (something that simply publishes through the signalr client) without republishing the Hub, the publisher times out trying to connect over and over and over again. uhg.
Comment: ya, I experienced something similar to this problem last year by just implementing Async get requests on regular ASP.NET pages.  The problem was the F5 device would start a session and the user would click on an Async post back, the server that got the request wasn't the one that served up that page in the first place, so.... it didn't work.  The word at the time on the street was "we don't support sticky sessions"...  But get this, rather than fix it, they rolled back all the Async support in the applications!  Go Forward 4 Go Back 6....
Comment: @user1522548 - yeah, I hear ya.


Como usar os módulos e o decorador NgModule?

Question: Tenho dúvidas de como usar os decoradores do NgModule. Quais são e pra que servem?
Como criar um módulo separado para carregar uma ou mais bibliotecas e componentes?
Por exemplo, um para carregar as dependências e elementos do Material Design, outro para carregar e iniciar o AngularFire e outro para carregar determinados componentes da aplicação
Comment: Amigo, se você seguir esse guia, acredito que todas suas dúvidas vão ser esclarecidas.
https://angular.io/guide/ngmodules
Answer: Módulos são a principal forma de organização e arquitetura que o angular fornece. Sendo assim como qualquer coisa relacionada a arquitetura depende principalmente de cada projeto. Mas de acordo com a documentação official e ha alguns guidelines gerais recomendados como uma arquitetura por features.
Primerio vamos ver os campos existente nos modulos:
Obrigatórios:
imports - > Onde vc importa outros módulos que seu modulo depende.
declarations -> Declarações dos componentes que estão dentro do seu modulo.
Opicionais
exports -> Declarações dos componentes sera visíveis para outros módulos que importarem este modulo.
providers -> Declarações dos services a partir do angular 6 vc pode declarar o service pra ter um escopo global dentro dele deixando essa opção meio inutil pra aplicacoes mais novas.
Você tem sempre o app.module que o modulo principal que carrega outros módulos e suas rotas principais. Tudo que for carregado aqui terá um escopo global na sua aplicação mas deixara mais lento ao inciar.
O ponto de ter vários módulos e justamente poder ter uma aplicação extremamente eficiente onde cada modulo so usa o que realmente necessita. Por exemplo nem todo modulo necessita de forms por exemplo. Dessa forma e com a ajuda de lazy loading o angular consegue deixar sua aplicação bem leve.
Sendo assim vamos ver um exemplo:
Se vc tiver uma feature produtos que tem por exemplo uma rota /produtos que permite listar e inserir produtos faz sentido ter um produtos module onde vc carrega esses componentes(lista-produto, inserir-produto) e este seu modulo so sera carregando quando seu usuário entrar em /produtos o que e bem util para aplicacoes grandes.
Outro ponto interessante são shared modules ou modulos compartilhados que normalmente contem componentes exclusivamentes de vizualizacao (so contem inputs e outputs e nenhuma logica de negocio). Neste shared modules vc pode ter por exemplo um ter cards module que pode no nosso exemplo teria uma card de produto e onde esse modulo for importado vc tera acesso a esse componente. Eu recomendo ter varios desses shared modules assim vc pode ter um botoes.module, cards.module ou qualquer coisa genérica que vc imaginar. Ajudando a compartilhar o código e dando a consistência a visualização do seu projeto.


Django - Sum annotation adding group by id to clause

Question: The following queryset
subquery = aggregate_table.objects.filter(
    item__parent__name=OuterRef(dimension)
).annotate(
    aggregate_value=Sum(aggregation_field)
).order_by(
    'item__parent__name'
).values('aggregate_value')

queryset = queryset.annotate(aggregate_value=Subquery(subquery))

produces SQL that groups by both the name (supplied by the order_by) and by the id.
SELECT SUM(U0."data") AS "aggregate_value"
FROM "aggregate_table" U0
INNER JOIN "item" U1 ON (U0."item_id" = U1."id")
INNER JOIN "parent" U2 ON (U1."parent_id" = U2."id")
WHERE U2."name" = (T12."name")
GROUP BY U0."id",
            U2."name"
ORDER BY U2."name" ASC

the ID produces incorrect results and instead of summing and returning 1 row with say 100 as the sum'd value, it returns 100 rows with 1 as the sum'd value.
any ideas on how to remove the u0.id from the group by clause?
edit:
Models
class aggregate_table(models.Model):
    id = models.BigAutoField(primary_key=True)
    item_id = models.ForeignKey(Item, blank=False)
    data = models.FloatField(blank=True, null=True)

class item(models.Model):
    id = models.BigAutoField(primary_key=True)
    parent = models.ForeignKey(Parent, blank=False, related_name='parent')

class Parent(models.Model):
    id = models.BigAutoField(primary_key=True)
    name = models.TextField(blank=True, null=True)

The goal here is to get the sum of data by parent name. Ideally if I were to write this purely in sql I'd write something like this
select sum(data) from aggregate_table
join item on item.id = aggregate_table.item_id
join parent on parent.id = item.parent_id
where xxx
group by parent.name

Comment: You can not `ORER BY U2.name`, since that is part of the aggregate, it will thus prevent you from aggregating `SUM(U0.data)` effectively, since for each `U0` there is at least one `U2.name`.
Comment: Perhaps it is better to share your models and ask *what* you want, not *how* you want to achieve this.
Comment: @WillemVanOnsem I updated to include models and ideally the sql that I'd like to generate (to show what the return set should be). This needs to be as a subquery so that it can fix as an annotation into a much larger query.
Comment: The goal of the order by u2.name was to add it to the group by, which from what I'm reading in Django is (aside from .values_list()) the only way to add it to the group by clause
Answer: You can .filter(…) [Django-doc] with:
from django.db.models import Sum

Parent.objects.annotate(total=Sum('parent__aggregate_table__data'))
One can use double underscores (__) to look "through" relations.
The Parents that arise from this queryset will have an extra attribute .total that is the sum of the related data of the aggregate_tables that have a related item that is related to that Parent.
Comment: That does not produce the results needed. The query needs to begin with the aggregate_table to filter results only from the outer query (using OuterRef), and then find the related parents.
Comment: @Mcscrag: but that does not make sense, you perform a `GROUP BY` on the `parent.name`, so that means you query is tailored towarts the `parent` model: you want to obtain the sum *per parent*, otherwise the `GROUP BY` would not make any sense.
Comment: @Mcscrag: if you do not want `Parent`s that have no related `aggregate_table`, that can be filtered out.
Comment: @Mcscrag: but I think that this question "suffers" from the *XY-problem*: you want to achieve *X*, and you assume that you need *Y* for that problem, so you ask how to achieve *Y*, but there are better ways to solve *X* than with *Y*. In other words, I think you should explain what the underlying problem is.