Redirecionamento de página conforme país de acesso

Question: Possuo 2 páginas PHP, uma em português e a outra em inglês, e gostaria que o usuário fosse redirecionado automaticamente para a inglês, caso o acesso fosse fora do Brasil. Existe alguma maneira ? Ou a solução mas eficaz é criar um link para a página em inglês, igual possui em muitos sites.
Comment: http://stackoverflow.com/questions/12553160/getting-visitors-country-from-their-ip
Comment: Você pode redirecionar a página caso o IP do usuário não comece com 200... é o primeiro número do IP brasileiro.
Answer: Você pode usar o $_SERVER["HTTP_ACCEPT_LANGUAGE"] do php. Ela traz informações enviadas do browser.
<?php
 echo $_SERVER['HTTP_ACCEPT_LANGUAGE'];
 // pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4,gl;q=0.2

$arralang = explode(',', $_SERVER['HTTP_ACCEPT_LANGUAGE']);
$lang = $arrlang[0];

switch ($lang) {
    case 'pt-BR':
      // redireciona para página em português
      break;
    case 'en-US':
    case 'en':
      // redireciona para página em inglês
      break;
    default:
      // redireciona para página em português
 }

Comment: há um problema com isso, eu estou em portugal, no entanto depois de testar deu-me "en-US"
Comment: Interessante Miguel!!! Fiz uma pesquisa rápida. Parece que o Firefox dá preferência à "en-US". Me confirme qual foi o navegador que você usou e, se possível, teste em outro navegador para termos mais dados.
Comment: Sim foi esse mesmo, mas não consigo testar noutro porque de momento o chrome está-me a crashar
Answer: Dependendo da tua versão do PHP, dê uma lida na documentação do PHP sobre Classe de localização
Answer: Pode usar uma API, neste caso usei http://ip-api.com/, e fazer:
// descobrir o ip do utilizador
if (!empty($_SERVER['HTTP_CLIENT_IP'])) {
    $ip = $_SERVER['HTTP_CLIENT_IP'];
} elseif (!empty($_SERVER['HTTP_X_FORWARDED_FOR'])) {
    $ip = $_SERVER['HTTP_X_FORWARDED_FOR'];
} else {
    $ip = $_SERVER['REMOTE_ADDR'];
}

$ip_test = '18.104.22.168';
$local_json = @file_get_contents('http://ip-api.com/json/' .$ip_test); // aqui usa o $ip, este é só para teste

$countries_en = array('gb', 'us'); // definir codigo iso2 dos paises para os quais o site será apresentado em inglês
$lang = 'pt'; // linguagem default
if($local_json !== false) {
    $local_array = json_decode($local_json, true);
    if($local_array !== null && isset($local_array['countryCode'])) {
        if(in_array(strtolower($local_array['countryCode']), $countries_en)) {
            $lang = 'en'; // definir site com a linguagem Inglês
        }
    }
}
echo $lang; // en, pois o país (neste exemplo) são os EUA, cujo código é US, está dentro do nosso array $countries_en

Note que no servidor de produção em vez de $ip_test usa o $ip
Answer: Você pode usar também esta alternativa, redirecionando para default ou a linguagem, caso exista.
echo $_SERVER['HTTP_ACCEPT_LANGUAGE']; // pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4,gl;q=0.2

$arrlang = explode(',', $_SERVER['HTTP_ACCEPT_LANGUAGE']);
$lang = $arrlang[0];
$default_lang = "pt-br";

if(isset($lang)){
     header("Location: url/".$lang);
} else {
     header("Location: url/".$default_lang);
}

Answer: Uma forma de fazer isso, é usando uma API, que identifique o país, veja o exemplo abaixo:
function isBrazil() {
    $location = file_get_contents('http://freegeoip.net/json/');
    if ($location) {
       $data = json_decode($location, true);
       if ($data['country_name'] == 'Brazil') {
          return true;
       }
    }
    return false;
}

if (isBrazil()) {
   header('Location: pagina-brasil.php');
} else {
   header('Location: pagina-ingles.php');
}

Ou se quiser só redirecionar para fora...
 if (!isBrazil()) {
    header('Location: pagina-ingles.php');
 }


Environment variable is not setting globally

Question: I am using csh terminal.
.cshrc
setenv $files /home/ec2-user/files

.login
if [ -f ~/.cshrc ]; then
     . ~/.cshrc
fi
I am trying to echo $files values from plink.
It showing the error files: undefined variable
Comment: Use export files=/home/ec2-user/files instead
Comment: Don't use `csh`. Try [zsh](http://zsh.org/). Read [csh considered harmful](https://www-uxsup.csx.cam.ac.uk/misc/csh.html)
Comment: @RamanSailopal, this is csh, not bash or posix. `export` does not exist.
Answer: You don't use a dollar sign when setting a variable, you only use it when you refer to that variable.
setenv files /home/ec2-user/files

The test command should be a built-in in most csh/tcsh implementations, and has most of the same functionality you'll see listed under man test.
test -f ~/.cshrc && source ~/.cshrc

Note that normally, csh/tcsh will run your .cshrc or .tcshrc file automatically, before it runs .login.


Connecting a limited user to Windows server 2003 through LogMeIn

Question: LogMeIn support can't help me, maybe someone here can.
I have a user on a Windows Server 2003 (non administrator).  When I first created him and tried to connect through remote desktop I got the error
<code>The local policy of the system does not permit you to logon interactively.
</code>
I looked up the error and saw that this user was not a member of the Remote Desktop Users group. I added him to the group and was able to remote desktop in just fine. Then I tried to connect this user to the server through LogMeIn (Pro trial version). I got the same error as when the user was not a part of the Remote Desktop Users group.
Can anyone tell me any other groups the user must be a member of in order to be allowed access through LogMeIn? I would imagine this is a common requirement - LogMeIn support seemed to have no idea.
Comment: You can add the user to the group: Remote Desktop Users and then, he will be able to log in thru RDP.
Answer: you must change this local policy (gpedit.msc) (or through GPO):
Computer \windows settings\ security settings \ local policies\ User Rights Assignements:

Allow log on locally

Add the user to the list of this right
Comment: Define "Cause trouble". I hate to say it but this is a quite simple equation: If you do not trust someone enough to give them admin rights in the first place then you really need to think twice about letting them log on to your domain controller interactively.
Comment: I get an ominous warning that modifying this setting may affect compatibility with clients, service...  Also, the 'Add User or Group' and 'Remove' buttons are disabled.
Comment: Is it a SBS2003 or Windows 2003 SRV
Comment: yes, sbs with SP1. I need to update this.
Comment: I was just thinking, the reason for all this is for some remote users to run a client/server app from the server. I suppose it may be better to set up an XP machine on the same LAN as the server and just let them run the app from there. I suppose these issues will go away if they're connecting to an XP machine instead of directly to the server.
Comment: it's a domain controller, must be done through gpo
Comment: I noticed that 6 groups are already showing up in the 'allow log on locally' group; Account Operators, Administrators, Backup Operators, IUSR_servername, Print Operators, and Server Operators. Can I just make my users a part of one of these groups? Which would be the one least likely to cause trouble, Print Operators?


Sierra Mail Plugin

Question: A client of mine who has a Mail plugin ("unofficial" of course since Apple does not provide support for mail plugin) wants me to update its pluginto Sierra.
When I look at the code swizzled, I see that one of the main classes he used is the SignatureBundle, which inherits from MVMailBundle. In this class, we can find many signature related functions like addSIgnature, signatureWithId and accountsEmails.
My problem is that this class no longer exists on Sierra, and I can't find any other class that I can use.
Do you have any clue that would help me figure out who to update his plugin? I know it is not official SDK, but who knows, maybe one of you already tried to update a mail plugin
Thanks for your help
Comment: No, anyone? Thanks :-)
Answer: As you rightly say, the SignatureBundle class is no longer present inside Mail's main executable.  Such is life when you are using undocumented classes. :-)
Best advice I can give you is to take a look at the Mail-related frameworks inside /System/Library/PrivateFrameworks.  If you are lucky, Apple may have simply moved this class into one of the private frameworks.  However, if you are using NSClassFromString to find the class and getting nil back, then you're probably out of luck.
In which case, it's a long, tedious trawl around in the innards looking for the exact functionality you want.  Your best bet in this case is Mail.framework itself.  It contains a couple of hopeful-looking classes called MFSignature and MFSignatureManager.  Some disassembly required...
Comment: Thanks for your answer though it came a little late. A friend pointed out MFSignatureManager and MFSignature, and hopefully it did the trick :-). Thanks a lot for your answer though, I'll use it as the rightful answer ^^

Sierra Mail Plugin


How do I set up HTTPS on a multi-site?

Question: I made a couple websites on a drupal multisite setup.. and I want to start using https. And.. I'm just lost. Can anyone help me get started?
So, here's the details of how I have things set up..
So I have like a drupal 7 install on ubuntu, with 3 sites.. all in the var/www/html/sites folder, so for example, the folders are named
site1.example.com
site2.example.com
site3.example.com

and the sites.php file directs the requests so like
$sites = array(
    'domain1.com' => 'site1.example.com',
    'domain2.com' => 'site2.example.com',
    etc
);

and as far as dns is concerned, each domain just points at the IP of the ubuntu server and drupal handles the rest.
All that is so far so good. Got several working sites.. each domain goes to the right site.. etc. But they use http. How.. do I go about.. getting a certificate that handles the 3 sites so that they can use https?  I've not set this up before.. I've been trying to read through whatever I can find.. but I'm confused! Sorry, not sure what to do, I'm just .. lost!
Comment: The "let's encrypt" service might help. Here is a helpful article: Installing free of charge SSL certificate. https://mobilefish.de/installing-free-charge-ssl-certificate
Answer: Alright .. solution time. Found a solution.. took me a little while to understand.
(I, um, guess there's more than one way to skin a cat.. and more than one way to do multi site. This is a solution that works with my particular setup)
So the first step is to set up a virtual host (Once again I'm using Ubuntu.. version 14.04 if you wanna know.. although I'd bet the overall steps would work for any other distribution even if the details were slightly different)
So I set up a virtual host as described here: https://www.digitalocean.com/community/tutorials/how-to-set-up-apache-virtual-hosts-on-ubuntu-14-04-lts
So for like, domain1.com on my multisite instillation, my virtual host file looked like this, more or less
<VirtualHost *:80>
    ServerAdmin firstname.lastname@example.com
    ServerName domain1.com
    ServerAlias www.domain1.com
    DocumentRoot /var/www/html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

My DocumentRoot is exactly the same as my drupal base instillation
Also, my DNS doesn't use www so .. I'm not sure if I really needed that ailas but I had it
Then.. for this domain.. I had to use LetsEncrypt and got a free certificate as described here: https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-14-04
(And I had to open port 443 to traffic on my server too .. I was using AWS and they use a thing called "security groups" so I added port 443 to the server's security group on the admin panel)
And then.. bingo! My site in my multisite setup can use https now. I expect I would have to simply just repeat the steps for any other sub-site that I want to use https for. I feel like there would be a way to use some kind of like... multi domain certificate.. for like, every site.. somehow ...
But this way is free and doesn't take too long!
Answer: Are you using a module to handle the multisite configuration? A place where you can associate each site to a domain, theme and other configuration settings. Also you should be able to add https protocol to the desired domain.
If you don't use any module, check settings.php of each site looking for $base_url where you can change it from http to https.
Comment: I'm not using a module, I just have an Ubuntu server and that is the directory structure and sites.php setup that I have. DNS wise, the domains all point at the IP address of the server. Works pretty much out of the box, with no modules or server configuration or anything. I was mainly wondering how to set up a certificate for these domains. Think I might have to mess with the virtual hosts or something.. hmm. If I can find an answer I'll post it..

How do I set up HTTPS on a multi-site?


Automate Openshift pod rebuild on new Docker image

Question: I have a docker container image in our intranet hosted Gitlab registry. I can manually pull the image from our OpenShift installation and run up an arbitrary number of pods successfully. If I rebuild the image locally and push to Gitlab I can trigger a pod rebuild manually from Openshift. All this is working well.
How can I trigger the pod rebuild automatically whenever I push a new image to the Gitlab registry? I don't see anywhere to place hooks between OpenShift and Gitlab and all my reading about Image Streams hasn't resulted in a successful automated deployment pipeline. The deployed versions below;
GitLab Community Edition 9.4.6 23ec1ec
Version
OpenShift Master:
v184.108.40.206
Kubernetes Master:
v1.5.2+43a9be4
Any help greatly appreciated
Answer: You possibly need to schedule update of image meta data.

https://docs.openshift.com/container-platform/3.6/dev_guide/managing_images.html#importing-tag-and-image-metadata

That requires that feature be enabled globally in the OpenShift cluster though.
A better option may be to push the image direct into the OpenShift internal registry. That can trigger the new deployment automatically.

https://docs.openshift.com/container-platform/3.6/dev_guide/managing_images.html#accessing-the-internal-registry
Comment: Thanks Graham, I'm still completely unclear how to actually *create* an image stream. If I use oc new-app  I get a successful *deployment* which runs up a pod etc. But this doesn't actually create an image stream and so I cannot set a 'scheduled' policy on anything. Any attempts I have made to manually create an image stream to reference my external image have failed. Any pointers on how to do this would be great!
Comment: Running ``oc new-app`` with an image on an external registry should still create an image stream. What do you get for ``oc get is``? Also, go work through the interactive tutorial at https://learn.openshift.com/ on deploying from an image. It provides various examples and discusses image streams and how one can create one from an image before deploying it.
Comment: The question is though whether the cluster is even been configured to allow use of the scheduled checks for image updates.
Comment: oc get is
No resources found.
Comment: Only way there couldn't be an image stream is if you were creating raw resource objects for the deployment configuration and hardwiring external image registry image reference in the deployment configuration. So if you did that and passed the raw object definitions to 'oc create'. If using 'oc new-app image' it should create an image stream. You can confirm what it would create using the '--dry-run -o json' option to see.
Comment: @slowko I am going through similar issue. Did you figure out how? Thansk!


Windows Store App Push Notifications via Azure Service Bus

Question: I'm trying to send a push notification to my Windows Store App developed in my laptop using VS 2013 Express. It's been few days now but I cannot figure out why I get "The Token obtained from the Token Provider is wrong" error. I use Windows Azure Notification Bus for this purpose. I use VS Server Explorer to send test notifications. I can see my laptop is being registered as a device in the Device Registration tab too. I tried the Azure portal as well, but the same error. However, when I try to connect to Service Bus Explorer 126.96.36.199 providing the connection string it throws below error.
<21:47:14> Exception: The remote server returned an error: (401) Unauthorized. Manage claim is required for this operation..TrackingId:c0c4fea2-08bc-4def-964c-ec6e690b7551_G45,TimeStamp:10/12/2014 4:17:11 PM. Method b__7e: retry 2 of 10.
FYI: I'm following below article step by step.
http://azure.microsoft.com/en-us/documentation/articles/notification-hubs-windows-store-dotnet-get-started/
Please help me. Thank you.
Mahesh
Answer: Looks like token is just expired. Make sure you obtain token each time application starts. In terms of the article you refer it means you  should call method InitNotificationsAsync() to do it. Here is that method:
<code>private async void InitNotificationsAsync()
{
    var channel = await PushNotificationChannelManager.CreatePushNotificationChannelForApplicationAsync();

    var hub = new NotificationHub("<hub name>", "<connection string with listen access>");
    var result = await hub.RegisterNativeAsync(channel.Uri);

    // Displays the registration ID so you know it was successful
    if (result.RegistrationId != null)
    {
        var dialog = new MessageDialog("Registration successful: " + result.RegistrationId);
        dialog.Commands.Add(new UICommand("OK"));
        await dialog.ShowAsync();
    }

}
</code>


how to consume kafka topics in sequence in spring boot

Question: i have a problem i made an apache kafka consumer in spring boot to consume 3 different topics. but I need to consume all the data from the first topic first and then consume the data from the following topics, is there any way to do that? or will you always read them the same way?
@Component
public class KafkaTestListener {

@KafkaListener(topics = "${message.topic.name}", groupId = "${message.group.name}")
    public void listenTopic1(String message) {....}

@KafkaListener(topics = "${message.topic.name2}", groupId = "${message.group.name}")
    public void listenTopic3(String message) {....}

@KafkaListener(topics = "${message.topic.name3}", groupId = "${message.group.name}")
    public void listenTopic3(String message) {.....}

}

Answer:
Give each listener an id; set autoStartup to false.
Set the container property idleEventInterval to some value.
Add an @EventListener method to receive ListenerContainerIdleEvents - see https://docs.spring.io/spring-kafka/docs/2.5.3.RELEASE/reference/html/#events and https://docs.spring.io/spring-kafka/docs/2.5.3.RELEASE/reference/html/#event-consumption
Use the KafkaListenerEndpointRegistry to start and stop the containers (via id) as needed.


Vowpal Wabbit hash collision works better than L1 as regularization

Question: I have VW classification model which i wanted to inspect for number of features and number of hash collisions. I trained it and test it on different datasets. The datasets conatins more than 400k features, so with 18bit VW space, it is possible to save only 260k without collisions.
Then, to inspect it, i created two readable models: one with parameter --readable_model, to get number of all hashes, second one with parameter --invert_hash to get number of all features, even those which were in hash collisions. There were 425k features, and 208k hashes (not 260k, because i used some filtering with --keep argument, and as i understand it, vw saves to hash table also ignored namespaces). Then i measured quality of model, with ROC_AUC, MCC, and Average Precision. Results were 0.903, 0.564, 0.591.
As you can see, hash collision is huge, so i decided to increase bit space with -b argument value set to 24. Then i trained model again. Now, there were 425k features, and 425k hashes = no collisions. However, results on same metrics, were worse (0.902,0554,0.587).
For now everything looks like there was big overfitting with 24bit space, while 18bit prevented model against overfitting better - that would be good explanation why results were better on test dataset with 18bit model.
But then i decided to reduce number of features on 24bit model with L1 regularization. I was playing with it for a while, and when i got model which had 208k hashes and 208k features i was very suprised, that its results were still worse, than results of 18bit model with same number of hashes. It was 0.901,0.584,0.552.
This makes me to believe, that random hash collisions, even for huge portion of features, works like better regularizer, than L1 regularization. How is this possible?
Answer: What is going on here (bottom line)?
What you experience is a strong sign of over-fitting. Details follow.
How is this even possible?
Both regularization and random hash-collisions are discounting mechanisms for a subset of model features.  They pick some features, and make them less important (or completely discounted) in the model.
But that's where the similarity ends.  The two discounting mechanisms are very different in:

The subset of features they discount (specific vs random)
The discounting method (complete vs partial-blend)
The discounting direction and magnitude

Specific vs random discounting
L1 regularization (--l1 ...) picks very specific weights (the ones nearest zero vs the norm) whereas random hash collisions "picks" random weights, some of which may be large.
Complete vs partial-blend discounting
L1 regularization completely trims/removes the weights it selects, whereas random-hash collisions create blends with other features. In one sense (effect-mixing) blends are similar, though not identical, to what vw feature-crossing with -q <XY> does.
Direction & magnitude of discounting
Unlike specific selection by absolute value of weight, random selection can affect weights of any magnitude.
Moreover, a feature might be a bad feature when considered alone, but actually contribute when considered in combination with another feature. One example would be a blending of two bad features, one with a positive and one with a negative weight. By partially canceling each other they might result in a plausibly good feature that is somewhat correlated with the target label. IOW: feature blending can sometimes turn a bad effect into a good effect.
How can discounting help (or hurt) a model?
This is very common in Machine Learning, and especially in big-data problems. Whether feature selection, trimming, or blending improves accuracy is data-dependent.
If it happens to discount a "bad" feature (one that agrees with the training-data, but doesn't help in generalization) it makes the model better. However, if we discount a good, well-generalizing feature, it would make the model appear worse in testing (out of sample data).
Related: the idea of randomly discounting or dropping features, even potentially important ones, has been proven to be a a powerful and beneficial technique in deep neural-nets (NNs), where it is called dropout. dropout has become the de-rigueur method to avoid over-fitting in deep learning.
Bottom line
Creating good models takes practice. When you have a very large number of features, over-fitting due to random effects (both small and/or large weights) is possible. This over-fitting needs to be avoided.
There are many ways to avoid over-fitting. Regularization is just one specific way to reduce over-fitting. Sometimes, a more significant, random approach, which affects all features, not just those with low weights, might be beneficial overall.
When this happens, it is a hint that the number of features may be too large, and you may be over-fitting them.
Generally I would suspect any model where the number of examples (data-set rows) used to train, is not much larger than the number of features (distinct data-set columns). If you have hundreds of thousands (10^6) of features, you should probably have 10^12 (trillions) of examples to avoid over-fitting.
Another thing I would do with a large number of features, is to randomly-shuffle the order of examples, and blend several models to ensure that a specific order doesn't cause over-fitting. Online learning tends to overweight early examples over later ones due to learning rate decay.


Get div ID of element attached to Fine Uploader

Question: I am using fineuploader version 3.2. I have multiple instances of fineuploader embedded on a single HTML page like this

And this is how I attach fineuploader
$('.uploader').fineUploader({
request: {
    endpoint: './uploadfile',
    params: {
    imageid: function() {
        return this.id;
     }
    }
},
multiple: false,
validation: {
    allowedExtensions: ['jpeg', 'jpg', 'gif', 'png'],
    sizeLimit: 551200 // 50 kB = 50 * 1024 bytes
},
text: {
    uploadButton: 'Changepic'
}
}).on('complete', function(event, id, fileName, responseJSON) {
if (responseJSON.success) {
    imgs[this.id]="data:image/jpg;base64,"+ data;
}
});

I want to pass the id of the div attached to fineuploader to the fileupload request. However I see that 'undefined' is returned wherever 'this.id' is referenced. I also tried using this.attr('id') instead and get an error "Uncaught TypeError: Object # has no method 'attr' ".
I found a similar question here(FINEUPLOAD pass the ID as parameter) however the solution seems too crude. Can someone suggest a better solution, rather than repeating the same code?
Answer: You can loop through the elements and call fineUploader on each, retrieving each ID in the process:
$('.uploader').each(function() {
  var uplId= $(this).prop("id");
  $(this).fineUploader({
    request: {
        endpoint: './uploadfile',
        params: {
        imageid: function() {
            return uplId;
         }
        }
}
...
});


How can I learn Soukous guitar?

Question: I really like the sound of Soukous guitar playing, and would like to learn a bit of that. What would be good ways of doing this? I'm not great at picking at all, I'm a happy amateur chord strummer, so both newbie resources as well as advanced stuff for getting better later is appreciated. Online, books or DVD's are all OK.
Comment: You'll need to be able to pick. Whether you practice on western-style picking, or go straight in and learn to pick as you learn Soukous patterns, probably doesn't matter. Maybe practice some bluegrass rolls? (Google it)
Answer: For every genre, the method is the same. You listen to a lot of the music, and you try to play along. You transcribe your favorite bits and figure out the structure, the chords and the scales. There are some genres where a lot of those bits are well-established. Blues, for example, is the 12-bar, the minor pentatonic, bending, etc. etc. and I don't know the music of the Congo well enough to give you that head start, but if you're going to learn a genre, that's how you do it.

How can I learn Soukous guitar?


Why does software-synthesized MIDI music sound so bad?

Question: It seems like converting MIDI into sound should be simple enough: record all possible notes, and then just superimpose the individual sounds according to the MIDI data (offsets, duration and volume), and compress the result into MP3 or send it to the sound card.
Yet, the MIDI synthesizers I tried (timidity on Ubuntu, and a few others I don't recall) sound distorted and overall much worse than MP3 recordings of actual musicians.
Why is that?
Update: I just tried Fluid Synth on Ubuntu, which uses about 100MB worth of "sound fonts", compressed. It seems to be vastly better, but still rather bad.
Comment: I think you're vastly over-simplifying.

One off-the-shelf midi file played on another off-the-shelf sample set has little hope of matching the nuance even an unskilled creator could make using the same source for both creation & playback.
Comment: The software instruments you tried aren't in the appropriate weight class. A decent "sound font" for a single instrument generally takes up a DVD or a dozen DVDs, and costs comparably to a decent physical instrument.
Comment: Check out professional MIDI software, like [trillian](https://www.youtube.com/watch?v=ZYU93OtvzjA) bass, [Steven Slate Drums](http://www.stevenslatedrums.com/products/platinum/) or [Vienna Symphonic Library](https://www.vsl.co.at/en/Products/Instruments).
Comment: Sound cards don't usually decode MP3. MP3 is decoded into PCM before being sent to the sound card, and software synths just go right to PCM (or samplers just read from PCM files).
Comment: "It seems like converting MIDI into sound should be simple enough" Could OP be [a physicist](https://xkcd.com/793/)? :-)
Comment: Here's an example of what I understand is one of the most popular Virtual Studio Technology's (VST's) piano packs. It sounds terrible to me: https://www.youtube.com/watch?v=I69waNu1Nd4
Answer: MIDI is not sound.  The MIDI specification does not dictate what any instrument sounds like, it's up to the synthesizer to generate the sound.  Free synths sound like crap, but good ones can sound as good as the creators can make them.  For example, the Pirates of the Caribbean soundtrack is entirely synthesized, yet most people don't even realize it.
Comment: "MIDI is not sound." – indeed, and that's really the whole answer to the question. — I don't quite agree with how you contrast free/bad synths with good ones – truth is, some of the more expensive ones make it more feasible to achieve results close to a human performance on a real instrument; but ultimately the quality depends a lot on how well the arranger knows the particular synth. Professional synth-only productions are usually _not_ just MIDI, at least not in the traditional GM sense: there's a lot of dynamic tweaking of all the parameters required.
Comment: According to [Wikipedia](https://en.wikipedia.org/wiki/Pirates_of_the_Caribbean:_The_Curse_of_the_Black_Pearl_%28soundtrack%29) there were real musicians (mixed with sequenced sounds) for the first Pirates soundtrack, at least.
Comment: I'm sorry, on a second thought, where *the hell* does that Wikipedia article say that (part of) the soundtrack was sequenced? It says that Zimmer first produced a sequenced demo, but that is 100% standard practice, especially with "composers" who can't orchestrate (an orchestrator and a copyist will take it from there). You don't book an orchestral session only to hear the director telling you "hey, actually this sucks, we're not using this".
Comment: (And FWIW, I don't think the final soundtrack of Pirates of the Carribean is entirely synthesized (though Zimmer apparently rendered a full-synth preview); at least some of the less bombastic passages are played on real instruments.)
Comment: Indeed - there are MIDI controllers with many sources of modulation modeled after real instruments, so players can impart all kinds of expression that is then translated into computer code.  With even a very modest synth this can result in totally unique and ultimately expressive music.  It's all in how you use it.  A typical MIDI file from the internet contains nothing but note values and durations, whereas there's a LOT more information than that in most music performances.
Comment: *In theory there's no reason you couldn't translate every single bit of information from a music performance into MIDI if you had working models of the acoustics and sound generators.
Comment: I *think* it is hugely common in TV to use sequencers and then layer a real violinist and a woodwind player on top, just to make it suck less. But Hollywood? Heh, that Zimmer guy is weird.
Comment: @SomeDudeOnTheInterwebs: I interpreted [this part](https://en.wikipedia.org/wiki/Pirates_of_the_Caribbean:_The_Curse_of_the_Black_Pearl_%28soundtrack%29#Orchestration) in that way. My comment was intended to disprove that it was "entirely synthesized", no need to get riled up...
Comment: Yes, you're right. It mentions sampled drum beats and stuff. Although I don't see citations for that paragraph so it may be something the author made up.
Answer: MIDI is only a specification for what instrument (patch) to use, what notes to play, how long and loud to play a note and other things like tempo, time signature and text lyrics. The concept is very similar to how an old player piano works. The midi data is like the piano roll, the sound you hear is from the physical sound produced when the hammers strike the strings. Likewise, the way music sounds when played via the MIDI specification is up to the quality of the patch sounds.  Its possible to even hook up a MIDI connection to a machine that played physical acoustic instruments and that could sound great. Likewise I could give you a set of my singing for every instrument patch and it would sound awful.  It all depends on the patch set you have or the instrument that you've hooked up the MIDI sequencer or controller too.
An example of how the sound quality can be different is back in the 90s there was a game called Doom II which used MIDI files for the music. Depending on what sound card you had the game would sound better or worse because of the quality of the instrument patches that your sound card included.  It was even possible to hook up MIDI cables from your computer to an external synthesizer and use the sound synthesis there. One time we hooked it up to a Kurzweil K2000 and it was kinda like having a live band playing the Doom II music for you while playing the game.
As a protocol and file format, MIDI has been surprisingly adept and has required little revision over the last 30+ years. Although in the beginning of 2015 the MIDI Manufacturers Association announced that there is a new MIDI "HD Protocol" under development.
Comment: However, the MIDI standard is under review; the idea is first to increase resolution of various continuous controls like velocity and modulation, and second to standardize some uses of MIDI for software control, such as HUI, which currently use a more or less proprietary combination of NRPNs. All this, while maintaining backward compatibility with the 30+ years of gear and software designed to use MIDI "1.0".
Comment: There is a Wurlitzer organ in Phoenix (in a pizza place of all things) that does exactly what you describe; a complex software/hardware MIDI-based system controls not only the several dozen actual organ stops the thing has, but an additional complement of instruments from pianos and xylophones to various woodwinds and brass and even a couple guitars.
Comment: This might also be of interest. I made a video for climagic a couple years ago that demonstrates how compact the MIDI file format can be: https://www.youtube.com/watch?v=zRF1S-8P6_k  I was able to fit a short MIDI file and its decoder into a 140 character tweet.
Comment: "MIDI has been surprisingly adept and has required little revision over the last 30+ years" welll... I'd put it less flatteringly: _people have developed ever more workarounds to compensate for the inadeptnesses of MIDI, because nobody could be bothered to implement a proper replacement standard_.
Answer: MIDI is just a stream of instructions, like:

"Tell channel 1 to turn on note 60"
"Tell channel 2 to turn off note 72"
"Tell channel 3 to set parameter 1 to value 231"

There is a set of conventions such as:

Channel 1 is piano, 34 is electric bass, etc.
Parameter 1 is modulation, 7 is volume, 64 is sustain, etc.

This is called General MIDI (Wikipedia).
General MIDI was specified later than MIDI itself. Users of MIDI are free to use any instrument on channel 1. You don't even have to control an instrument -- MIDI can be used to control lights, or anything else.
If you get a MIDI file from the internet, it's likely to be written for GM, but sometimes it's not.
Imagine the simplest case of a MIDI file - a recording of one person playing a piano part on a keyboard. It's a bit like getting a robot to exactly mimic the movements of a pianist, then putting the robot at the keyboard of a different instrument.
If the robot is at a good piano, it'll probably sound good. This piano might not have exactly the same response of key velocity to volume, so expression will sound subtly different. But it'll sound pretty good.
If you sit the robot at a toy keyboard, it's likely to sound awful. A good musician could probably coax a decent sound out of that toy keyboard. But here the musician has "played" a nice instrument and had their input moved to a different instrument, so they can't take account of its limitations.
Then there's all the other parameters. What effect does "modulation 127" have on the sound? How much does "100 pitch bend" change the pitch? Even when GM specifies these, not all implementations get it right.
Then multiply it by a number of instruments, and take into account that if you were mastering your own piece of music you'd be tweaking aspects that GM doesn't specify, and it becomes obvious why MIDI playback can sound terrible.
A high quality "sound font" -- the set of things that translate events into sounds -- can improve matters, but even then you don't reach the standards of a full professional recording.
Many electronica records are made entirely from MIDI sequenced instruments -- but the musicians are also carefully configuring their instruments and effects to get the sound they want.
There is great fun to be had sending MIDI files to the built-in sounds of the cheapest sound card you can find, and playing "guess the tune". Or swapping instruments around.
Answer:
It seems like converting MIDI into sound should be simple enough
  record all possible notes, and then just superimpose the individual sounds
  according to the MIDI data (offsets, duration and volume), and compress the > result into MP3 or send it to the sound card.

And this kind of thinking (the wrong kind, that is) gives you exactly the results you hear.
First and foremost, you have two main problems here.
a. Crappy sound chains.
b. Crappy sequences
Let's see why.

On instruments other than piano, no two notes sound alike. Say, guitar. There are a million ways to change the sound of a note on a guitar. Choose between pick, fingers, fingerpicks and you have different sounds. Choose where to fret the same note and you have different sounds. - or registers.
Electric instruments and effects in widespread use in music production are often nonlinear. If you input two notes into an overdriven guitar amp, the ouput will not be the sum of the single notes.
Even good old acoustic pianos have so-called sympathetic resonance.

You address this with a good ad hoc instrument+effect chain, solving problem a - but timidity doesn't even qualify.
I'm saying ad hoc because to play Limp Bizkit you need a different setup than you need to play gospel, in real life and on the computer too.
You need to chain the right effects, the right samples, etc.
You don't want to play back your Limp Bizkit thrashing midi part with a piccolo snare sample, do you?
If you want to do it properly, you also want your virtual instruments to be able to interpret articulation data (picking position, etc), usually fed via CC.
This is very tricky.
So tricky that in fact people just prefer to record a real guitar.

Articulation and expression.
Even on something comparatively non-tricky like a piano, a real pianist does not play in a robotic fashion like a badly programmed sequencer where every note is exactly 1/8th, 1/16th and glissandos or "bendings" are perfectly linear.
The folks who make the midifiles you can find on the Internet are usually pretty lazy and provide you with poor transcriptions to boot, where even trivial articulation is omitted.

If you just take the output from a sequencer the performance will sound robotic and devoid of expression, unsurprisingly, even if you get articulation and ornamentation right.
But if you record the notes played by a real player with, say a MIDI keyboard and a MIDI guitar, the results will be very different.
This takes care of b.
And now I'm dropping the bomb, friend.
Most of what you hear on the radio these days is MIDI.
Recording and editing equipment does use MIDI and most backing tracks are either sequenced or recorded via MIDI.
Even the autotune for vocal track probably is driven by MIDI events.
And if you go to a live concert they probably use MIDI timecode and events to drive the light show and the effect chain on the guitarist's pedalboard.
Thing is, Celine Dion does not use timidity for her backing tracks and her producers spend some time on the sequences.
Answer: The free sound fonts typically installed with Timidity are of very mixed quality.  Routing playback through some vintage Midi expander of good quality will greatly improve results, routing it through some reasonably current good quality offering will give some more improvements at much more portable hardware size.  Naturally, the analog paths involved here will decrease quality again.
You can improve Timidity's action somewhat by working with other sound fonts and adjusting its parameters, but it does not really compare.
There may be some commercial software MIDI expanders doing a good job but their expected useful life time is a lot less than hardware expanders (hardware expanders are likely to continue running in 20 years, software expanders not unless you keep some hardware+OS around with them) and there is a lot of variation regarding pricing and quality.  And they will not likely run on GNU/Linux, and may get on your nerves with DRM/licensing schemes.
Feeding a good MIDI expander with good MIDI data will actually produce sound that's rather useful for a number of purposes (partly depending on what the MIDI expander specializes in).  Some instruments with continuous and "analog" controls like solo strings are pretty hard to do convincingly particularly when not using continuous MIDI controllers (analog pedals, MIDI accordion or wind instrument with pressure sensor etc): string sections work better in that regard.
Answer: MIDI-based music will be as good as its designer. It's not that MIDI-based music sound quality is inherently bad; the final results depend on whoever designed the MIDI, and whoever is using it. In other words, if it sounds bad it is because of the people involved with it. Even if you didn't design it and you are just using it you are responsible of its output quality since you are deciding which device with which settings will be used.
There's two distinct areas in MIDI music design:

The MIDI file. If it was captured the quality will depend on the performer. If it was programmed the quality will depend on the designer/programmer.  There's no sound involved here (yet), so you can think of this step as the quality of the carving of the instructions that will later be interpreted by a device that understands MIDI.
The sound design. Now the instructions need to be interpreted by something like a sampler, a drum machine, a synth. The quality here depends in both the device you are using and your skills as a sound designer.

If you don't like the output then something went wrong in these steps.
You don't give much details, so it's hard to know for sure what's wrong in your case. I'm not familiar with Timidity, but if it's a synth (subtractive, FM, etc) you are the one programming it, so the quality of the output is a reflection of your skill. If Timidity is a sampler, maybe the samples are low quality. Maybe the distortion comes from clipping, it can be many things.
There shouldn't be distortion, so try using a good quality synth and/or a good quality sampler with good quality samples.
Answer: there are 2 possible problems here.
1) The midi file is created straight from sheet music.  every note starts at exactly the right time and lasts the exact duration and all the note velocities are 100.  That makes for a song with absolutely no feel.  If you have a musician play the song with expressive velocities, tempo variations, arrangement improvements then it'll sound much much better.
2) The synth you're using may be lame.  Timidity uses soundfonts I think.  That's a pretty old tech where the sample rate is not always CD quality, the samples are short loops, and not very many samples per instrument.  Modern softsynths usually have a ton of overlapping samples per instrument.
Regardless, if you're a musician, that might be the only sheet music you'll find of your favorite pop song.  I find those things invaluable.
Answer: They sound 'bad' because they are not naturally created sounds. They are not the sounds of acoustic instruments, which have a much wider array of characteristics as compared to just some particular frequencies sounding for a particular time.
Comment: Wrong and easily falsifiable. Also, the only "naturally created sounds" out there is birdsong, Mother Nature doesn't make pipe organs.


Widget settings hook for Drupal 7

Question: What hook is the equivalent in Drupal 7 for hook_widget_settings($op, $widget), used in Drupal 6?
What hook is the equivalent in Drupal 7 for hook_widget_settings_save(), used in Drupal 6?
Answer: You can use hook_field_widget_info().
Here you can give the widget settings.
Comment: Also the Examples project has a sample field implementation with some settings. http://drupal.org/project/examples

Widget settings hook for Drupal 7


Facing Issues Sending FormData with 'multipart/form-data' In fetch API

Question: I've this form with pug code
form.form.form-user-data
 .form__group
   label.form__label(for='name') Name
   input#name.form__input(type='text', value=`${user.name}`, required)
 .form__group.ma-bt-md
   label.form__label(for='email') Email address
   input#email.form__input(type='email', value=`${user.email}`, required)
 .form__group.form__photo-upload
   img.form__user-photo(src=`/img/users/${user.photo}`, alt=`Photo of${user.name}`)
   input.form__upload(type='file' accept='image/*' id='photo' name='photo' )
   label(for='photo') Choose new photo

This is the Backend with Node&Epress
const multerStorage = multer.memoryStorage();

const multerFitler = (req, file, cb) => {
  if (file.mimetype.startsWith('image')) {
    cb(null, true);
  } else cb(new AppError('Not an image! Please upload an image.', 400));
};
const upload = multer({
  storage: multerStorage,
  fileFilter: multerFitler,
});
exports.uploadUserPhoto = upload.single('photo');

exports.resizeUserPhoto = (req, res, next) => {
  if (!req.file) return next();

  req.file.filename = `user-${req.user._id}-${Date.now()}.jpeg`;

  // console.log(req.file);

  sharp(req.file.buffer)
    .resize(500, 500)
    .toFormat('jpeg')
    .jpeg({ quality: 90 })
    .toFile(`public/img/users/${req.file.filename}`);

  next();
};
function filterObj(obj, ...allowedFields) {
  const newObj = {};

  Object.keys(obj).forEach((key) => {
    if (allowedFields.includes(key)) newObj[key] = obj[key];
  });

  return newObj;
}
exports.updateMe = catchAsync(async (req, res, next) => {
  if (req.body.password || req.body.passwrodConfirm)
    return next(
      new AppError(
        'This route is not for password updates. Please use /updatePassword route',
        400,
      ),
    );

  const filterdBody = filterObj(req.body, 'name', 'email');
  if (req.file) filterdBody.photo = req.file.filename;

  const updatedUser = await User.findByIdAndUpdate(req.user._id, filterdBody, {
    new: true,
    runValidators: true,
  });

  res.status(200).json({
    status: 'success',
    data: {
      user: updatedUser,
    },
  });
});

This is my route for handling the request
router.patch('/updateMe', uploadUserPhoto, resizeUserPhoto, updateMe);

This is my Frontend code
async function updateData(endpoint, body) {
  const res = await fetch(`http://127.0.0.1:3000/api/v1/users/${endpoint}`, {
    method: 'PATCH',
    body,
  });
  const data = await res.json();

  console.log(data);
}

updateDataForm.addEventListener('submit', (e) => {
    e.preventDefault();
    const form = new FormData(updateDataForm);
    form.append('name', updateNameInput.value);
    form.append('email', updateEmailInput.value);
    form.append('photo', updatePhotoInput.files[0]);

    updateData('updateMe', form);
});

The console.log(data) Gives me this error
{
code: "LIMIT_UNEXPECTED_FILE"
field: "photo"
message: "Unexpected field"
name: "MulterError"
}
This error comes from the browser but Postman works very fine
I tried to add headers to the fetch options object
headers: { 'Content-Type': 'multipart/form-data' }

but this is another error which is
{
message: 'Multipart: Boundary not found'
}

In Postman 'multipart/form-data; boundary=calculated when request is sent'
It's just the error happens when the URL is requested from the browser I've asked Chatgpt but nothing
Comment: Please trim your code to make it easier to find your problem. Follow these guidelines to create a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example).
Answer: //Why are you adding 'updateDataForm' in FormData try after removing it.
async function updateData(endpoint, body) {
      const res = await fetch(`http://127.0.0.1:3000/api/v1/users/${endpoint}`, {
        method: 'PATCH',
        body,
      });
      const data = await res.json();

      console.log(data);
    }

updateDataForm.addEventListener('submit', (e) => {
        e.preventDefault();
        const form = new FormData();
        form.append('name', updateNameInput.value);
        form.append('email', updateEmailInput.value);
        form.append('photo', updatePhotoInput.files[0]);

        updateData('updateMe', form);
    });

Comment: I saw this at the MDN Docs
Comment: I removed it as you said and everything just works very fine
Thanks, alot
Comment: No problem and happy Coding


drush xmlsitemap-regenerate domain is incorrect

Question: Whenever I run:
drush xmlsitemap-regenerate

on our live server, the domains in the sitemap all refer to:
http://default

Only when I generate the sitemap from the Drupal admin backend, do I get the correct sitemap.xml with the correct domain setup.
I need to have the drush command working correctly, as we're using it with a Capistrano deploy script.
Can I somehow pass the domain name to the drush command?
Answer: Did you set your default base URL under the 'Advanced' fieldset. in admin/config/search/xmlsitemap (D7 version) ?
Answer: Add --uri to your command.
Example:
drush --uri=example.com xmlsitemap-regenerate


Passing coordinates of selected Annotation to new View Controller Swift 2.0

Question: I am trying to pass the coordinates, title, and subtitle of a selected MKAnnotation from ViewController1 (VC1) to ViewController2 (VC2) when the DetailDisclosure button of rightCalloutAccessoryView is tapped. I have a segue from VC1 to VC2 with the identifier viaSegue. I have a label with identifier viaSegueLabel in VC2 where I want to display the coordinates as a String.
The function that customizes the MKAnnotation's call out so that it displays a DetailDisclosure button in the rightCalloutAccessoryView looks like:
// Customize Annotation Callout
    func mapView(mapView: MKMapView, viewForAnnotation annotation: MKAnnotation) -> MKAnnotationView? {
        // 1
        let identifier = "Capital"

        // 2
        if annotation.isKindOfClass(Capital.self) {
            // 3
            var annotationView = mapView.dequeueReusableAnnotationViewWithIdentifier(identifier)

            if annotationView == nil {
                //4
                annotationView = MKPinAnnotationView(annotation:annotation, reuseIdentifier:identifier)
                annotationView!.canShowCallout = true

                // 5
                let btn = UIButton(type: .DetailDisclosure)
                annotationView!.rightCalloutAccessoryView = btn
            } else {
                // 6
                annotationView!.annotation = annotation
            }

            return annotationView
        }

        // 7
        return nil
    }

The function that takes the user from VC1 to VC2 when the DetailDisclosure button is tapped looks like:
// When righCalloutAccessoryView is tapped, segue to newView
func mapView(mapView: MKMapView, annotationView view: MKAnnotationView, calloutAccessoryControlTapped control: UIControl) {
    self.performSegueWithIdentifier("newView", sender: view)
}

And the function I think I need to implement to accomplish this looks like:
// Pass data to newView
override func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject!) {
    if (segue.identifier == "newView") {
        let destViewController:BusStopSettingsViewController = segue.destinationViewController as! BusStopSettingsViewController
        destViewController.viaSegue = // not sure how to reference selected Annotation here
    }
}

In the last line of prepareForSegue(), I need to reference the MKAnnotation that is currently selected. Is there a method built into Swift that lets me do this, or should I make the Annotation global?
Answer: Was able to figure it out in case anyone in the future needs to implement something similar.
self.performSegueWithIdentifier("newView", sender: view)

programmatically segues your program to the View Controller (VC) that is connected to the VC you are currently in through a segue with identifier "newView". Before it completely segues, the program calls prepareForSegue(). This function is where you will handle sending information to the VC you are segueing to. My issue was, I didn't know exactly what I was sending (in terms of type, variable name, etc.). If you notice, prepareForSegue() and self.performSegueWithIdentifier("newView", sender: view) both have a parameter sender. What you send using performSegueWithIdentifier() will be passed into prepareForSegue() and will be received in your destinationViewController by a variable with the name viaSegue. This isn't a standard name, it's just what I chose to name that variable and if you study the code above, you'll see where it's used and how it works.
So I wanted to send information about an MKAnnotation I had tapped. So, I needed to send an Object of type MKAnnotationView to my receiving VC "BusStopSettingsViewController" (BSSVC). In BSSVC, I needed a variable called "viaSegue" of type MKAnnotationView. To send BSSVC an object of type MKAnnotationView I needed to do
override func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject!) {
    if (segue.identifier == "newView") {
        // pass data to next view
        let destViewController:BusStopSettingsViewController = segue.destinationViewController as! BusStopSettingsViewController
        destViewController.viaSegue = sender as! MKAnnotationView
    }
}

Notice how viaSegue is specified as the variable that'll be receiving this object.
Hope this helps!


How to surround the cells with just one pixel

Question: I have a grid control and I'm setting a style for it.
In the border of cells, I want one pixel around each cell, but it appears two pixel because one from the left and one from the right.
The CSS:
<code>.RadGrid_MyCustomSkin .rgRow td, .RadGrid_MyCustomSkin .rgAltRow td
{
    border: 1px solid #F0D88C;
}
</code>
Any help !!
Answer: Use the border-collapse property.
Answer: give to td and table like this
<code>table, td
{
    border-color: #F0D88C;
    border-style: solid;
}

table
{
    border-width: 0 0 1px 1px;
    border-spacing: 0;
    border-collapse: collapse;
}

td
{
    margin: 0;
    padding: 4px;
    border-width: 1px 1px 0 0;

}
</code>
Answer: You can set a border for one side only, maybe this will point you in the right direction :)
<code>.RadGrid_MyCustomSkin .rgRow td, .RadGrid_MyCustomSkin .rgAltRow td
{
    border-left/*or -right*/: 1px solid #F0D88C;
}
</code>
Comment: Thanks, but it'll not cover the first cell.


NDB/DB NoSQL Injection Google Datastore

Question: Is there any SQL injection equivalents, or other vulnerabilities I should be aware of when using NoSQL?
I'm using Google App Engine DB in Python2.7, and noticed there is not much documentation from Google about security of Datastore.
Any help would be appreciated!
Answer: Standard SQL injection techniques rely on the fact that SQL has various statements to either query or modify data. The datastore has no such feature. The GQL (the query language for the datastore) can only be used to query, not modify. Inserts, updates, and deletes are done using a separate method that does not use a text expression. Thus, the datastore is not vulnerable to such injection techniques. In the worst case, an attacker could only change the query to select data you did not intend, but never change it.
Comment: If you're not using GQL - is there any way you can change the query?
Comment: No. If you use the function-based filters, there's no way to change the query. This is the equivalent to using parameterized queries in SQL.

NDB/DB NoSQL Injection Google Datastore


How can one intuitively understand the notion of quantile preferences in stochastic cooperative game theory?

Question: Background
For my thesis, I'm trying to wrap my head around a relatively small field within mathematics called "Stochastic Cooperative Game Theory". To that end, I've read a few papers on the subject, including Cooperative games with stochastic payoffs (Suijs et al., 1999, link) and Convexity in stochastic cooperative situations (Timmer et al., 2005, link).
In classical cooperative game theory, the payoffs (rewards) to players are deterministic. The players know beforehand what the benefits are of forming coalitions with the other players. In stochastic cooperative game theory, however, uncertainty is introduced. This means players do not know for sure what payoff they will receive: the payoffs are modeled as random variables.
The players try to decide which payoffs based on random variables are most beneficial to them. However, there many ways to define "most beneficial". In order to determine which random variable is preferred by player $i \in N$, Timmer et al. introduce the notion of a preference relation $\succeq_{i}$ (on page 4). For some stochastic payoffs $X$ and $Y$ (where $X$ and $Y$ are nonnegative random variables with finite expectation), they write $X \succeq_{i} Y$ if player $i$ weakly prefers the stochastic payoff $X$ to receiving $Y$. Furthermore, $X \succ_{i} Y$ means that player $i$ strictly prefers $X$ to $Y$.
On page 5, the same authors give an example of such a preference relation. The preferences of player $i$ could be such that $X \succeq_{i} Y$ if and only if $E(x) \geq E(Y)$, where $E(X)$ is the expectation of $X$. I suppose I understand this type of preference. As a player, you'd want to maximize the expectation of your payoffs $(*)$.
However, I don't understand the second type of preference. It involves quantiles of random variables. They define $ u_{ \beta_{i} }^{X} = \sup \{t \in \mathbb{R} | \Pr \{ X \leq t \} \leq \beta_{i} \} $ be the $\beta_{i}$-quantile of $X$. Furthermore, they define the utility function $U_{i}$ by $U_{i} (X) = u_{ \beta_{i} }^{X} $. They say that a player has quantile-preferences if $X \succeq_{i} Y$ if and only if $U_{i} (X) \geq U_{i} (Y)$.
Questions

Could you provide me with an intuitive explanation of the quantile preference relation? (Perhaps similar to the one I gave myself in $(*)$, but a bit longer/detailed?). Can you give an example with actual distributions?

Suijs et al. introduce a similar preference relation in their paper on p. 197. They add that agent $i$ is more risk averse than agent $j$ if and only if $\beta_{i} < \beta_{j}$

Could you explain and motivate this definition of risk aversiveness? And how does it relate to this type of risk aversiveness?

Finally, I tried understanding the quantile preference relation by comparing two cumulative distribution functions. These are the cumulative exponential distribution with $\lambda = 0.5$ (distribution $X$) and $\lambda = 1.5$ (distribution $Y$), respectively. Suppose we take $\beta_{i} = 0.3$. As $E[X] = 1/0.5 = 2 > 0$ and $E[Y] = 1/1.5 = 2/3 > 0$, we need to find the utility with $1-\beta_{i} = 0.7$. Then we get the following picture:

(adapted from an image on Wikipedia). I suppose now we have $u_{ \beta_{i} }^{X} \approx 2.3$ and $u_{ \beta_{i} }^{Y} \approx 0.9$, so we have $U_{i} (X) \geq U_{i} (Y)$

How do we interpret this result? Is there an intuitive explanation?
Answer: N.B. I will occasionally call random variables "lotteries" or "gambles". This is standard in the literature, so I hope you are comfortable with the terminology.
Explaining the Quantile Preference Relation
Quantile preferences are similar to expected-value preferences ($ X \succeq_i Y \iff E(X) \ge E(Y) $) in that they both depend only on a summary statistic of the distribution of a random variable.
The only difference is that while expected-value preferences depend only on the means of the distribution, quantile preferences depend on a specific quantile of the distribution.
Observe that the $\beta_i$-quantile is essentially the inverse of the CDF of the distribution. This means that the $\beta_i$-quantile of a random variable $X$ is the largest value $u_{\beta_i}^X$ such that $\Pr (X \le u_{\beta_i}^X) \le \beta_i$. (For continuous distributions, you can replace the $\le$ with $=$.)
For example, the $\frac 1 2$-quantile is the median of the distribution. If an agent has $\frac 1 2$-quantile preferences, then they only care about what the median realisation of a random variable is. Similarly, the $\frac {1} {10}$-quantile is the first decile of the distribution, and so on.
Risk Aversion
Suijs, et al. say (also on p. 197) that $i$ is more risk averse than $j$ whenever, for all random variables $X$,
$$ \{ Y \vert Y \succeq_i E(X) \} \subset  \{ Y \vert Y \succeq_j E(X) \} .$$
In other words, whenever $i$ prefers some lottery $Y$ to some sure amount $E(X)$, $j$ also prefers the lottery $Y$ to $E(X)$. Loosely, you can think of this as $j$ always being willing to gamble whenever $i$ is willing to do so. This has a natural interpretation as $j$ preferring risk more than $i$ does, or that $i$ is more risk-averse than $j$.
Suijs, et al then go on to assert that this definition is satisfied if and only if $\beta_i < \beta_j$. This step isn't obvious to me, but perhaps this is a result that is well-known by people who work in the field. I'll try to think more about this and edit my answer if I get somewhere.
The Wikipedia page you link on risk-aversion is related, but has more to do with what Suijs, et al call "absolute risk aversion". Notice that the Wikipedia page emphasises categorising agents as risk-averse, risk-neutral, or risk-loving (consistently with the definitions that Suijs, et al give for these terms on p. 197). However, as Suijs, et al note, agents with quantile preferences are not risk-averse, risk-neutral, nor risk-loving. However, one agent with quantile preferences can be more or less risk averse than another agent also with quantile preferences.
Your Example
I think there seems to be an issue with your calculation, as you seem to have calculated the utility for an agent with $0.7$-quantile preferences, not $0.3$-quantile preferences. However, it does look like an agent with $0.3$-quantile preferences will prefer lottery $X$ to lottery $Y$ as well.
The intuition of this result is easy. Observe that the CDF of $X$ is always below the CDF of $Y$. (In more formal language, $X$ first order stochastically dominates $Y$.) This means that $Y$ is always (and uniformly) relatively more likely to realise as one of the values on the lower (and hence worse) end of the distribution, and, conversely, $X$ is relatively (uniformly) more likely to realise as one of the values on the higher (better) end of the distribution.
Answer: I am by no means an expert in this field but have had to consider this kind of preference before. Consider the following two wagers:

Wager A: With probability $p = {{1}\over{1 \times 10^6}}$ you will win $1 \times 10^7$ dollars, and with probability $1-p$, you lose $1$ dollar.
Wager B: With probability $p = {{9}\over{10}}$ you will win $10$ dollars and with $1-p$ you will lose $1$ dollar.

It is easy to compute that the expected value of Wager A exceeds that of Wager B. However, for any quantile ${1\over{10}} \lt \beta \lt 1-{{1}\over{1 \times 10^6}}$, Wager B is preferred to Wager A on a quantile basis. On quantiles $\beta \leq {1\over{10}}$, the two wagers are equivalent and on quantiles $\beta \geq 1-{{1}\over{1 \times 10^6}}$, Wager A is preferred.
Essentially, quantile preference is, indeed, a way to model a certain kind of risk-aversion/seeking although I'm not exactly sure how to relate it to the standard definitions as outlined in the wiki article. Intuitively, I would say that if someone is seeking to optimize a high quantile outcome, they're looking to score a long-shot; if someone is seeking to optimize a low quantile outcome, they're looking to avoid a worst-case scenario.
Regarding your example of the two exponential distributions, I'm not sure that it is particularly illuminating since $X$ appears to be preferred to $Y$ in both expectation and at all quantiles $\beta$. Perhaps there might be something to be said regarding the magnitude by which $X$ is preferred to $Y$ at different quantiles versus expected value?


Azure Mobile Services Push Tile Notification and Clear Tile Properties

Question: I'm using the Azure Mobile Services Scheduler service to periodically hit another api and send live tile push notifications with updated information. However, sometimes a piece of data is not available from the external api and I need to clear one of the tile properties. The problem is, I can't seem to get the values to clear from the tile when I push out a new notification. I've tried sending null, empty string, and undefined. But nothing I do will clear it. Here is my scheduler code:
<code>push.mpns.sendFlipTile(channel.uri,
{
    title: 'My Title',
    backgroundImage: getTileUrl('front', 'medium', word),
    backBackgroundImage: getTileUrl('back', 'medium', word),
    wideBackgroundImage: getTileUrl('front', 'wide', word),
    wideBackBackgroundImage: getTileUrl('back', 'wide', word),

    backTitle: null,        // Doesn't clear value
    //backTitle: '',        // Doesn't clear value
    //backTitle: undefined, // Doesn't clear value

    count: '0'              // Sending 0 as a string will clear the counter
},
{
    success: function(pushResponse) {
        console.log("Sent push:", pushResponse);
    },
    error: function(errorResponse) {
        console.error("Error push:", errorResponse);
    }
});
</code>
Can anyone help me?
Answer: Thanks for bringing this up. Currently we don't support the ability to clear a value. The addition of the feature is being tracked by this bug. Please help us prioritize fixing this by making an entry on our UserVoice page!
Answer: According to this documentation:
http://msdn.microsoft.com/en-us/library/windowsphone/develop/jj206971(v=vs.105).aspx
You would need to set Action attribute to Clear for that backTitle.
Don't know how that could be done on Windows Azure Mobile Services MPNS.


Is a group with order $3^3\cdot 5\cdot 7$ possible?

Question:
Is the group $|G|=3^3\cdot 5\cdot 7$ possible?

I've been examining it, and it seems like there can't be enough elements in the Sylow-$p$-subgroups.
That is, there can only be $21$ (or $1$) groups of order $5$ (the only number that is equal to $1 + 5k$ and divides $189$). We also know they can't contain a proper subgroup, so their intersection must be trivial. This implies there are $84$ elements of order $5$, at most.
Similarly, the number of $7$-sylow subgroups can only be $1$ or $15$, and again the intersection is trivial. So, at most $90$ elements are of order $7$.
This leaves elements of order $3$, $9$, or $27$. The number of Sylow subgroups is $1$ or $7$, so the most amount elements can be found if there are seven Sylow $3$ groups which intersect non-trivially, which means there are at most $182$ elements of this type.
But $1+182+84+90 \neq 945$.
Comment: Just take the cyclic group, which exists **for any** $\;n\in\Bbb N\;$ ...
Comment: Basically, you are forgetting that there can be elements of the group which are **outside** the Sylow subgroups; elements whose orders are divisible by more than one prime.
Comment: You forgot to take into account all the possible of elements of order $\;3\cdot5=15\,,\,3^2\cdot5=45\;$ and etc. There are many possibilities!
Comment: Okay so then if a group of order 945 is possible, where am I going wrong while calculating the number of elements ?
Answer: Consider the cyclic group of order $3^3\cdot 5\cdot 7$.
In fact, for any $n\in\Bbb N$, there exists at least one group, namely the cyclic group of order $n$.


Find a quadratic equation with integral coefficients whose roots are $\frac{α}{β}$ and $\frac{β}{α}$

Question: The roots of the equation $2x^2-3x+6=0$ are α and β. Find a quadratic equation with integral coefficients whose roots are $\frac{α}{β}$ and $\frac{β}{α}$.
The answer is $4x^2+5x+4=0$
I don't know how to get to the answer. Could someone explain the steps?
Answer: Rename $a= \alpha $ and $b=\beta$
So this is $(x-{a\over b})(x-{b\over a})=0$ thus
$$x^2-{a^2+b^2\over ab}x+1=0$$
Since $ab = {6\over 2}=3$ and $a^2+b^2 = (a+b)^2-2ab = ({3\over 2})^2-6 = {-15\over 4} $ we get:
$$x^2+{5\over 4}x+1=0$$
and thus the conclusion.
Answer: By Vieta's theorem we know that $\alpha+\beta=\frac{3}{2}$ and $\alpha\beta=3$. If follows that
$$ \frac{\alpha}{\beta}+\frac{\beta}{\alpha} = \frac{(\alpha+\beta)^2}{\alpha\beta}-2 =\frac{3}{4}-2=-\frac{5}{4}$$
and obviously $\frac{\alpha}{\beta}\cdot\frac{\beta}{\alpha}=1$. It follows that a polynomial vanishing at $\frac{\alpha}{\beta}$ and $\frac{\beta}{\alpha}$ is given by $z^2+\frac{5}{4}z+1$ or by $\color{red}{4x^2+5x+4}$ as stated.
Answer: Hint:
First approach: Find $\alpha$ and $\beta$ first.
Second approach:
$$\frac{\alpha}{\beta} + \frac{\beta}{\alpha}=\frac{(\alpha^2+\beta^2)}{\alpha\beta}=\frac{(\alpha+\beta)^2-2\alpha\beta}{\alpha\beta}$$
Answer: $$\alpha'=\frac{\alpha}{\beta} ,\beta'=\frac{\beta}{\alpha}$$ now find sum and product of $\bf new$ roots
$$\quad{S'=\alpha'+\beta'\\p'=\alpha'\times \beta'\\
S'=\alpha'+\beta'=\frac{\alpha}{\beta} +\frac{\beta}{\alpha}=\frac{\alpha^2+\beta^2}{\alpha\beta}=\frac{s^2-2p}{p}=\frac{(-\frac{-3}{2})^2-2.\frac{6}{2}}{\frac{6}{2}}\\p'=\alpha'\times\beta'=\frac{\alpha}{\beta} \times\frac{\beta}{\alpha}=1}$$  then put into $$\quad{x^2-s'x+p'=0\\x^2-\frac{-5}{4}x+1=0 \to \times 4 \\4x^2+5x+4=0}$$
Answer: As $\alpha+\beta=\dfrac32, \alpha\beta=\dfrac62$
let $y=\dfrac\alpha\beta\iff y+1=\dfrac3{2\beta}\iff\beta=\dfrac3{2(y+1)}$
But as $\beta$ is a root of  $$2x^2-3x+6=0$$
$$2\left(\dfrac3{2(y+1)}\right)^2-3\left(\dfrac3{2(y+1)}\right)+6=0$$
As $y+1\ne0,$ multiply both sides by $\dfrac{2(y+1)^2}3$ to find $$0=3-3(y+1)+4(y+1)^2=4y^2+5y+4$$
By symmetry, we can surmise that the same equation will be reached if we start with  $y=\dfrac\beta\alpha$
Comment: See also : https://math.stackexchange.com/questions/1837156/the-roots-of-the-equation-x2-6x-7-0-are-%CE%B1-and-%CE%B2-find-the-equation/1837170   https://math.stackexchange.com/questions/1909362/product-of-one-minus-the-tenth-roots-of-unity/1909366#1909366     https://math.stackexchange.com/questions/1909362/product-of-one-minus-the-tenth-roots-of-unity   https://math.stackexchange.com/questions/1811081/problem-based-on-sum-of-reciprocal-of-nth-roots-of-unity
Answer: $$
\left( x - \frac \alpha \beta \right)\left( x - \frac\beta\alpha\right) = 0
$$
$$
(\beta x-\alpha)(\alpha x-\beta) = 0
$$
$$
\alpha\beta x^2 - (\alpha^2+\beta^2) x + \alpha \beta = 0 \tag 1
$$
If $2x^2-3x+6=0$ then $x = \dfrac{3 \pm \sqrt{-39}} 4.$ So
$$
\alpha\beta = \frac{3+\sqrt{-39}} 4 \cdot \frac{3-\sqrt{-39}} 4 = 3.
$$
$$
\alpha^2+\beta^2 = \left( \frac{3+\sqrt{-39}} 4 \right)^2 + \left( \frac{3-\sqrt{-39}} 4 \right)^2 = \frac{-30}{16} + \frac{-30}{16} = \frac{-15} 4.
$$
So line $(1)$ becomes
$$
3x^2 + \frac{15} 4 x + 3 = 0
$$
which is equivalent to
$$
4x^2 + 5x + 4 =0.
$$


Connect method in winsock2 does not seem to work

Question: #define PORT 3000

int sock()
{
    int valread;
    struct sockaddr_in serv_addr;
    const char* hello = "Hello from client";
    char buffer[1024] = { 0 };
    SOCKET sock = socket(AF_INET, SOCK_STREAM, 0);
    if (sock < 0)
    {
        printf("\n Socket creation error \n");
        return -1;
    }

    serv_addr.sin_family = AF_INET;
    serv_addr.sin_port = htons(PORT);

    // Convert IPv4 and IPv6 addresses from text to binary form
    if (inet_pton(AF_INET, "127.0.0.1", &serv_addr.sin_addr) <= 0)
    {
        printf("\nInvalid address/ Address not supported \n");
        return -1;
    }

    if (connect(sock, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) < 0)
    {
        printf("\nConnection Failed \n");
        return -1;
    }
    send(sock, hello, strlen(hello), 0);
    printf("Hello message sent\n");

    return 0;
}

I have this code for the sockets, to be sent at localhost:3000 address. But they are not sent, and Connection failed error pops up. I have node.js server listener socket open at that port.
Comment: Have you called `WSAStartup`?  If so, was the call successful?  Sorry, but the error message seems fairly clear.
Comment: What is the error?  Use [`perror`](https://man7.org/linux/man-pages/man3/perror.3.html) to get the relevant information.
Comment: @G.M. Error code is 10093 And it says "Either the application has not called WSAStartup, or WSAStartup failed." I have no idea what it means though xD
I have used GetLastError because I am using windows 10
Answer: On Windows only 1, you must call WSAStartup() to initialize the Winsock library before using any other socket functions, like connect(). See Initializing Winsock. And you should call WSACleanup() when you are done using Winsock.
1: On non-Windows platforms, there are no equivalent startup/cleanup functions that need to be called when using sockets.
Try something more like this:
#include <winsock2.h>
#include <windows.h>
#include <stdio.h>

#define PORT 3000

int main()
{
    WSADATA wsa;
    int iResult = WSAStartup(MAKEWORD(2,0), &wsa);
    if (iResult != 0)
    {
        printf("WSAStartup failed: %d\n", iResult);
        return INVALID_SOCKET;
    }

    printf("Winsock initialized\n");

    struct sockaddr_in serv_addr;
    ZeroMemory(&serv_addr, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_port = htons(PORT);

    // Convert IPv4 and IPv6 addresses from text to binary form
    if (inet_pton(AF_INET, "127.0.0.1", &serv_addr.sin_addr) <= 0)
    {
        printf("Invalid address/ Address not supported \n");
        WSACleanup();
        return -1;
    }

    SOCKET sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
    if (sock == INVALID_SOCKET)
    {
        printf("Socket creation failed: %d\n", WSAGetLastError());
        WSACleanup();
        return -1;
    }

    if (connect(sock, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) == SOCKET_ERROR)
    {
        printf("Connection failed: %d\n", WSAGetLastError());
        closesocket(sock);
        WSACleanup();
        return -1;
    }

    printf("Connection successful\n");

    const char* hello = "Hello from client";

    if (send(sock, hello, strlen(hello), 0) == SOCKET_ERROR)
    {
        printf("Send failed: %d\n", WSAGetLastError());
        closesocket(sock);
        WSACleanup();
        return -1;
    }

    printf("Hello message sent\n");

    closesocket(sock);
    WSACleanup();

    return 0;
}

That said, you really should be using getaddrinfo() instead of inet_pton(), especially if you want to support both IPv4 and IPv6, eg:
#include <winsock2.h>
#include <ws2tcpip.h>
#include <windows.h>
#include <stdio.h>

#define PORT "3000"

int main()
{
    WSADATA wsa;
    int iResult = WSAStartup(MAKEWORD(2,0), &wsa);
    if (iResult != 0)
    {
        printf("WSAStartup failed: %d\n", iResult);
        return INVALID_SOCKET;
    }

    printf("Winsock initialized\n");

    struct addrinfo hints, *addrs;
    ZeroMemory(&hints, sizeof(hints));
    hints.ai_family = AF_UNSPEC;
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_protocol = IPPROTO_TCP;

    // Convert IPv4 and IPv6 addresses from text to binary form
    iResult = getaddrinfo("127.0.0.1", PORT, &hints, &addrs);
    if (iResult != 0)
    {
        printf("Get Address Info failed: (%d) %s\n", iResult, gai_strerror(iResult));
        WSACleanup();
        return -1;
    }

    SOCKET sock = INVALID_SOCKET;

    for(struct addrinfo *addr = addrs; addr != NULL; addr = addr->ai_next)
    {
        sock = socket(addr->ai_family, addr->ai_socktype, addr->ai_protocol);
        if (sock != INVALID_SOCKET)
        {
            if (connect(sock, addr->ai_addr, addr->ai_addrlen) != SOCKET_ERROR)
                break;

            closesocket(sock);
            sock = INVALID_SOCKET;
        }
    }

    if (sock == INVALID_SOCKET)
    {
        printf("Socket creation failed/Connection failed\n");
        WSACleanup();
        return -1;
    }

    printf("Connection successful\n");

    const char* hello = "Hello from client";

    if (send(sock, hello, strlen(hello), 0) == SOCKET_ERROR)
    {
        printf("Send failed: %d\n", WSAGetLastError());
        closesocket(sock);
        WSACleanup();
        return -1;
    }

    printf("Hello message sent\n");

    closesocket(sock);
    WSACleanup();

    return 0;
}

Comment: It seems to be sent? But there's no reponse on node.js and in wireshark - https://prnt.sc/ze160d
Comment: @ddd What does your node.js server *expect* you to send? You can't just send it arbitrary data, unless it is acting as an echo server. Otherwise, you have to send it what it wants. For instance, if it is acting as a web server, you have to send it a valid HTTP request. And how do you know it is not responding? This code is not trying to read any response after `send()` is done. As for Wireshark, it can't capture localhost traffic without [additional setup](https://gitlab.com/wireshark/wireshark/-/wikis/CaptureSetup/Loopback).
Comment: @ddd You can't use `(PCSTR)3000` with `getaddrinfo()`, it expects real null-terminated strings, so you have to use `"3000"` instead, like I showed in my example
Comment: Node.js code - https://pastebin.com/kapan7RZ
After using "3000", it shows:
Winsock initialized
Connection successful
Hello message sent
Comment: @ddd I don't see a `"new connection"` message in your server log. Are you sure your client is actually connecting to your node.js server and not to something else? Use [`netstat`](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/netstat) to verify. I'm not familiar with node.js, but shouldn't `io.on("connection", ...)` be *before* `io.listen()`, per the [documentation](https://socket.io/docs/v3/server-initialization/#Initialization)? Also, the accepted `socket` is waiting for several events, but I don't see your client sending anything that matches those events.


How to Customize View Output

Question: I'm working on some views and I came across a requirement I'm not sure how to handle.
I need the title for each returned item to link to a specific URL. However, if another field (a link field) isn't empty, link to that link instead. I believe I need to edit the view template files but I'm not sure how to get started.

In my view, I've found the "Theme: Information" section and am able to get template files working for my view. However, it's listing display, style, row, and field templates. Where would I incorporate the logic I described above for the title field?
Once in a template file I am unsure of what variables are available to me. Is there a list or a way to discover these?
Answer: Views is rendered by some of *.tpl.php files, they are:

views-view.tpl.php. Main view template.
views-view-unformatted.tpl.php. Default simple view template to display a list of rows.
views-view-fields.tpl.php. Default simple view template to all the fields as a row.
views.view-field.tpl.php. This template is used to print a single field in a view.

If you want to use two different fields from one view in one file, I recommend to use views-view-fields.tpl.php where you can manipulate of all fields which you assign to view.
The views-view-fields.tpl.php has three variables to use: $view, $fields and $row. I suggest to use only $fields.
Then, if you have field named field_image you can render its using $fields['field_image']->content or $fields['field_image']->raw.
Also, I recommend to read more about it: Views template files
Comment: Fields in `view-views-fields.tpl-php` still are rendered by Views UI module. Thats why I mentioned about `$fields['field_name']->raw`. Also in Views UI you have to set that link generated by Link module is plain URL.
Comment: Thanks, that's really helpful. I've created a tpl file names something similar to views-view-fields--page-topic.tpl.php and it is affecting the correct content on the site. However, I've looked around in the $fields var and all the content is already rendered as HTML in $fields['field_lede']->content. I don't have individual pieces of content. Do you know what might cause that?

How to Customize View Output


What is the modulus of a tensor on a Riemannian 3-manifold?

Question: Let $v^i$ be a vector on a Riemannian 3-manifold with metric $g_{ij}$ embedded inside a 3+1 space-time such that for some constant $N_M$ it satisfies the inequality $g_{ij}v^iv^j \leq  N_M ^2$. Let $K$ be a symmetric rank-2 tensor on the 3-manifold. Then apparently the following holds:
$$\vert K_{ij} v^i v^j \vert \leq \vert K \vert _g N_M ^2.$$
This looks like some sort of a Cauchy-Schwarz inequality but given that $K$ is a tensor as described, I don't understand what the notation on the RHS means. For a rank-2 symmetric tensor $K$ what does $\vert K \vert _g$ mean?
If one knows that for some function $N$, $g_{ij}v^iv^j \leq  N^2,$ where the function $N$ is itself bounded between constants
$$N_m \leq N \leq N_M,$$
then using inequalities like the above one can apparently show the following bound:
$$\int _{t_1} ^t \frac{1}{N} \Big(-v^i \partial _i N - \frac{dN}{dt} + K_{ij}v^iv^j\Big) dt
 \leq -2\log N_m + \frac{1}{N_m} \int _{t_1}^t (\vert \nabla N \vert _g N_M + \vert K \vert _g N_M ^2 )dt,$$
for some fixed $t_1$ and $t$.
I can't understand that first "log" term in the above.
Also once the above bound is shown does it follow that the integral can be unbounded above or below depending solely on the property of the function $N$? If yes then what would be needed of $N$ to make the integral unbounded above or below?
Comment: I think the $|K|_g$ here comes from the following fact:  An inner product on V induces an inner product on all tensors.  In fact an inner product on $V$ gives a natural isomorphism between $V$ and $V^*$ by $v\rightarrow \langle v,\cdot\rangle$ (I'm assuming $V$ is finite dimensional here).  Then, by declaring this isomorphism to be an isometry, we get an inner product on $V*$.  If $e_i$ is an o.n. basis of $V$, then $\langle e_i, \cdot \rangle$ is a basis of $V*$.  Now by declaring things of the form $e_i\otimes e_j^*$, etc to be orthonormal, one gets an inner product on all tensors.
Comment: I don't know why I never answered the second part of your question.  The $\log$ term comes from using the fundamental theorem of calculus, and the other terms come from using Cauchy-Schwartz with the induced norm I mentioned in my answer and what is known in geometric analysis as Kato's inequality, $|\nabla |T|| \le |\nabla T|$.
Comment: I believe $|K|_g$ is the [induced norm](http://en.wikipedia.org/wiki/Matrix_norm#Induced_norm).
Comment: @Kenny could you elaborate a little more? Like if you could add in a few lines of the derivation.
Comment: It's just a speculation because I remembered a similar thing in real analysis. I don't have a derivation. Or probably $|K|_g$ is just *defined* as $\max_{v\ne 0}\frac{|K_{ij}v^iv^j|}{g_{ij}v^iv^j}$.
Answer: I am just going to answer the first question.  For a two-tensor we have
$$|T|^2 = \left< T,T \right> = g^{ik}g^{jl}T_{ij}T_{kl},$$
where $g$ is the metric, and the summation convention is understood.  Note that this is the standard inner product structure induced by $g$, as mentioned by Jason DeVito.  You might like to try and prove that this really does define a norm in the Riemannian case.
More generally, for a $(k,l)$ ($k$ times contravariant, $l$ times covariant) tensor field $T$ (on the manifold) we have
$$|T|^2 = \left< T,T \right> = g^{j_1q_1}g^{j_2q_2}\cdots g^{j_lq_l}g_{i_1p_1}g_{i_2p_2}\cdots g_{i_kp_k}T^{i_1i_2\cdots i_k}_{j_1j_2\cdots j_l}T^{p_1p_2\cdots p_k}_{q_1q_2\cdots q_l}.$$
For your other questions, you could do worse than look in 'Einstein Manifolds' by Besse.


Correcting p values for multiple tests where tests are correlated (genetics)

Question: I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The method I am thinking about (a variant of Fisher's Product Method, Zaykin et al., Genet Epidemiol, 2002) needs the correlation between the p values.
In order to estimate this correlation, I am currently thinking about bootstrapping cases, running the analyses and correlating the resulting vectors of p values. Does anyone have a better idea? Or even a better idea for my original problem (correcting for multiple testing in correlated tests)?
Background: I am logistically regressing whether or not my subjects are suffering from a particular disease on the interaction between their genotype (AA, Aa or aa) and a covariate. However, the genotype is actually a lot (30-250) of Single Nucleotide Polymorphisms (SNPs), which are certainly not independent but in Linkage Disequilibrium.
Answer: This is actually a hot topic in Genomewide analysis studies (GWAS)! I am not sure the method you are thinking of is the most appropriate in this context. Pooling of p-values was described by some authors, but in a different context (replication studies or meta-analysis, see e.g. (1) for a recent review). Combining SNP p-values by Fisher's method is generally used when one wants to derive an unique p-value for a given gene; this allows to work at the gene level, and reduce the amount of dimensionality of subsequent testing, but as you said the non-independence between markers (arising from spatial colocation or linkage disiquilibrium, LD) introduce a bias. More powerful alternatives rely on resampling procedures, for example the use of maxT statistics for combining p-value and working at the gene level or when one is interested in pathway-based approaches, see e.g. (2) (§2.4 p. 93 provides details on their approach).
My main concerns with bootstraping (with replacement) would be that you are introducing an artificial form of relatedness, or in other words you create virtual twins, hence altering Hardy-Weinberg equilibrium (but also minimum allele frequency and call rate). This would not be the case with a permutation approach where you permute individual labels and keep the genotyping data as is. Usually, the plink software can give you raw and permuted p-values, although it uses (by default) an adaptive testing strategy with a sliding window that allows to stop running all permutations (say 1000 per SNP) if it appears that the SNP under consideration is not "interesting"; it also has option for computing maxT, see the online help.
But given the low number of SNPs you are considering, I would suggest relying on FDR-based or maxT tests as implemented in the multtest R package (see mt.maxT), but the definitive guide to resampling strategies for genomic application is Multiple Testing Procedures with Applications to Genomics, from Dudoit & van der Laan (Springer, 2008). See also Andrea Foulkes's book on genetics with R, which is reviewed in the JSS. She has great material on multiple testing procedures.
Further Notes
Many authors have pointed to the fact that simple multiple testing correcting methods such as the Bonferroni or Sidak are too stringent for adjusting the results for the individual SNPs. Moreover, neither of these methods take into account the correlation that exists between SNPs due to LD which tags the genetic variation across gene regions. Other alternative have bee proposed, like a derivative of Holm's method for multiple comparison (3), Hidden Markov Model (4), conditional or positive FDR (5) or derivative thereof (6), to name a few. So-called gap statistics or sliding window have been proved successful in some case, but you'll find a good review in (7) and (8).
I've also heard of methods that make effective use of the haplotype structure or LD, e.g. (9), but I never used them. They seem, however, more related to estimating the correlation between markers, not p-value as you meant. But in fact, you might better think in terms of the dependency structure between successive test statistics, than between correlated p-values.
References

Cantor, RM, Lange, K and Sinsheimer, JS. Prioritizing GWAS Results: A Review of Statistical Methods and Recommendations for Their Application. Am J Hum Genet. 2010 86(1): 6–22.
Corley, RP, Zeiger, JS, Crowley, T et al. Association of candidate genes with antisocial drug dependence in adolescents. Drug and Alcohol Dependence 2008 96: 90–98.
Dalmasso, C, Génin, E and Trégouet DA. A Weighted-Holm Procedure Accounting for Allele Frequencies in Genomewide Association Studies. Genetics 2008 180(1): 697–702.
Wei, Z, Sun, W, Wang, K, and Hakonarson, H. Multiple Testing in Genome-Wide Association Studies via Hidden Markov Models. Bioinformatics 2009 25(21): 2802-2808.
Broberg, P. A comparative review of estimates of the proportion unchanged genes and the false discovery rate. BMC Bioinformatics 2005 6: 199.
Need, AC, Ge, D, Weale, ME, et a. A Genome-Wide Investigation of SNPs and CNVs in Schizophrenia. PLoS Genet. 2009 5(2): e1000373.
Han, B, Kang, HM, and Eskin, E. Rapid and Accurate Multiple Testing Correction and Power Estimation for Millions of Correlated Markers. PLoS Genetics 2009
Liang, Y and Kelemen, A. Statistical advances and challenges for analyzing correlated high dimensional snp data in genomic study for complex diseases. Statistics Surveys 2008 2 :43–60. -- the best recent review ever
Nyholt, DR. A Simple Correction for Multiple Testing for Single-Nucleotide Polymorphisms in Linkage Disequilibrium with Each Other. Am J Hum Genet. 2004 74(4): 765–769.
Nicodemus, KK, Liu, W, Chase, GA, Tsai, Y-Y, and Fallin, MD. Comparison of type I error for multiple test corrections in large single-nucleotide polymorphism studies using principal components versus haplotype blocking algorithms. BMC Genetics 2005; 6(Suppl 1): S78.
Peng, Q, Zhao, J, and Xue, F. PCA-based bootstrap confidence interval tests for gene-disease association involving multiple SNPs. BMC Genetics 2010, 11:6
Li, M, Romero, R, Fu, WJ, and Cui, Y (2010). Mapping Haplotype-haplotype Interactions with Adaptive LASSO. BMC Genetics 2010, 11:79 -- although not directly related to the question, it covers haplotype-based analysis/epistatic effect
Comment: Wow, thanks for going to all this trouble! I understand your qualms about bootstrapping, and I'm almost convinced. I think my main complication is the numerical covariate I have that will certainly be necessary (either by itself or in interaction with genotype), and that seems to rule out mt.maxT and plink, although I may need to look into plink again. But I will certainly dig through the references you provided!
Comment: You can always work with the residuals of your GLM to get ride of your covariates, though you lost some Df that will be difficult to account or reintroduce afterwards (e.g. for computing p-value).
Comment: Hm, residuals from my logistic regression? Would that be legitimate?
Comment: Yes, why not? It is not uncommon to remove the variance accounted for by other covariates and then move on 2nd-level analysis with your residualized data. It is often faster (for instance, plink is pretty slow with categorical covariates, while it's ok with continuous ones; `snpMatrix` or simply `glm()` performs quite better on this point but you cannot embed a lot of SNPs within `glm()`...); the problem is that getting the corrected p-value at the end of your 2nd analysis is rather tricky (because you have to account for the parameters already estimated).
Comment: For an illustration of how people are working with residuals, see for example p. 466 of Heck et al. Investigation of 17 candidate genes for personality traits confirms effects of the HTR2A gene on novelty seeking. *Genes, brain, and behavior* (2009) vol. 8 (4) pp. 464-72
Comment: I think the Nyholt's paper is worth looking at. I'm just re-reading it and it seems to provide good reference and a well-founded framework. Maybe you'll have to check if there isn't more recent publications on this topic, e.g. http://j.mp/9jHTpI but I just look at the abstract.
Comment: Last but not least, Nicodemus has been very active for promoting the use of Random Forests when working with SNPs, and RFs are great because you get measure of variable importance and % of classification error on resampled subjects, see e.g. http://j.mp/cstgeO. There is also a paper on how to use haplotype block instead of PCA like suggested in the aforementioned paper, http://j.mp/c4V45i
Answer: Using a method like bonferroni is fine, the problem is that if you have many tests you are not likely to find many "discoveries".
You can go with the FDR approach for dependent tests (see here for details) the problem is that I am not sure if you can say upfront if your correlations are all positive ones.
In R you can do simple FDR with p.adjust.  For more complex things I would take a look at multcomp, but I didn't go through it to see for solutions in cases of dependencies.
Good luck.
Comment: Hi Tal, thanks! Bonferroni does not seem appropriate to me - if one of my SNPs *is* causal and others are associated with it, there should be a signal, and Bonferroni has always looked too conservative to me (I usually prefer Holm's stepwise correction). The FDR you link to and p.adjust do not consider *combined* evidence (and the FDR still requires me to understand the correlation of my tests, the original question). multcomp may help, though at first glance it seems like it deals more with multiple tests within a *single* model, whereas I have *multiple* models. I'll dig deeper...
Comment: Hello Stephan,
I still think you can still use the method=BY (for Benjamini Hochberg Yekuteli Procedure) in p.adjust in R, as pointed by Tal.

Definitely, using Bonferroni can be conservative.
Comment: suncoolsu, I think that this method only works when the correlation is positive (not negative) between the variables.  Cheers.
Comment: Hello Stephan.  I understand, sorry for not helping more.  Good luck!  Tal
Answer: I think Multivariate Normal Models are being used to model the correlated p-values and to get the right type of multiple testing corrections.  Rapid and Accurate Multiple Testing Correction and Power Estimation for Millions of Correlated Markers. PLoS Genet 2009 talks about them and also gives other references. It sounds similar to what you were talking about, but I think other than getting a more accurate global p-value correction, LD structure knowledge should also be use to remove spurious positives arising from markers correlated with causal markers.
Answer: I am looking for a working solution for exactly the same problem. The best I found is the Null Unrestricted Bootstrap introduced by Foulkes Andrea in his book Applied Statistical Genetics with R(2009). Contrary to all bunch of other articles and books he considers specifically the regressions. Besides other methods he advises the Null Unrestricted Bootstrap, which is suitable where one cannot easily compute residuals (as in my case, where I model many independent regressions (basically simple correlations), each with the same response variable and different snip). I found this method to be also called the maxT method.
> attach(fms)
> Actn3Bin <- > data.frame(actn3_r577x!="TT",actn3_rs540874!="AA",actn3_rs1815739!="TT",actn3_1671064!="GG")
> Mod <- summary(lm(NDRM.CH~.,data=Actn3Bin))
> CoefObs <- as.vector(Mod$coefficients[-1,1])
> B <-1000
> TestStatBoot <- matrix(nrow=B,ncol=NSnps)
> for (i in 1:B){
+    SampID <- sample(1:Nobs,size=Nobs, replace=T)
+    Ynew <- NDRM.CH[!MissDat][SampID]
+    Xnew <- Actn3BinC[SampID,]
+    CoefBoot <- summary(lm(Ynew~.,data=Xnew))$coefficients[-1,1]
+    SEBoot <- summary(lm(Ynew~.,data=Xnew))$coefficients[-1,2]
+    if (length(CoefBoot)==length(CoefObs)){
+       TestStatBoot[i,] <- (CoefBoot-CoefObs)/SEBoot
+    }
+ }

Once we have the all the TestStatBoot matrix (in rows we have bootstrap replications, and in columns we have bootstrapped $\hat{\vec{T}^*}$ statistics) we find for which $T_{\text{crit.}}$ we observe exactly $\alpha=0.05$ percent of more significant $\hat{\vec{T}^*}$ statistics (more significant means that with bigger absolute value than $T_{\text{crit.}}$).
We report $i$-th model component significant, if its $\hat{\vec{T}_i} > T_{\text{crit.}}$
The last step can be accomplished with this code
p.value<-0.05 # The target alpha threshold
digits<-1000000
library(gtools) # for binsearch

pValueFun<-function(cj)
{
   mean(apply(abs(TestStatBoot)>cj/digits,1,sum)>=1,na.rm=T)
}
ans<-binsearch(pValueFun,c(0.5*digits,100*digits),target=p.value)
p.level<-(1-pnorm(q=ans$where[[1]]/digits))*2 #two-sided.