Is this the correct place to use MAPE as a loss function?

Question: I've made a neural network designed to do regression. However, my dataset is unbalanced, and the data in the smaller section of the dataset have very different target values than the target values in the bulk of the dataset (by orders of magnitude).
My network gets good results for the bulk of the data, but bad results in the tail. I would like to try to improve this as much as I can before I resort to making more data.
An idea is to modify the loss - ie, not use MSE. I imagine, that since the majority of the data is quite similar, this area is "swamping" the loss function. If we have two points, x=1 and y=0.001, even if when they are predicted the distance from the points are the same it can have a very different meaning. (If they are both .1 out, this will affect y "more"). Is it, therefore, sensible to write a loss function where the relative difference contributes? So instead of the loss summing over absolute differences, it sums over % differences.
It seems that MAPE is exactly what I am looking for - the loss function will treat all the data equally (though I understand there are other issues with this method). Am I correct in this assessment, and if so, are there any variants of MAPE that would be even better suited?
Answer: I think it would be worthwhile for you to read carefully through my answer to What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? (Yes, I'll admit I also wrote the question, because I believe my input is helpful.) In particular, pay attention to the last bullet point in that answer, and to the explanation that follows.
The solution in my opinion lies in understanding what a point forecast is, and what a point forecast error measure (PFEM) attempts to do.

A point forecast is a one-number summary of a predictive distribution. This distribution is very often not explicit, but it always lurks in the background, and understanding is is fundamental. Of course, there are many different possible one-number summaries of a distribution: the mean, the median, the 90% (or any other) quantile, the mode, ... All these are so-called functionals of the predictive distribution.
What does the MAPE, the MSE or any other PFEM do? It evaluates how close our estimate for the target functional gets to the actual functional. The MSE is minimized in expectation if you get the correct expected value of the future distribution. The MAE is minimized in expectation by the correct median. And so forth.

Here is the problem: if you want an unbiased expectation forecast, then the MAPE will mislead you, because it is minimized in expectation by quite a different functional than the expectation. You will need to use the (R)MSE. If you think your forecasts get "better" if you use the MAPE, then I would suspect that changing the PFEM without understanding these relationships may be a case of walking down a blind alley.
You don't tell us what distributional assumptions your NN architecture makes. If your NN models changing expectations but constant variance (which I strongly suspect), then it simply does not correctly model the variation in your data distributions. (Higher means usually come with higher variances, especially if the differences in means are as large as you explain. If variance was constant, you would not have a problem in using the MSE.) If you change your error measure to the MAPE without addressing this point, then you are hearing weird noises from your car engine and thumping the hood until the weird noises cease. For the moment. It's better to understand where the noises come from.
I would suggest making sure you are using a method that can deal with changes in data distributions, and that outputs full densities. Then assess these densities using proper scoring rules.
Comment: Re your first comment: note that I wrote "it is minimized in expectation" and "by quite a different functional than the expectation". On the one hand, we have the expected value of the future distribution, which we may want to elicit (i.e., an unbiased expectation forecast), this is the second "expectation". The first "expectation" refers to the fact that the property of (say) the MAE to be minimized by the median of the future distribution is something that of course only holds *in expectation*. For any finite sample, we will find some other MAE minimizer than the true median.
Comment: Unfortunately, this is confusing. Sorry, but I don't see how to be precise, correct and non-confusing about this...
Comment: thank you. This is a great answer, very informative, lots to think about! I have a few follow-up questions: when you say "expectation by quite a different functional than the expectation" are you referring to the points you made in your other answer about the asymmetry and high coefficient of variation problems?
Comment: Then, looking at your last paragraph, if I plot the output of the network (say distributions of %error and residuals) what am I looking for to justify attempting to use MAPE? If my output using MSE is not constant (residuals increase as target increases) is that evidence that MSE is the correct choice of loss? You said if the model is constant it "does not correctly model the variation in your data distributions", or am I misinterpreting?
Comment: Re your second comment: even if your model outputs non-constant point forecasts, it may still not be modeling non-constant variances. I am arguing that it then still doesn't correctly model the future distributions. You may still want to use the MAPE - if the forecasts are more useful to you, by all means do so (we don't forecast for the fun of it, but because we use the forecasts for something) - I just believe it's good to keep the influence of the future distribution in the back of your head.


Load WinForm Object from CliXml PS

Question: How can one save and load the properties of GUI objects from a CliXml file?   Saving mwe is below, with failed attempt to load commented out. directly importing changes object type from System.Windows.Forms.Button to System.Management.Automation.PSObject. Attempts to loop over the saved properties failed.
Add-Type -AssemblyName System.Windows.Forms
[System.Windows.Forms.Application]::EnableVisualStyles()
Remove-Variable * -ErrorAction SilentlyContinue

$form = New-Object system.Windows.Forms.Form
$form.ClientSize = New-Object System.Drawing.Point(100,100)
$form.text = "Form"
$form.TopMost = $false
$form.FormBorderStyle = "FixedSingle"
$form.MaximizeBox = $false

$TestButton = New-Object system.Windows.Forms.Button
$TestButton.text = "Test"
$TestButton.width = 85
$TestButton.height = 30
$TestButton.location = New-Object System.Drawing.Point(0,0)
$TestButton.Font = New-Object System.Drawing.Font('Microsoft Sans Serif',12)
$TestButton.Enabled = $true

$path = "./vars/test.xml"
$TestButton | Export-CliXml $path

$TestButton.Enabled = $false

#$TestButton.GetType().FullName #For comp below
#$TestButton = Import-CliXml $path #Import the saved properties - intention is that this enables button again
#$TestButton.GetType() #The import changes the type. How to load saved properties and avoid this?

$form.Controls.Add($TestButton)
[void]$form.ShowDialog()

Comment: A `System.Windows.Forms.Button` is not marked as serializable, so exporting it with Export-CliXml and re-importing it is not possible.You will have to write your own custom functions for that..
Comment: Ah - I had assumed the inverse, since a file in indeed generated, or, indeed, a string of I serialise. Is there a test for serialisability outside of documentation? Or is my code above it? EDIT: Type.IsSerialisable. thanks!
Answer: Why not "serialize" and save it as a PowerShell expression:
{
    New-Object system.Windows.Forms.Button -Property @{
        text = "Test"
        width = 85
        height = 30
        location = New-Object System.Drawing.Point(0,0)
        Font = New-Object System.Drawing.Font('Microsoft Sans Serif',12)
        Enabled = $true
    }
} | Out-File .\Vars\Test.ps1

Note: using the outer curly brackets will validate the expression, but you might also just use (here) quotes
And then load it using dot-sourcing:
$TestButton = . .\Vars\Test.ps1

Comment: What the MWE does not show is I was looking for a way to take all of the attributes of a non-serializable object and save them (I tried with the disable before the save). The hope was that I could loop through a whole Form and have a quick way to save / load everything. I have a well populated user drive-GUI for a test set with many boxes / buttons. Was hoping to add a load/save feature without writing a text-grabber for text boxes, a check/uncheck for tickboxes etc...

Load WinForm Object from CliXml PS


How can I execute Ruby scripts via Apache (not necessarily using Rails)?

Question: I'm trying to get Apache to interpret my Ruby files but I don't understand how to do it. I'm not concerned with Rails right now, only Ruby. What I've found by Googling only confuses me. Any advice?
Answer: The most common way to run ruby code in a webserver environment is by having the code implement a rack interface. It is a very simple API that allows the webserver to speak to your ruby application. For Apache, the most common rack handler is passenger (aka. mod_rails and mod_rack). Almost all current ruby web frameworks (like rails, sinatra, camping, ...) connect to the webserver using a rack handler. You can use one of those frameworks, but you can also write rack apps directly without those. All ruby application servers (mongrel, thin, unicorn, rainbows, ...) implement the rack interface.
Another alternative would be to use (f)cgi, but I'd advise you to stay away from that route, as the interface and protocols are awkward and do not really provide modern management facilities.
Answer: I've been trying to find the simple answer to this question for a while now, and figured out all you have to do is put the ruby script in your webserver's cgi-bin directory and load the corresponding URL. So for example, in my default CentOS 6.2/httpd install, I put a script called hello.rb in /var/www/cgi-bin/ and then loaded up http://localhost/cgi-bin/hello.rb in my browser and it executed the script and showed me what went to STDOUT (i.e. from puts, etc).
What happens here is every time the URL is loaded, the script runs completely. This is not ideal for a high volume webserver or where speed is crucial -- so something like fastCGI keeps the script running always, but you have to write the script to handle that.
hope that helps.
Answer: You can execute ruby scripts with fast cgi. Look at the fcgi project for more information.
Answer: Try modruby.
Comment: I tried it. It didn't do anything and if there's any documentation, they did a great job of hiding it.
Comment: http://perfectionlabstips.wordpress.com/2008/12/23/writing-apache-handlers-with-mod_ruby/
Comment: That didn't work, either. `Invalid command 'RubyAddPath'` Is this some kind of esoteric thing nobody ever does? I'm surprised by how hard it is to get Ruby working with Apache compared to PHP.
Comment: Add "LoadModule ruby_module "path_to_mod/mod_ruby.so"
Comment: Adding `AddHandler ruby-object .rbx`, `LoadModule ruby_module /path/to/mod_ruby`, `RubyRequire apache/ruby-run`, and `RubyHandler Apache::RubyRun.instance` to `httpd.conf` should be enough for a simple configuration.
Comment: Mod_ruby is not maintained anymore and does some surprising things. You probably don't really want to use this.


How to add a .sql file to sqlite db in Xcode

Question: I am writing a offline iPhone app in which I need to read and display from a database which consists of a few number of tables. The tables have over 100k entries in it, so it is almost impossible to enter it all individually. I have downloaded the .sql file of the entire db. Now how can I import the db as such into sqlite/Xcode so that I can readily start accessing the data in it??
Edit: I have seen that when using a database, the file extension that is used is a .db file. Is there anyway I can convert the .sql file to a .db file. And if I can do that, can I simply access the .db file by placing it in the project directory?
Answer: If you have created the .sqlite file and have the tables in it,Then add the .sqlite file into xcode like just drag from desktop into your bundle(Resources).And then use NSFileManager to access the sqlite file.You need to write methods for createdatabase and initialize database and also you can see the sqlite file in your documents folder/simulator folder.
- (void)createEditableCopyOfDatabaseIfNeeded {
    NSLog(@"Checking for database file");
    NSFileManager *fileManager = [NSFileManager defaultManager];
    NSError *error;
    NSString *dbPath = [self getDBPath];
    NSString *defaultDBPath = [[[NSBundle mainBundle] resourcePath] stringByAppendingPathComponent:@"ihaikudb.sql"];
    BOOL success = [fileManager fileExistsAtPath:dbPath];

    NSLog(@"If needed, bundled default DB is at: %@",defaultDBPath);

    if(!success) {
        NSLog(@"Database didn't exist... Copying default from resource dir");
        success = [fileManager copyItemAtPath:defaultDBPath toPath:dbPath error:&error];

        if (!success)
            NSAssert1(0, @"Failed to create writable database file with message '%@'.", [error localizedDescription]);
    } else {
        NSLog(@"Database must have existed at the following path: %@", dbPath);
    }
    NSLog(@"Done checking for db file");
}

Comment: how to get [self getDBPath] ?
Comment: My doubt is, the `.sql` file actually has the commands to create the database and the tables and to insert the values into the tables. How is it possible to access values from such a file by using sql commands?
Comment: '[self getDBPath]' shows error. How to overcome it?
Comment: We use to read database from a `.db` file right? Rather, I have a `.sql` file with me, not `.sqlite`
Comment: no problem you just drag and drop the file into your folder and start working.the type of file extension doesn't matter.If you have copy then drag and drop into your workng folder is called sandboxing.
Comment: It is the same as .sqlite and .sql are same


Unable to send base64 String to server in Swift on iOS

Question: In my application I have pick the image using UIImagePickerController from photos and then compress and convert to base64 string. Finally I upload the base64 string to server. Here the server was not accept the base64 string and does not work, but in Android and Postman it works well. I couldn't find the mistake in my code.
Here I mention the UIImagePickerControllerDelegate:
// MARK: UIImagePickerControllerDelegate
func imagePickerControllerDidCancel(_ picker: UIImagePickerController) {
    dismiss(animated: true, completion: nil)
}
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any]) {
    selectedImage = info[UIImagePickerControllerOriginalImage] as! UIImage
    selectedImage = selectedImage.resizeWithWidth(width: 700)!
    picker.dismiss(animated: true, completion: nil)
    DispatchQueue.main.async {
        self.collPhotos.reloadData()
    }
}

Here I mention the Base64 Conversion and post the parameter string to server:
func addImagesApiCAll() {
    let imagedata1:UIImage = selectedImage as! UIImage
    let compressData = UIImagePNGRepresentation(imagedata1)
    let base64 = compressData?.base64EncodedString(options: .lineLength64Characters)
    print("charCount",base64!.count)

    if Reachability()!.isReachable {
        let id = Singleton.sharedInstance.selectedCategory!
        let parameterStr = "property_id=\(self.PropertyID)&photos=\(base64!)&lang_code=\(lanuguage_selection.value(forKey: "language") ?? "en")&base_id=\(id)&user_id=\(login_session.value(forKey: "UserId")!)"
        Network.shared.POSTRequest(withParameterString: parameterStr, serviceURL: SAVE_PHOTO_LISTING, APIKEY: "SAVE_PHOTO_LISTING")
    } else {
        showInformation(title: "Network Error", message: "Please check your internet connection")
    }
}

extension AddPhotoViewController: HTTP_POST_STRING_REQUEST_PROTOCOL {
func httpPostRequest(APIKEY: String, requestURL: String, responseDict: NSDictionary, errorDict: String) {
    ListingActivityDelegate.hideActivity()
    if APIKEY == "SAVE_PHOTO_LISTING"{
       if errorDict.count == 0 {
        print(responseDict)
        let mod = RentYourSpaceModel(fromDictionary: responseDict as! [String : Any])
        if mod.status! != 0 {
           Singleton.sharedInstance.rentYourSpace = mod
            if Singleton.sharedInstance.rentYourSpace.result[0].step5.productImage.count == 0{
                imageFromResponse = "NO"
            } else {
                imageFromResponse  = "YES"
            }
        }
        self.showInformation(title: "Application", message: mod.message)
    }
       else {

  }
}
}
}

Here I mention the code for Select Image from Camera or Gallery:
@IBAction func act_AddPhoto(_ sender: UIButton) {
    let actionSheet = UIAlertController(title: "Home Stay", message: "Choose Image", preferredStyle: .actionSheet)
    actionSheet.addAction(UIAlertAction(title: "Camera", style: .default, handler: { _ in
        self.openCamera()
    }))
    actionSheet.addAction(UIAlertAction(title: "Photos", style: .default, handler: { _ in
        self.openGallary()
    }))
    actionSheet.addAction(UIAlertAction.init(title: "Cancel", style: .cancel, handler: nil))
    //If you want work actionsheet on ipad then you have to use popoverPresentationController to present the actionsheet, otherwise app will crash in iPad
    switch UIDevice.current.userInterfaceIdiom {
    case .pad:
        actionSheet.popoverPresentationController?.sourceView = sender
        actionSheet.popoverPresentationController?.sourceRect = sender.bounds
        actionSheet.popoverPresentationController?.permittedArrowDirections = .up
    default:
        break
    }
    self.present(actionSheet, animated: true, completion: nil)
}

//MARK: - Open the camera
func openCamera() {
    if(UIImagePickerController .isSourceTypeAvailable(UIImagePickerControllerSourceType.camera)){
        imagePicker.sourceType = UIImagePickerControllerSourceType.camera
        //If you dont want to edit the photo then you can set allowsEditing to false
        imagePicker.allowsEditing = true
        imagePicker.delegate = self
        self.present(imagePicker, animated: true, completion: nil)
    }
    else{
        let alert  = UIAlertController(title: "Warning", message: "You don't have camera", preferredStyle: .alert)
        alert.addAction(UIAlertAction(title: "OK", style: .default, handler: nil))
        self.present(alert, animated: true, completion: nil)
    }
}

//MARK: - Choose image from camera roll
func openGallary(){
    imagePicker.sourceType = UIImagePickerControllerSourceType.photoLibrary
    //If you dont want to edit the photo then you can set allowsEditing to false
    imagePicker.allowsEditing = true
    imagePicker.delegate = self
    self.present(imagePicker, animated: true, completion: nil)
}

Here I give the POSTRequest Method:
//MARK:- Post request with parameter String.
func POSTRequest(withParameterString: String , serviceURL: String , APIKEY: String)
{
    var RESPONSE_ERROR = String()
    var RESPONSE_DATA = NSDictionary()
    let Url = String(format: serviceURL)
    guard let serviceUrl = URL(string: Url) else { return }
    var request = URLRequest(url: serviceUrl)
    let postString = withParameterString
 //   print(postString)
    request.httpBody = postString.data(using: String.Encoding.utf8);
  //request.addValue("application/json", forHTTPHeaderField: "content-type")
    request.httpMethod = "POST"
    let task = URLSession.shared.dataTask(with: request, completionHandler: {
        data, response, error in
        if let response = response {
            print(response)
        }
        if let resdata = data {
            do {

              //  print(response)

                let json =  try JSONSerialization.jsonObject(with: resdata, options: .mutableContainers) as? NSDictionary

                   if let parseJSON = json {

                      //print(json)
                    if parseJSON.object(forKey: "status") as! NSInteger == 1 {
                        if error != nil {
                           RESPONSE_ERROR = (error?.localizedDescription)!
                        }
                        DispatchQueue.main.async {
                            RESPONSE_DATA = parseJSON
                            self.HTTP_POST_STRING_REQUEST_DELEGATE?.httpPostRequest(APIKEY: APIKEY, requestURL: serviceURL, responseDict: RESPONSE_DATA, errorDict: RESPONSE_ERROR)
                        }
                    } else {
                        DispatchQueue.main.async {
                             RESPONSE_DATA = parseJSON
                            self.HTTP_POST_STRING_REQUEST_DELEGATE?.httpPostRequest(APIKEY: APIKEY, requestURL: serviceURL, responseDict: RESPONSE_DATA, errorDict: RESPONSE_ERROR)
                        }
                    }
                } else {
                    DispatchQueue.main.async {
                        RESPONSE_ERROR = "No Data"
                        self.HTTP_POST_STRING_REQUEST_DELEGATE?.httpPostRequest(APIKEY: APIKEY, requestURL: serviceURL, responseDict: RESPONSE_DATA, errorDict: RESPONSE_ERROR)
                    }
                }
            }catch {
                 DispatchQueue.main.async {
                RESPONSE_ERROR = "Check your input datas"
                self.HTTP_POST_STRING_REQUEST_DELEGATE?.httpPostRequest(APIKEY: APIKEY, requestURL: serviceURL, responseDict: RESPONSE_DATA, errorDict: RESPONSE_ERROR)
            }
            }
        } else {
            DispatchQueue.main.async {
                RESPONSE_ERROR = (error?.localizedDescription)!
                self.HTTP_POST_STRING_REQUEST_DELEGATE?.httpPostRequest(APIKEY: APIKEY, requestURL: serviceURL, responseDict: RESPONSE_DATA, errorDict: RESPONSE_ERROR)
            }
        }
    })
    task.resume()
}

Answer: It looks like you are trying to send base64 string as query string. It would be too long to be a query string. You need to share details for POSTRequest method. For more information.
Network.shared.POSTRequest(withParameterString: parameterStr, serviceURL: SAVE_PHOTO_LISTING, APIKEY: "SAVE_PHOTO_LISTING")

And you need to avoid using exclamation (!) in your application. Your code would crash in many points.
For example:
if Reachability()?.isReachable

You can use optional values to prevent crashes.
 let parameters = ["property_id":self.PropertyID,
                "photos":base64 ?? "",
                "lang_code":lanuguage_selection.value(forKey: "language") ?? "en",
                "base_id":id,
                "user_id":login_session.value(forKey: "UserId") ?? ""];
 Network.shared.POSTRequest(parameters: parameters, serviceURL: SAVE_PHOTO_LISTING, APIKEY: "SAVE_PHOTO_LISTING")

in method add json data:
if let jsonData = try? JSONSerialization.data(withJSONObject: parameters) {
    request.httpBody?.append(jsonData)
}

Comment: Check My Updated Question.


testing undecorated function with .__wrapped__ gives error "requires 1 more positional argument"

Question: I have looked for this on the forum but i could find anything precisely dealing with functool wrapped...
I have a class with a function copyfile and a decorator fileprocessing defined as :
class Sync():
    ...
    def fileprocessing(func):
        "decorator for copyfile methods"
        @functools.wraps(func)
        def wrapped_f(*args):
            ...#some code invoking copyfile
        return wrapped_f

    @fileprocessing
    def copyfile(self,src, dst, file):
        "try copying with shutil file in src folder to dst folder, otherwise with python"
        try:
            shutil.copy2(f'{src}/{file}',f'{dst}/{file}', follow_symlinks=False)
        except Exception as err:
            print(err)
            self.pythoncopyfile(f'{src}/{file}',f'{dst}/{file}')

I am trying to test this function with pytest and it works fine when it is decorated.
However, i want to test the undecorated function.
I put in my test_file.py :
def test_undecorated_copyfile():
    sync=Sync()
    for file in range(3):
        sync.copyfile.__wrapped__('source_folder', 'dest_folder', f'{file}.txt')

And when i run pytest, it throws me "TypeError: copyfile() missing 1 required positional argument: 'file' "
So i guess this has something to do with how to handle "self" in the copyfile arguments, but i dont know where to begin with to understand what .__wrapped__ is doing wrong
I tried to look on the forum but all i get is how to undecorate a function (with ._wrapped_ ), how to deal with self in normal situations.
I dont know how to deal with this error and which objects or methods to use to investigate
Answer:
So i guess this has something to do with how to handle "self" in the copyfile arguments, but i dont know where to begin with to understand what .__wrapped__ is doing wrong

You're exactly right.
The .__wrapped__ attribute is an (unbound) function here (not a method)
print(sync.copyfile.__wrapped__)
#  prints <function Sync.copyfile ...>,
#  not    <bound method Sync.copyfile of <__main__.Sync object ...>

and therefore needs a "self" argument supplied explicitly, e.g.:
# Change
sync.copyfile.__wrapped__('source_folder', 'dest_folder', f'{file}.txt')

# to                      ↓↓↓↓
sync.copyfile.__wrapped__(sync, 'source_folder', 'dest_folder', f'{file}.txt')

Comment: thank you that was it! unbound function is same as static function right?
Comment: @kr1p Yeah, in python3 you can just think of it as any normal function (as you can static methods)


C# - How do I change the value of a property based on changes to another property (ObservableCollection)?

Question: How do I change the value of TotalPublicationsRead, when the Read property of a Publication has changed?
public class Report
{
   public ObservableCollection<Publication> Publications { get; set; }
   public int TotalPublicationsRead { get; set; }
}

public class Publication : INotifyPropertyChanged
{
   private bool read;
   public bool Read
   {
      get { return this.read; }
      set
      {
         if (this.read!= value)
         {
             this.publications = value;
             OnPropertyChanged("Read");
         }
      }
   }

   #region INotifyPropertyChanged Members

   public event PropertyChangedEventHandler PropertyChanged;

   #endregion

   private void OnPropertyChanged(string property)
   {
       if (this.PropertyChanged != null)
       {
           PropertyChanged(this, new PropertyChangedEventArgs(property));
       }
   }
}

Thanks in advance.
Answer: If you're trying to do what I think you are, then I'd change the TotalPublicationsRead property and forget about the events. In the code below I just count the items in the list where the Publication has been Read.
The way you were trying to do it you would have to have an event handler for when the ObserableCollection changed. Then you would have to attach an event handler to the PropertyChanged event which would increment or decrement the TotalPublicationsRead property. I'm sure it would work, but it would be a lot more complicated.
public class Report
{
    public List<Publication> Publications { get; set; }
    public int TotalPublicationsRead
    {
        get
        {
            return this.Publications.Count(p => p.Read);
        }
    }

}

public class Publication : INotifyPropertyChanged
{
    private bool read;
    public bool Read
    {
        get { return this.read; }
        set { this.read = value; }
    }
}

Answer: You can use Dependency Property.
Please check details at:
http://www.wpftutorial.net/dependencyproperties.html
http://msdn.microsoft.com/en-us/library/ms745795(v=vs.110).aspx


Interpretation of interaction terms with time in linear mixed-effects model

Question: As I posted here in reference of which model fits better the assumption, I reached the conclusion that this model is the better:
(lme(variable ~ time:group + group + time, random = ~ 1| subject)
,omitting the random slope because I only have 2 time points, following the advice from LuckyPal.
My experiment is based on multiple individuals organized in 3 groups under 3 different treatments with measurements in pre- and post-intervention. The goal is to observe significant difference between groups in several variables.
The output of my model varies according of which group (= grup_int as variable) of intervention should I put as reference level:
genes_long_gapdh$grup_int <- relevel(genes_long_gapdh$grup_int, "X"), X = 1, 2 or 3

As I said the output varies according to the reference group. The values I have are kind of like this (I ommit the df (minimum of 89), the efect(all fixed), the std error and the statistic):

#Group 1 as reference level

   var  group   term          estimate          p.value
ppara   NA  (Intercept)       3,6772            0,0000
ppara   NA  grup_int2         0,0723            0,7516
ppara   NA  grup_int3         -0,0979           0,6614
ppara   NA  time               0,0243           0,6893
ppara   NA  time:grup_int2   -0,0232            0,8004
ppara   NA  time:grup_int3    -0,0235           0,7901
ppard   NA  (Intercept)       0,8672            0,0000
ppard   NA  grup_int2        0,3188             0,1225
ppard   NA  grup_int3        -0,1764             0,3771
ppard   NA  time             -0,0409             0,4727
ppard   NA  time:grup_int2   -0,1242              0,1425
ppard   NA  time:grup_int3   0,0305              0,7092

#Group 2 as reference level

var group   term                estimate        p.value
ppara   NA  (Intercept)          3,7495         0,0000
ppara   NA  grup_int1            -0,0723        0,7516
ppara   NA  grup_int3            -0,1702         0,4707
ppara   NA  time                 0,0012          0,9863
ppara   NA  time:grup_int1        0,0232         0,8004
ppara   NA  time:grup_int3         -0,0003       0,9971
ppard   NA  (Intercept)            1,1860         0,0000
ppard   NA  grup_int1              -0,3188        0,1225
ppard   NA  grup_int3              -0,4952         0,0188
ppard   NA  time                   -0,1651          0,0088
ppard   NA  time:grup_int1           0,1242         0,1425
ppard   NA  time:grup_int3           0,1546         0,0720

#Group 3 as ref level

var    term                      estimate             p.value
ppara   (Intercept)             3,579282373           4,05E-43
ppara   grup_int1               0,097936121           0,661358561
ppara   grup_int2                0,170195822          0,470658635
ppara   time                     0,000834153          0,989590284
ppara   time:grup_int1           0,02348789           0,790110789
ppara   time:grup_int2           0,000337136          0,997128189
ppard   (Intercept)              0,690819735            3,62E-06
ppard   grup_int1                0,176353734            0,377121561
ppard   grup_int2                 0,49519246          0,018796677
ppard   time                     -0,01045055           0,858293253
ppard   time:grup_int1             -0,030457613       0,709158076
ppard   time:grup_int2              -0,154610426       0,072025362

As you can see:

What that the term time stands for? It is changing according the reference level, but is this the term meant to measure differences with the other 2 groups, like an overall p-value? Doesn't seem like that
Are the interaction time:grup_intX comparisons one-to-one group as a "two-sample t-test" in linear mixed-model (with the pertinent adjustments). I mean if the p-value showed comes from a direct comparison between groups
Is it possible to obtain an overall p-value comparing all three groups interaction with time?
Comment: You seem to be interested in comparing the fixed effects (the interactions in particular) and not in the random effects themselves. In that case, there are alternatives to the mixed model (eg. generalized least squares), if you'd like to explore. PS: The goal shouldn't be to "observe significant differences between groups" but to estimate what the differences are.
Comment: The interpretation of the interaction terms has little to do with whether or not you use a linear mixed model or not. The random effects have no interactions.
Comment: @SextusEmpiricus I presented my work, I ignore if using different model, interpretation of interaction in predictors behave the same way. Is usually the same pattern?
Comment: @JavierHernando so your question is purely about the interpretation of interaction terms and the random effects are just an additional unrelated aspect of your problem?
Comment: @SextusEmpiricus at the beginning yes, but if you consider something related to random effects could contribute because of a different criteria or something I've failed to understand, is welcome.
Answer: There are two frequent sources of confusion here related to your first question.
For one, with a multi-level categorical predictor under the default treatment/dummy coding in R, the reports of coefficients for non-reference levels are for differences from the reference category. The p-value reports are for the significance of the difference of each coefficient from 0. Thus, even without an interaction, the coefficients and their individual "significance" depends on the choice of reference.
For the other, when a predictor is involved in an interaction its individual coefficient represents its association with outcome when its interacting predictors are at reference or 0 values. So for your individual "time" coefficient (evidently modeled as linearly associated with outcome), you find what is expected: its value is its estimated association with outcome in whatever group you have chosen as the reference. Similarly, the "group" coefficients are their differences from the reference group in associations with outcome when time = 0. If you re-centered your time values those "group" coefficients would change.
With respect to your second question, the p-values for all coefficients, including for interaction terms, are based on the coefficient estimates and their variances. That's a t-test, but the coefficient estimates and variances are based on all of the data, not just a a comparison among specific groups. See this page for an explanation in ordinary least squares; the principle is similar in your mixed model although there is dispute about the choice of the appropriate number of degrees of freedom. In particular, there is a single estimate of residual error, based on all the data, that is used to estimate the coefficient variances.
With respect to your third question, some flavor of an anova() function can be used to evaluate all terms involving a single predictor, or any set of predictors, or all interaction terms. For example, you can use a likelihood-ratio-based anova() to compare a model without the interactions of interest to one with the interactions. For any combination of coefficients, you can use a Wald "chunk" test based on the variance-covariance matrix of the coefficient estimates. The Anova() function in the R car package is a convenient way to perform such tests on all coefficients involving each individual predictor and the combinations of interaction terms.
Comment: 1st question  Great explanation, appreciate it.I don't follow why anyone should center time (avoid multicollinearity is mentioned in the post), and why has this effect
2nd question Should I interpret time:grup_int2 : p< 0.05 with an estimate of 0.5 as an increasing trend across time of group 2 in reference to reference level? The way is calculated the interaction term allows to state this?
3rd question -  ANOVA with and without interaction to test the model, with and without the interaction or factor, and this will provide sthg similar an overall p-value between all levels of the factor?
Comment: @JavierHernando Many people are trained to center continuous variables involved in interactions, but there's seldom a need to. [This answer](https://stats.stackexchange.com/a/417159/28500) illustrates what's going on in a simple case. An interaction coefficient is the _extra_ change in outcome _beyond_ what you would predict based on the lower-level coefficients alone. So your interpretation of the coefficient seems correct. ANOVA comparison of a model with and without the interaction provides one type of _p_-value; it evaluates the significance of all the terms omitted in the simpler model.
Answer: This is how your effects looks like. You have three groups and two time points giving 2x3=6 values. In the graph the two time points are shown on the x-axis and the three groups are shown by using lines with different colours.
Comment: This is how it looks like, but is there  something to grasp related to random effects?
I've run with with random slope and without, and there is almost no change in the estimates.


Does the density $g(y) \propto (1-y^2)^{(n-3)/2} e^{\delta y} \quad\text{for}\quad |y| \leqslant 1$ have a name?

Question: The following probability density function has a particularly simple form, and it was produced when deriving a confidence interval for $\frac{\mu}{\sigma^2}$ , $$g(y;\delta)=c_\delta(1-y^2)^{(n-3)/2}e^{\delta y}\quad\quad \text{for}\quad-1\leq y\leq 1 $$
where $$c_\delta ={{(|\delta |/2)^{n/2-1}}\over{\sqrt{\pi}\, \Gamma((n-1)/2)I_{n/2-1}(|\delta|)}} $$ and where $I_{n/2-1}(|\delta|)$is the modified Bessel function of the first kind of order $n/2-1$.  For a description of the context and derivation of the $g(y;\delta)$ density from the Normal distribution, see Bornholt (2022) (pp. 26, 37).
My question is, does this probability density function have a name already? If not, has anyone seen it used in other settings?
Comment: This is not a generalised Beta...
Comment: I've not seen this distribution before, but it is quite an interesting one.  Would you mind elaborating on the context in which it was derived?
Comment: This density came up in my research that involves complementing hypothesis test results with confidences in the competing hypotheses (based on the sample evidence alone). For one example, I needed a confidence interval for $\frac{\mu}{\sigma^2}$ for n i.i.d. $X_i\sim N(\mu,\sigma^2)$. From a suggestion in the Cox and Hinkley (1974) text, the method is based on the conditional distribution of $Y=\bar X(n/\sum_iX_i^2)^{1/2}$ given $Y_2=y_2$, where $Y_2=\sum_iX_i^2$. The $\delta$ in the above density is defined by $\delta=\sqrt{ny_2}\mu/{\sigma^2}$.
Comment: For a fuller description of the context and derivation of the $g(y;\delta)$ density from the Normal distribution, see pages 26 and 37 of https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3973254
Answer: I don't know a name for this distribution, but here is some information that might (or might not) be useful
I have not seen this density function before; a quick search from me did not yield a name.  However, it looks to me like this density might be something that emerges from some kind of monotonic density transform, so I'm going to explore that to see if anything useful comes up.  (You haven't specified your parameter ranges, so I'm going to assume that $\delta>0$
.)  Taking $a \equiv (n-3)/2$ for purposes of simplification, your density function is proportionate to the kernel:
$$g(y) \propto (1-y^2)^a \exp(\delta y) \cdot \mathbb{I}(-1 \leqslant y \leqslant 1).$$
Suppose we now investigate this density only over its support so that we can omit the indicator term.  We will try to find a transformation that is monotonic over this support and gives some simplified form.  Below I will show two transformations that are interesting (hat tip to JimB in comments for the second one).

Sinusoidal transformation: Taking $y = \sin(x)$ we have $dy = \cos(x) \ dx$ which then gives:
$$\begin{align}
g(y) \ dy
&\propto (1-\sin^2(x))^a \exp(\delta \sin(x)) \ dy \\[12pt]
&= \cos^2(x)^a \exp(\delta \sin(x)) \ dy \\[12pt]
&= \cos^{2a}(x) \exp(\delta \sin(x)) \ dy \\[12pt]
&= \cos^{n-3}(x) \exp(\delta \sin(x)) \ dy \\[8pt]
&= \cos^{n-2}(x) \exp(\delta \sin(x)) \ \frac{dy}{\cos(x)} \\[6pt]
&= \cos^{n-2}(x) \exp(\delta \sin(x)) \ dx. \\[6pt]
\end{align}$$
Consequently, the related random variable $X = \arcsin(Y)$ has the density kernel:
$$f(x) \propto \cos^{n-2}(x) \exp(\delta \sin(x)) \cdot \mathbb{I}(-\tfrac{\pi}{2} \leqslant x \leqslant \tfrac{\pi}{2}).$$
This is also not a density I recognise as any named class, but it is quite an interesting density.  In the special case where $n=3$ this latter density has a closed form for its CDF, given by:
$$F(x) = \frac{\exp(\delta \sin(x)) - \exp(-\delta)}{\exp(\delta) - \exp(-\delta)}
\quad \quad \quad \quad \quad \text{for } -\tfrac{\pi}{2} \leqslant x \leqslant \tfrac{\pi}{2}.$$
For higher values of $n$ the distribution becomes more complicated and it does not have a closed form CDF.

Hyperbolic-tangent transformation: Taking $y = \tanh(x)$ we have $dy = \text{sech}^2(x) \ dx$ which then gives:
$$\begin{align}
g(y) \ dy
&\propto (1-\tanh^2(x))^a \exp(\delta \tanh(x)) \ dy \\[12pt]
&= \text{sech}^2(x)^a \exp(\delta \tanh(x)) \ dy \\[12pt]
&= \text{sech}^{2a}(x) \exp(\delta \tanh(x)) \ dy \\[12pt]
&= \text{sech}^{n-3}(x) \exp(\delta \tanh(x)) \ dy \\[8pt]
&= \text{sech}^{n-1}(x) \exp(\delta \tanh(x)) \ \frac{dy}{\text{sech}^2(x)} \\[6pt]
&= \text{sech}^{n-1}(x) \exp(\delta \tanh(x)) \ dx \\[6pt]
\end{align}$$
Consequently, the related random variable $X = \text{arctanh}(Y)$ has the density kernel:
$$f(x) \propto \text{sech}^{n-1}(x) \exp(\delta \tanh(x)).$$
This is also not a density I recognise as any named class, but it is quite an interesting density.  In the special case where $n=3$ this latter density has a closed form for its CDF, given by:
$$F(x) = \frac{\exp(\delta \tanh(x)) - \exp(-\delta)}{\exp(\delta) - \exp(-\delta)}
\quad \quad \quad \quad \quad \text{for } x \in \mathbb{R}.$$
For higher values of $n$ the distribution becomes more complicated and it does not have a closed form CDF.

Hypergeometric transformation: We can abstract from the above transformations to use a transformation $y = H(x)$ satisfying the nonlinear ordinary differential equation:
$$H'(x) = (1-H(x)^2)^{-a}.$$
The solution to this differential equation uses a hypergeometric function:
$$x = \text{const} +  y \cdot {_2}F_1(\tfrac{1}{2}, -a; \tfrac{3}{2}; y).$$
It lets us write:
$$\begin{align}
g(y) \ dy
&\propto (1-H(x)^2)^a \exp(\delta H(x)) \ dy \\[12pt]
&= \frac{1}{H'(x)} \exp(\delta H(x)) \ dy \\[12pt]
&= \exp(\delta H(x)) \ dx, \\[12pt]
\end{align}$$
which gives:
$$f(x) \propto \exp(\delta H(x)) \cdot \mathbb{I}(x \in \mathscr{X}).$$
This is a complicated distribution, since it involves the hypergeometric function.  It is also not a named class of distributions that I'm aware of.
Comment: I wonder if $\tanh ^{-1}(y)$ might be a useful transformation as you describe because with n=5 and δ=0, the resulting density is that of the sample correlation coefficient from a bivariate normal with ρ=0 and sample size 6.
Comment: Thanks @Ben and@JimB. The only link I have to named distributions so far is that the n=3 case can be related to one case from the Von Mises-Fisher Distribution, and I think that is what you have found Ben. The connection with the sample correlation coefficient is interesting also.


Is not catching an exception undefined behavior?

Question: Consider the following code:
#include <iostream>

class Widget {
public:
    ~Widget() {
        std::cout << "Destructor Called!";
    }
};

void doStuff() {
    Widget w;
    throw 1;
}

int main() {
    doStuff();
}

Because the exception is not caught in main, the compiler can arrange for the program to call terminate immediately after the throw with ~Widget not called.
Since the standard does not specify whether or not the destructors are called, is this undefined behavior?
Note: The example is from this CppCon 2015 talk by Fedor Pikus.
Edit: This question asks about the compiler's freedom for destructor elision, however it is in a special, different situation. The answer to that question does not apply to this question.
Comment: @Anoop Rana you closed this as a dupe to a question where none of the answers provide any detail on the actual standard rules. As this question particularly asks about the formal details of what the language says (language-lawyer tag), I've reopened the question. If there's an appropriate language-lawyer (LL) dupe target we should definitely connect to it, but I propose we do not close LL questions to targets where neither the question nor the answers contain any LL details.
Comment: *Because the exception is not caught in main, the compiler can arrange for the program to call terminate* - this is the well defined behaviour, w/o the un prefix.
Comment: The bad happens if a destructor raises an exception, then abort happens immediately.
Comment: @273K *this is not true. All destructors are being called while stack unwinding until the thread root function is out.* I'm not sure what you mean. I ran the code above with MSVC and ```Destructor Called!``` was not output.
Comment: Implementation defined/unspecified behaviour does not mean the undefined behaviour in any case
Comment: I am glad you aware of it, but I have not asked you, so please don't discuss it with me. `std::terminate` by default simply calls C function `abort`.
Answer:
Is not catching an exception undefined behavior?

No, it is well-defined that this will result in a call to std::terminate (with no UB), albeit whether the stack is unwound before the call is implementation-defined, as per [except.terminate]:

/1 In some situations exception handling is abandoned for less subtle
error handling techniques.

[...]
when the exception handling mechanism cannot find a handler for a thrown exception ([except.handle]), or

/2 In such cases, the function std::terminate is called. In the
situation where no matching handler is found, it is
implementation-defined whether or not the stack is unwound before
std::terminate is called. [...] An implementation is not permitted to finish stack unwinding prematurely based on a determination that the unwind process will eventually cause a call to the function std::terminate
Comment: @MarkWallace Optional copy elision, for example in _named return value optimization_, and elision of `operator new`/`operator delete` calls are other examples where the standard allows the compiler to choose whether or not functions are called, independent of their potential side effects. It is not that uncommon in C++.
Comment: I was not aware that calling functions with side effects could be implementation defined. TIL.
Comment: @user17732522: Saying that a function call be omitted is far less horrible than characterizing all situations where useful optimizations might observably affect program behavior as UB.


Expectation of picking a number from a uniform distribution

Question:
Ms. A selects a number X randomly from the uniform distribution on $[0, 1]$. Then Mr. B repeatedly, and independently, draws numbers $Y_1$,$Y_2$,.... from the uniform distribution on $[0,1]$ until he gets a number larger than $X/2$, then stops. The expected number of draws that Mr. B makes equals
  (a) 2ln2 (b) ln 2 (c) 2/e (d) 6/e

I tried reaching a solution through integrating from X/2 to 1 to find the probability that a random variable Y would achieve a value greater than X/2. I then defined an RV Z which is equal to the maximum of all Ys picked, and imposed the condition that Z > X/2. But I got a hunch that my method was wrong and stopped midway, I cant come to think of any method either. A solution that I saw uses conditional expectation and the law of iterated expectations, but I'm finding it difficult to grasp in one stretch. This is the link to the solution that I saw,
https://sites.google.com/a/econschool.in/econschool/stuff-of-interest/anotherpost/dse-2014-q22
How do I solve this question, can somebody please help me out with a hint ?
Comment: Hey Parcly, Thanks a lot for inquiring. I've posted my working above.
Answer: Hint:
If $N$ denotes the number of draws needed by Mr. B then $N$ follows geometric distribution with "random parameter" $1-X/2$.
That gives: $$\mathbb EN=\mathbb E\left[\mathbb E[N\mid X]\right]=\mathbb E \left[\frac{1}{1-X/2}\right]$$
Comment: Actually $N$ is not a random variable with geometric distribution. Though for every fixed $x\in[0,1]$ random variable $N$ has a geometric distribution with parameter $1-x/2$ *under condition* $X=x$. This is enough to find $\mathbb EN$ and it is not necessary to dive into the (unconditional) distribution of $N$. If you still want to then to be found is $P(N=k)=\int_0^1 P(N=k\mid X=x)dx=\int_0^1(x/2)^{k-1}(1-x/2)dx$. Doing so you will find that $N$ taken unconditionally is not a geometric random variable.
Comment: How can we define N, a discrete geometric RV on something that follows a continuous distribution. Could you please give some insight, thanks a lot. I guess I've understood a lot more than before.
Comment: Thanks a lot drhab!
Comment: You are very welcome.
Answer: For a fixed $x\in [0,1]$, let $w$ be the expected number of rounds until $y > {\large{\frac{x}{2}}}$.

Then we have the equation
$$w=\left(1-{\small{\frac{x}{2}}}\right)(1) + \left({\small{\frac{x}{2}}}\right)(1+w)$$
which yields
$$w=\frac{2}{2-x}$$
hence, given that $x$ is uniformly distributed on $[0,1]$, the expected number of rounds is
$$\int_{0}^{1}\frac{2}{2-x}\;dx = 2\ln(2)$$


Is it possible to put existing/shared Content type fields as Feature code each in their own individual file to simplify/avoid git merge conflicts?

Question: We have a set of content types that share common fields - in that we re-use them in several content types.
When recreating these content types as a Feature, the fields are all in one large file. This has proven to be difficult when resolving git merge conflicts when people adjust/remove/or add fields to content types as a normal part of development. The git ===== >>>>> <<<<< conflict markers 'cross over' different fields and make it hard to see how the merge should be done correctly.
I think a better solution is to have each field in its own file, even if there are hundreds, the host file system should be able to handle this, perhaps even use of directories may help, this would avoid complex merge conflict resolutions.
Is this single file per field possible?
Answer: We faced similar problems, what we did was, we created a 4 features of similar fields that were used in multiple content types. This made sure that if a person is working on a similar funtionaltiy, he is responsible for all the content types.
Yes, we still need to be careful for this, but dividing it into multiple features did solve many issues.


how can a telegram bot send an aligned table? I tried sending a table image but it clogs up the user's memory?

Question: I'm making a telegram bot shop and there is a problem related with a basket. I want my telebot to send an aligned table but i got this
This is what my bot sends

This is a pandas table converted to Markdown. How can i fix my bot ?
Answer: Telegram doesn't have the ability to render tables inside messages. The best thing you can do is use <code>```</code> quotes to make font monospaced. In that case, you can write your table using tabulation


Understanding a counterexample to Hasse principle

Question: I'd like to understand a step in a counterexample of Reichardt and Lind to the Hasse principle. The example is given by the equation $2y^2=x^4-17z^4$ (1).
(1) has no rational solutions ($\neq (0,0,0)$), but it has non-zero solutions in $\mathbb{Q}_{p}$ for all $p$. The part that I don't get is the justification that the equation has solutions in $\mathbb{Q}_{p}$ for all $p$. First one can find explicit solutions for $p=2,17,\infty$. For the other values of $p$ the idea is to first show the existence of non-zero solutions over $\mathbb{F}_{p}$, and then use Hensel's lemma to lift these solutions. To show that (1) has solutions over $\mathbb{F}_{p}$ what I've seen people doing is using Hasse's bound on number of points over $\mathbb{F}_p$ of an elliptic curve. (see http://www.uni-math.gwdg.de/aufzeichnungen/SummerSchool/SummerSchool_20060717-1430_Kresch/avi/SummerSchool_20060717-1430_Kresch_xvid.avi between minutes 15 and 30) or http://www.warwick.ac.uk/~maseap/arith/notes/elementary.pdf
The key point is the following observation:
"The curve defined by the equation (1) has genus 1"
and thats precisely what I don't get, so my question is:
Question: Why is that a curve? and moreover why does it have genus 1?
Thank you.
Comment: "genus 1 means it's an elliptic curve" that's true only if you have a point. "Any polynomial defines a curve as its set of solutions" that's definitely not true, in fact your example -which is intended to be over $\mathbb{R}$- is not a curve it is a 2-dimensional variety. The coincidence here is that as topological space you can identify it with $\mathbb{C}\cup \infty$, a 1-dimensional object over $\mathbb{C}$ with no holes.
Comment: The above comment has no sense now that user quanta erased his/her comments. I'll recall his/her comments so mine makes sense. "genus 1 means that it is an elliptic curve..." and "yes it is a curve, any polynomial defines a curve e.g. $x^2 + y^2 + z^2=1$ which is a curve of genus 0"
Answer: I don't really know what you mean when you ask "Why is that a curve?". The question can be taken to mean either "I don't know what an algebraic curve is", or "Since there are three unknowns and two equations, it looks more like a surface to me". I am going to assume the latter.
So recall that a curve is most often defined either by a set of homogeneous equations in $n+1$ variables, in which case their solutions form a well defined subvariety of the projective space $\mathbf P^n$, or a set of arbitrary equations in $n$ variables, in which case their solutions are taken inside of affine space $\mathbf A^n$. We can go from one to the other by setting one of the variables equal to one and homogenizing respectively (but note that this process is not bijective).
So note first that our equation is not homogeneous. Based on the previous paragraph it would make most sense to interpret it as a subvariety of $\mathbf A^3$, but this is clearly not what they mean since that would give us an algebraic surface and not an algebraic curve.
Here are two ways to make sense of this:

Set $z=1$, for example. Then we get the equation $2y^2 = x^4 - 17$ in two variables, and whenever we have a solution to this equation we also have a solution to the original one. This equation defines a plane curve in $\mathbf A^2$. By embedding $\mathbf A^2$ in a projective space and taking the closure we get a projective curve, and by desingularizing it we also get a smooth projective curve.
Give the variable $y$ degree $2$, so that the equation is homogeneous with respect to a shifted degree. Then we can talk about the set of solutions inside of a weighted projective space. (Over $\mathbf C$ this can be defined as $\mathbf C^3 / \mathbf C^\ast$ where $\mathbf C^\ast$ acts by $\lambda \cdot (x_0,x_1,x_2) = (\lambda^2 x_0, \lambda x_1, \lambda x_2)$.)

Your second question - how do we see that this curve has genus one - can be determined in several ways. I am not sure what is the simplest way but here is one, using model (1) above and calculations in affine coordinates. Your curve has a degree 2 map to $\mathbf A^1$ given by projection to the $x$-coordinate, and this map will have branch points wherever $x^4-17 = 0$. The map can be extended to a map from the smooth projective version of the curve to $\mathbf P^1$ (this is a general fact about smooth curves, but you can also prove this by computing in explicit coordinates, see the Wikipedia article on hyperelliptic curves). Using the Riemann-Hurwitz formula one sees that:

a map of degree two must have an even number of branch points, so the point $x = \infty$ can not be a branch point;
a curve which has a degree two map to $\mathbf P^1$ with four branch points has genus one.

Oh, and please tell me which parts of this need elaborating.
Comment: In principle there is an algorithm for resolving any singularity of a curve: just keep blowing up and it will eventually disappear. But I would avoid doing this kind of explicit computation as far as possible so maybe it was misleading that I talked about blow-ups in my answer. For a curve of the type $y^2 = f(x)$ where $\deg f = 2g$ or $\deg f = 2g+1$, you can get a nonsingular model by gluing two affine pieces $y^2 = f(x)$ and $z^2 = w^{2g}f(1/w)$, which is a lot easier to compute with than whatever you would get by blowing up...
Comment: Dear Dan: First, this is a great answer thank you. I've in fact thought this along the lines of approach (1) so my conclusion is that saying that "the equation defines a curve" is wrong, however, as you point out, a point in the plain curve produces a point on the surface. Here is something that I'vent been able to do though. I'm trying to resolve the singularity $[0:1:0]$ of the projective closure $2y^{2}u^{2}=x^4-17u^4$ of the curve, but I'm having not luck at it. I'm not sure how to do blow ups of projective curves so I tried to resolve $2u^{2}=x^4-17u^4$ at $[0:0]$ but for some reason...
Comment: ... I did not get a smooth curve. I'm guessing that if I do the right calculations I'd get a cubic plane curve - a Weierstrass model - which is birational to $2y^{2}=x^4-17$. I imagine there are references that show how to get a cubic equation from an equation $y^2=f(x)$ where $f$ is a quartic polynomial with no repeated roots - I see from your answer why such equation exists- but imagine in practice there is some sort of algorithm that shows how to do it, and does not mention coverings, genus, etc...
Comment: I'm trying to calculate the above example to see if I can derive such algorithm. If you know any reference for it that would be great. Thanks.
Comment: As for your algorithm, an elliptic curve is determined by the four points on $\mathbf P^1$ which are the branch points of the double cover (and the choice of one of them as the origin) and in the resulting equation $y^2 = f(x)$ the polynomial $f$ has degree three precisely if one of the four points is the point at infinity, and degree four otherwise. So you just need to pick an automorphism of $\mathbf P^1$ sending one of the roots to infinity and of course there are many such maps. This can be written as a change of coordinates, without mentioning geometry, but why would you want to do that?
Comment: I guess I just wanted to see the equation that one gets at the end without having to do blow ups. In my case I started with $2y^{2}=x^4-17$ but it seems I have to blow up several times the homogenization which I did not succeed. I don't understand what you mean by "get a non singular model by gluing two affine pieces" In this case I have $2y^{2}=x^4-17$ and $2z^2=1-17u^4$, so how can I get the singular model from these two?
Comment: above I meant non-singular instead of "singular model from..."
Comment: Let me change the notation so the degree is $2g+1$ or $2g+2$ (this is what I meant originally.) You would glue the part where $x \neq 0$ to the part where $w \neq 0$ by the identifications $w = 1/x$, $z = y/x^{g+1}$.


Evolution strategies global recombination question

Question: In evolution strategies the genotype has object variables and strategy variables as alleles (and sometimes the alpha values).
I'm reading the book Introduction to Evolutionary Computing (A.E. Eiben, J.E. Smith) and there is something that I don't understand:

An extension of this scheme allows the use of more than two
  recombinants, because the two parents x and y are drawn randomly for
  each position i {1,...,n} in the offspring anew.

I don't understand what the n value is:

The number of object variables.
The number of strategy parameters.
The number of object variables + the number of strategy parameters.

What is the correct value of n?
Answer: Take a look at: http://liacs.leidenuniv.nl/~nijssensgr/CI/2012/9%20genetic%20algorithm.pdf
"Per offspring gene two parent genes are involved
Choices:
-combination of  two parent genes:
--average value of parents (intermediate recombination)
--value of one randomly selected parent (discrete recombination)
-choice of parents:
--a different pair of parents for each gene (global recombination)
--the same pair of parents for all genes"


Can't display diacritic characters within Cygwin terminal

Question: Both in Cygwin (minnty) and Git Bash I have set font to pl_PL/ISO-8859-2. It results (in both cases) in:
$ locale
LANG=pl_PL.ISO-8859-2
LC_CTYPE="pl_PL.ISO-8859-2"
LC_NUMERIC="pl_PL.ISO-8859-2"
LC_TIME="pl_PL.ISO-8859-2"
LC_COLLATE="pl_PL.ISO-8859-2"
LC_MONETARY="pl_PL.ISO-8859-2"
LC_MESSAGES="pl_PL.ISO-8859-2"
LC_ALL=

But in Cygwin, when I display Git commits' comments - diacritical characters are not displayed properly (regardless of the font I use - e.g., Consolas, Lucida Console). The same commits' comments are properly displayed in Git Bash. The comments have been UTF-8 encoded while saving.
I didn't have this issue before, but once I reinstalled the OS now it occurs. Unfortunately I can't tell what I have done differently.
Is it possible to fix that? I use Windows 7 32-bit.
Comment: what is the issue using `LANG=pl_PL.UTF-8` ?
Comment: It doesn't show Polish diacritic characters of Git commits' comments. But, again, only within Cygwin - the Git Bash itself displays those characters correctly, with the same ISO-8859-2 and font.
Answer: In general, you're better off using UTF-8 and setting your locale to pl_PL.UTF-8.  UTF-8 is capable of supporting all languages, including Polish.  If that doesn't work, then there's a different issue which should be addressed independently.  Git by default uses UTF-8 in most cases, and so does pretty much everything else on a Unix or Cygwin system.
If you really, absolutely insist upon using ISO-8859-2, then you'll need to set the i18n.logOutputEncoding to ISO-8859-2 so that it will render things in the proper character set.  Note that if you have this set in one Git configuration (the Git Bash one) and not in the other (the Cygwin one), that will explain why you're seeing different behavior.
You may also want to set your system code page to UTF-8 by using chcp 65001 in a CMD window.
Comment: > If that doesn't work, then there's a different issue which should be addressed independently

Thank You for Your answer. With UTF-8 it also didn't work in my case (under Windows 7):

$ locale
LANG=pl_PL.UTF-8
LC_CTYPE="pl_PL.UTF-8"
LC_NUMERIC="pl_PL.UTF-8"
LC_TIME="pl_PL.UTF-8"
LC_COLLATE="pl_PL.UTF-8"
LC_MONETARY="pl_PL.UTF-8"
LC_MESSAGES="pl_PL.UTF-8"
LC_ALL=

If, however, the issue does not occur in Git Bash, what would be the easiest way You could recommend to solve this? I'm rather a non-technical user, so many of those things are still new and abstract to me.
Comment: Have you set the mintty terminal settings to UTF-8 from the Options menu?
Comment: Yes, mintty; Consolas; pl_PL; UTF-8

Can’t display diacritic characters within Cygwin terminal


If two groups are generated by the same number of generators, and the generators have the same orders, are the groups isomorphic?

Question: If, say, for two groups G and H, G = < a1, a2, ... , a_n > and H = < b1, b2, ... , b_n >,
such that |a_i| = |b_i| for all i from 1 to n,
is G isomorphic to H?
If so, what is the proof of this?
I know that for n = 1, then H and G are a cyclic group and are isomorphic to Z/|a|Z. Does this extend to larger generating sets?
Comment: Try to find counterexamples. $n = 2$ suffices.
Answer: NO.
In $S_4$ , $H=<(13),(1234)>$ is isomorphic to $D_4$ but in $\mathbb{Z_4}$, $K=<2,1>$ is isomorphic to $\mathbb{Z_4}$, so $K  \ncong H$ even when both are generated by two elements which have same orders respectively.
Answer: No; for n=2, we get two distinct groups, where $1$ is the identity
$$G=<a,b | a^2 = 1, aba^{-1}b = 1, b^2=1>$$
$$H=<a,b | a^2 = 1, b^2 = 1>$$
Notice that $G$ is finite - it is $\mathbb{Z_2\times Z_2}$. However, $H$ can be described as the set of words which alternate $a$ and $b$ (with the appropriate multiplication operator) - which is infinite.
Answer: HINT: (from @Daniel Fischer:)
Take the different groups $D_n$ of order $2n$ generated by two reflections (elements of order $2$) across two lines in the plane that form  an angle of $\frac{\pi}{n}$.


What role does the sampling fraction play in the simple random sample variance formula?

Question: Consider a survey where the population being estimated is finite.  Using a simple random sample, you draw $n<N$ data.  The sampling fraction is $f=\frac{n}{N}$.  The variance will be $(1-f)\frac{s^2}{n}$.  Can someone explain how the sampling fraction effects this formula?
Comment: Can you clarify your question? Is it about the finite population correction?
Comment: @gung Yes! How does it affect the variance of a simple random sample?
Comment: @gung Do you understand my question?
Comment: Did I get it wrong? If so, please edit to correct it. Just make sure it is clear what you are asking.
Answer: In many cases, $f$ is neglected when evaluating the sampling variance. Think about it : if you sample $n = 10 000~$ units among $~N = 300 000 000~$, then $~ f \approx 3 \cdot 10^{-6} \ll 1$.
The most important terms of this formula are : $n$, the sample size, and $s^2$, the variance of the variable you're trying to measure.
Nevertheless, it is somehow logical to see $f$ appear in the formula. Indeed, if $f = 1$ (census), variance of the SRS has to be $0$.


Multiple choice question - Able to have the question correct with any correct answer

Question: I am using the Quiz and the Multiple choice modules. In a question, I have two answers correct and I'm hoping to have the question marked right, if any of the answer is chosen.
Right now, if both answers are chosen, only then the question is marked as correct by the module.
Answer: There is an option " Score if chosen and Score if not chosen " in multichoice question.
Set Score if not chosen = 1 and Score If chosen = 1
I hope, It would fix the issue.
Comment: Thanks a lot. But in this way, if both of these are not choosen, it'll still correct it


How to create custom SwiftUI View playground template in Xcode?

Question: I want to create a custom Xcode playground template for SwiftUI View.
What file formats are used?
How can I find examples?
Where should I put those files?
Answer: You can make template by duplicating file templates at path
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Templates/File Templates/Playground

Note that each platform has it's own templates.

Copy two folders TEMPLATE_NAME.xctemplate and TEMPLATE_NAME___FileMenu.xctemplate
to ~/Library/Developer/Xcode/Templates/File Templates and rename them as you want.

(you can put it in subfolder SwiftUI, it's name will be used to separate your templates from others)

To change the displayed name of template you can change Name field in TemplateInfo.plist
Paste the code of your template into ___FILEBASENAME___.playground file (you can use placeholders to generate code base on parameters, see for example this article)

Here are my templates as an example: https://github.com/DanSkeel/XcodeTemplates/tree/main/Playground

Note: ...___FileMenu.xctemplate files will be used when you create new file from opened playground.


How to use Views to obtain this?

Question: I want to use the Views module to obtain the number of comments associated to a node, and show it as "1 comment," "5 comments."
How do I do this?
Answer: Checkout the Views Group By module, this will do it. Here is a quote about it (from the module's project page):

... enriches Views2 functionality with SQL Grouping and Aggregation capabilities.
A new field group: SQL Aggregation is added, in Views, once you install the module. The field group has one available field: SQL Aggregation: Group By Fields. The field allows to configure many aspects of "group by" SQL clause and "count()" fields.
Comment: Or native group by in Views 3.

How to use Views to obtain this?


how to create template file for single page in drupal 7

Question: i have created "about us" page in drupal 7 admin interface of basic page type. I want to apply a specific layout for this "about us" page only.
But after goggling, i found solution of applying specific theme to specific content type like
page--content_type.tpl.php.
i dont want to apply layout for all pages in that content type, but only for this specific page "about us". If this can be done by overriding like this "page--node--nodeid.tpl.php", then where to find the nodeid for this page.
Answer: use "page--node--nodeid.tpl.php" template. you can find the nodeid for this page when editing.
edit "about us" page, you will got url like this http://www.example.com/node/1/edit
digit "1" in the url is node-id.


How can I initialize a new object inside a construction function so that each of its function can reuse it?

Question: function x() {
  var obj1 = new ErrorPrinter();
}
x.prototype = {
 xFunction1 : function() {
    return obj1.printthis("x");
 },

 xFunction2 : function() {
    return obj1.printthis("y");
 }
}

But when I do the above, it wont work. The whole purpose of initializing the object on top is the so that I don't have to recreate the same object in each function(xFunction1 and xFunction2).
Thx for such a wonderful community, stackoverflow.com.
Answer: Declaring a variable as you did will only make it available inside the constructor function itself, not other functions on the prototype.
On the other hand, if you were to set this.obj1 = ... in the constructor, it would be available as this.obj1 in the other prototype methods when they are called on an instance.
function X() {
  this.obj1 = new ErrorPrinter();
}
X.prototype = {
  xFunction1: function() {
    return this.obj1.printthis("x");
  },

  xFunction2: function() {
    return this.obj1.printthis("y");
  }
};


convergence of infinite sum with indicator function of another variable

Question: Suppose that ${X_n}$ are independent, $S_n = X_1 + ... X_n \rightarrow S$ in probability for some random variable S.
If $Y_n$ are identically distributed with $E|Y_1|<\infty$,
show that $T_n = \sum_{i=1}^{n} X_i 1(|Y_i|\leq i) \rightarrow T$ almost surely for some random variable T.
Is there any good way to solve this? Usually, the form can be solved because the variance of infinite sum is finite, so we can use 3-series theorem. I think it requires similar way, but the existence of unrelated variable $Y_i$ makes it hard.
Comment: Are $Y_n$'s independent?
Comment: @d.k.o. In the problem, there is no such condition. Maybe the problem omits independence of Yn, but I'm not sure.
Answer: First, $S_n\xrightarrow{p}S \Leftrightarrow S_n\xrightarrow{a.s.}S$. Since $\sum_{i\ge 1}X_i(\omega)$ converges (for almost all $\omega$), it suffices to show that $\sum_{i\ge 1}|Z_i-Z_{i+1}|<\infty$ a.s., where $Z_i:=1\{|Y_i|\le i\}$. But
$$
\mathsf{E}\sum_{i\ge 1}|Z_i-Z_{i+1}|\le 2\sum_{i\ge 1}\mathsf{P}(|Y_1|>i)< \infty
$$
because $\mathsf{E}|Y_1|<\infty$ so that $\sum_{i}|Z_i-Z_{i+1}|<\infty$ a.s.


Get the total number of nodes and counting nodes

Question: Is it possible to get the total number of XML nodes?
Also, how does one do a for-statement with XSLT?
Answer: If you want to count the node of you XML you could use count(object/node) and it will be something like this:
<code> <xsl:value-of select="count(/root/*)"/>
</code>
The instruction above will let you know how many child nodes does your root node has
<code><xsl:for-each select="/root/element-you-wanna-loop" >
    <!-- Do something with your nodes here -->
</xsl:for-each>
</code>
Hope this helps you.
Comment: How would I get the current node number?
Comment: you should replace the /root for you current node if you're in that node you could just use the count(*)


Beginners Calculus Question: $\lim_{x\to 2}\frac {g(x)}{h(x)}$ when $\lim_{x\to 2} g(x) = -2$ and $\lim_{x\to -2}\lim h(x) =0$

Question: Beginners Calculus Question: $\lim_{x\to2}\frac{g(x)}{h(x)}$  when $\lim_{x\to 2}g(x)=-2$ and $\lim_{x\to 2}h(x)=0$
How can one evaluate when the denominator is 0? Please remember that I am a calculus student.
Answer: This limit doesn't exist.  The only way you can have a limit exist of a rational function when the denominator goes to 0 is if the numerator ALSO goes to 0.  In that case, we have what is called an indeterminant form,  for which you can either use algebra or L'hospital's rule to handle.
Here,  you don't have that.   Logically, what you have is the top is approaching 2, the bottom is approaching 0....as we take numbers close to 2, and divide them by numbers close to 0,  we get numbers that explode, either to positive or negative infinity...which don't approach any finite limit
Answer: If you divide $f(x)$ by $g(x)$ the factors cancels out so that your limit might be finite else it is infinite.


Confidence intervals for accuracy metrics: pros and cons of the bootstrap method compared to variance estimation

Question: I want to report the CI for metrics like F1 and AUC.
I'm a bit confused on when it's better to bootstrap it, or when to use a formula.
For F1 there are several works that estimate the variance and derive a (symmetric) confidence interval, like:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8936911.
Similarly for AUC there is the Delong method, and others.
However for all these metrics, we could bootstrap them by sampling with replacement, computing the metric for every sample, and then taking the percentiles of the distribution, like the 2.5% 97.5% and percentiles.
Which of these approaches is expected to be better?
I guess an issue with the bootstrap method is that it's slow to compute it thousands of times. Is that the only reason there are works to create a closed formula for the CI ?
Answer: I tend to have data with inherent structure, such as multiple samples per patient, multiple measurements per sample and the like.
In that situation, the statistically independent unit is typically the patient rather than the row of the data matrix. (In some cases, independence is even more complicated, with multiple top-level sources of variance.)
While I can easily bootstrap patients (or more general: set up a resampling procedure so that statistical independence is obeyed*) the analytical c.i. calculations I've met so far assume statistical independence at the row level. I.e., they do not accomodate more complex real-life data structure.
It's not that I would speak against using analytical formulations of variance (though I'd recommend to closely check the "small print" of the respective method) - I just don't meet nice data that would allow me to use them...
However, if your data meets the requirements of the analytical/parametric calculation, I'd expect it to have more power than bootstrapping.
What I do use is e.g. binomial c.i.s for proportions like sensitivity, specificity or the predictive values with patient number as sample size. Often, I can show that model instability is negligible compared to this, and I thus get a sensible approximation. (And the result is often anyways so wide that it does a good job serving as warning not to overinterprete the results...)
Sometimes I use this also as back-of-the-envelope calculation before experiments start: the c.i. won't get any better than that - if that is not satisfactory, more patients are needed right from the beginning.

Mixed models that describe the underlying structure for the prediction error may be a way here, but I haven't seen any papers applying this so far. It is, however, an approach that I'm looking into.
I have a sneaking suspicion that situations where the variance on the error estimate can sensibly be approximated as stemming from a single source are actually very rare (and my educated guess is that this is where patient-to-patient variance dominates everything and if that's the case with the final error estimate, it is rarely of practical use).
Of course, one may still sum the contributions of various sources of error into one total variance - but also there, analytical calculation would need to account for the data structure and the corresponding sources of variance.

*Since I typically also work with high-dimensional data, I anyways have to set up such splitting procedures to generate suitable test data to produce the generalization error estimates: the fanciest variance estimator is of no use if my validation procedure has a (large) optimistic bias. Also keep in mind that optimistically biased test results on classification tasks often lead to  optimistically biased (too narrow) variance estimates as well.

Update: back of the envelope calculations
Since I don't use F1, I'll outline mostly what I do for sensitivity (or analogously specificity). Sensitivity is the (observed) fraction of correctly predicted tested cases of the positive class, and we can thus use binomial confidence intervals. The width of these depends on the denominator, i.e. the number of tested cases of the positive class, and on the true value of the sensitivity (with maximum width for 50 % sens and minimum width for 0 and 100 % sens).
For the minimum confidence interval width, we can use the rule of 3: assuming we's observe no errors, the 95 % c.i. for sensitivity would be roughy ranging from 1 to $1 - \frac{3}{n}$
However, we can also calculate minimum and maximum c.i. width depending on the number of tested positive cases before any experimental results are available:

... and compare that to the the requirements of the application.
(One can also set up calculation to estimate reasonable sample sizes based on expected performance and application requirements. One scenario I've been using is specifying for an application situation "super nice", "OK" and "not really acceptable" performance. If for the discussed sample size "super nice" performance couldn't be reliably distinguished from "not really acceptable", sample size is insufficient)
To get guesstimates for other measures that are not simply an observed fraction of a pre-specified number of test cases (such as F1), you could either start with the variance of the binomially distributed simple fractions and do error propagation, or set up simulations (which would be quite similar to bootstrapping).
Comment: Depends if each case is an independent assessment (i.e. each case assessed once with no repeated assessments). If so, you can use a Dirichlet(0.5, 0.25, 0.25, 0.25, 0.5, 0.25, 0.25, 0.25, 0.5) prior (0.5 on diagonal, 0.25 off-diagonal) for being in each cell of the confusion matrix, get the posterior (add observed count in each cell), you then draw random variables (=random confusion matrices) from this distribution & calculate the F1-score on them. CI := the 2.5th to 97.5th percentile of these score. Is that easier than boostrapping? Not sure, but should be okay if some cells are near zero.
Comment: Nice answer and agree with almost everything. Added my own perspective as another answer (mostly adding something regarding situations where you just can't bootstrap easily). Regarding your comment on "mixed models that describe the underlying structure for the prediction error", we have done something like that [recently](https://doi.org/10.1016/j.jaip.2023.01.017) as I mention in my answer.
Comment: @Björn: thank you for the literature links :-)
Comment: Thank you so much for such a detailed answer ! How would you do back of the envelope calculations about the CI for F1 for example. Would you assume some distribution of the scores for that?
Comment: @Björn: we don't even need the assumption of each observation being an independent case. We "only" need the random drawing to account for the repeated assessments: F1 calculation does not need integer numbers in the confusion matrix, so we can weight observations so that each independent experimental unit gets a total weight of 1 (possibly distributed across cells). We'd also need assumptions/estimates of variance and distribution for the different sources of variance (e.g. separate within subject, between subject). In the end, the structure is probably rather similar to the bootstrapping.
Comment: If you figure out how to do that for a MRMC study where there's, say, 100 cases, 200 raters and each rater only diagnoses 20 randomly chosen cases then I'll be impressed. Even worse, assume there's two different groups of raters (that e.g. had different training) and you want the difference in F1-score. If you bootstrap cases, you don't account for the variation coming from rater, if you bootstrap raters you don't account for variation from cases. Perhaps bootstrapping cases and raters somehow works? But never seen that described. I'd really like to see a simulation study to show that works...
Comment: @Björn: I'll start with the somewhat easier but closely related problem of setting up adequate splitting for cross validation. "bootstrap raters -> don't account for variation from cases": even worse, where like here generalization error is concerned: the data leak in cases causes optimistic bias (and that optimistic bias for figures of merit like accuracy, sensitivity, F1 is linked to underestimating variance), so that doesn't work at all.
Answer: There's situations where you can bootstrap. If your sample size is large, then bootstrapping when possible is really convenient for a number of reasons:

If it works, then it works for almost any metric you can define, while frequentist analytical solutions tend to be derived for one metric at a time (bad luck if your metric of interest is not covered)
It's easy to implement (assuming a straightforward bootstrapping scheme is appropriate for the situation and the metric is easy to calculate for a sample).

Where bootstrapping has problems is

When sample size is small, then the discreteness of bootstrapping can make it inefficient.
When another simple solution is easily available/already implemented, why do it, when it often takes longer to run/is a little bit more complex to implement than running a well-validated package that someone else already wrote?
Sometimes you just cannot bootstrap in a valid way. E.g. multiple-rate-multiple-case (MRMC) studies tend to make it very hard to bootstrap (some solutions still have a bootstrap component, but it's nightmarishly complicated*), because everything is in a way correlated with everything (you have both the same raters looking at cases, as well as the same cases being looked).

As the other answer mentioned, one approach is a form of mixed effects model that describes the whole process. We took that approach for a MRMC study, where there were 3 possible diagnoses, recently. The Methods Section of the article outlines the approach:

We analyzed the primary outcome using a Bayesian model that jointly
modeled each patient's true disease status (ie, expert panel
diagnosis) as a categorical random variable and the diagnoses given
for each patient by each physician or algorithm (determined using
multinomial logistic regression). The multinomial logistic regression
model included a separate intercept term for each combination of
disease (asthma, COPD, and ACO) and group (PCPs, pulmonologists, and
the AC/DC tool), as well as a random case and random rater effect.

We took this approach, because we could not find any published methods that applied to the metrics our clinical colleagues wanted to look at (most MRMC literature is about AuROC and as mentioned, simple bootstrapping does not work). We ended up fitting this in a Bayesian way using Stan, because that deals with the uncertainties sensibly. It might be possible to fit such a model with something like PROC NLMIXED in SAS and to get confidence intervals that way (they have a decent automatic procedure for SEs there that uses the delta method) in a large study, but the Bayesian approach deals more naturally with (nearly) empty cells.
* Regarding "nightmarishly complicated", look at the complexity in these papers for getting AuROC estimates with CIs from MRMC studies:

Brandon D. Gallas, Gene A. Pennello, and Kyle J. Myers. Multireader
multicase variance analysis for binary data. J. Opt. Soc. Am. A, 24(12):
B70–B80, Dec 2007. doi: 10.1364/JOSAA.24.000B70.
Brandon D. Gallas, Andriy Bandos, Frank W. Samuelson, and Robert F.
Wagner. A framework for random-effects roc analysis: Biases with the boot-
strap and other variance estimators. Communications in Statistics - Theory
and Methods, 38(15):2586–2603, 2009. doi: 10.1080/03610920802610084.
Comment: Nice answer :-) I'll have to read the papers, but I think the MRMC scenario is part of what I refer to has situations with crossed random factors at top level - which requires independence on both (all top-level) random factors. If that suspicion is correct, at least for my data I anywanys better set up the nightmarishly complicated resampling since I need the same independent splitting already during hyperparameter tuning.
Comment: Thank you so much for the detailed answer. How can we determine if the sample size is large enough to bootstrap? For example if I want to compute the CI for F1 for 100 scores, is it large enough? how about 1000?
Comment: Wrt small sample size situations, I take the following approach: for very small n one may as well do an exhaustive calculation of all possibilities for "ordered drawing with replacement". For somewhat larger n, a usual random bootstrapping. Sure, the results are then discrete, but that's because the underlying binomial distribution is inherently discrete (and that's what causes a substantial part of the high variance problems with these figures of merit). So a bit of care is needed to actually get at least the desired coverage. However, in practice for what I've seen so far the inefficiency is
Comment: ... of small consequence: while the coverage properties are not nice, that typically doesn't matter since they still very clearly give the practically relevant message of "too uncertain to allow any meaningful conclusions". (And the same happens also for formula-based calcuation: e.g. for binomial proportions, a variety of different approximations for the c.i. are available, and they can differ for small n. But in practice, it won't matter if the confidence interval is 60 - 100 % or 65 - 100 %: both is typically not acceptable for application use)
Comment: With too few observations, permutation tests (and similarly bootstrapping) are inefficient. Few observations would be in terms of the number in each cell. For larger numbers per cell (traditionally people have had rules of thumb of >5 for the minimum amount in any cell), asymptotic frequentist methods should be okayish (but there's a considerable area where bootstrapping/exact permuation methods are possible, in which case why not use them?). However, Bayesian methods tend to perform pretty well for all settings (until you need to do MCMC with 10,000 or 100,000s of observations...).


How to get dropdown option label from POST request?

Question: I have this array that populates a dropdown menu. The key is my database key and the value is the corresponding database value. The issue I am having is that when I POST the form, I am getting the POSTed numerical value instead of the string value in the dropdown list. I could query the database to get the string but there has to be a better way of doing this as I already have both values in an array. I can't use array_search because the the array is multidimensional. Can someone please offer a hand?
Thanks
foreach($dd as $k=>$v)
{
    echo'<option value="'.$v['ace_id'].'">'.$v['arua'].'</option>';
}

Comment: You realize that the browser will post whatever is in the value= attribute of the selected ? Can you clarify what you're putting there versus what you're getting back?
Comment: Hey Dark. Yes, I know.. I am using the key for the html value that is posted. I need it for the database however, when the user posts and the validation fails, I need to show the user the string, not the db key.
Answer: The HTML input fields (this also includes select, textarea and button) only sends the name-value pairs to the server side (as been specified in the name and value attribues). They does not send any textual representation along it.
You already know the text and the labels in the server side beforehand (how else would you have printed them?), there's absolutely no point of having them in the request parameters.
You can solve this "problem" in several ways:

Just specify the desired value in the option value attribute, not as option label.
Add hidden input field(s) which passes the option label back.
Maintain a global array in the server side memory with those key-value pairs.
Go get the value from the DB.
Do nothing. In most cases you actually don't need them, just write smart code.

Update: here's a basic example (without sanitity checks like isset and htmlspecialchars, but that's up to you):
$selected = $_POST['dropdownname'];

foreach ($options as $value => $label) {
    echo '<option value="' . $value . '"' . ($value == $selected ? ' selected' : '') .  '>' . $label . '</option>';
}

Comment: Thanks for the help. I'd opt for #5 and that's what I've been trying to do. Drop downs have always been a bit of a challenge to get just right with a minimal amount of code when using the same form for inserts, updates and deletes. How would you go about doing this if it were you?
Comment: Set the option to selected if the array key matches the request parameter value.
Comment: Ok, that sounds super easy... Almost too easy. If you look at my array, the way it's structured, how would I test the array key? array_search wouldn't work with this type of array. Am I wrong?
Answer: If you want the text name in the dropdown instead of the key, then put that in the value="" part.  If you want both, then put both in there and seperate them with a comma and run split() on it.
Comment: This was what I had thought of originally doing and then using explode.. or split. Yes, this would work too. I figured that maybe there was a best practice for these dropdowns. I have always struggled with getting them just right and thought that I'd ask over here.


Convergence of $E[N/Q \mid Q>0]$ when $Q \sim B(N,p)$

Question: Let $Q$ be a binomial random variable with parameters $N$ and $p$.
We know, of course, that $N/Q \rightarrow_p 1/p$ when $N \rightarrow \infty$.
I was wondering whether we also have that $\mathbb{E}[N/Q \mid Q>0]\rightarrow 1/p$.
Comment: Well, $E[N/Q]=\infty$, since there is a nonzero chance that $Q=0$, meaning that $N/Q=\infty$. So this does not converge to $0$.
Comment: Oh, good point. What I wanted is to know whether $\mathbb{E}[N/Q | Q>0] \rightarrow 1/p$. I just edited the question. Thank you
Answer: Yes, assuming $p>0$. Here is one way to prove.
Assume $p>0$.  Let $\{Y_i\}_{i=1}^{\infty}$ be i.i.d. Bernoulli-$p$ variables. Define $Q_n = \sum_{i=1}^n Y_i$.  Then $Q_n \sim Binomial(n,p)$. We want to show $E[n/Q_n|Q_n>0]\rightarrow 1/p$.
Define
$$ X_n = \left\{\begin{array}{ll}
\left|\frac{n}{Q_n} - \frac{1}{p}\right| & \mbox{ if $Q_n>0$} \\
0 & \mbox{ else} \end{array}\right.$$
By the law of large numbers we know that
$$X_n\rightarrow 0 \quad \mbox{(with prob 1)}$$
It can be shown that there is a finite constant $M$ such that (see below)
$$ E[X_n^2] \leq M \quad \forall n \in \{1, 2, 3, ...\}$$
and so $X_n$ is uniformly integrable and thus
$$ \lim_{n\rightarrow\infty} E[X_n] = E[\lim_{n\rightarrow\infty} X_n] = 0 $$
Thus
\begin{align}
0 &= \lim_{n\rightarrow\infty} E[X_n] \\
&= \lim_{n\rightarrow\infty} E[X_n|Q_n>0]P[Q_n>0] \\
&= \lim_{n\rightarrow\infty} E[X_n|Q_n>0](1-(1-p)^n)
\end{align}
Since $1-(1-p)^n\rightarrow 1$, it follows that
$$ \lim_{n\rightarrow\infty} E[X_n|Q_n>0] = 0$$
and so
$$ \lim_{n\rightarrow\infty} E\left[\left| \frac{n}{Q_n} - \frac{1}{p}\right| | Q_n>0\right] = 0 $$
and so by Jensen's inequality for the convex function $|x|$ we have
$$ \lim_{n\rightarrow\infty}\left|E\left[\frac{n}{Q_n}|Q_n>0\right] - 1/p\right| = 0 $$
$\Box$

Footnote: Proof that $E[X_n^2]\leq M$ for all $n$ (for some finite constant $M$): Observe that
$$ X_n^2\leq \max\left\{n^2, \frac{1}{p^2}\right\}$$
So
$$
E[X_n^2] = \underbrace{E[X_n^2|Q_n>n/(2p)]}_{\leq 4p^2} P[Q_n>n/(2p)] + \underbrace{E[X_n^2|Q_n\leq n/(2p)]}_{\leq \max\{n^2, 1/p^2\}}P[Q_n\leq n/(2p)]$$
Thus
$$ E[X_n^2] \leq 4p^2 + \max\{n^2, 1/p^2\}P[Q_n\leq 1/(2p)]$$
By the Chernov-Hoeffding inequality:
\begin{align}
P[Q_n\leq n/(2p)] &\leq P[|Q_n-n/p|\geq n/(2p)]\\
&\leq 2\exp(-\frac{2(n/(2p))^2}{n})
\end{align}
and so $\max\{n^2, 1/p^2\}P[Q_n\leq 1/(2p)] \rightarrow 0$.  It follows that the sequence
$$\max\{n^2, 1/p^2\}P[Q_n\leq 1/(2p)], \quad n \in \{1, 2, 3, ...\}$$
is upper-bounded by some positive constant $c$ for all $n$, so $E[X_n^2]\leq 4p^2+c$ for all $n$.


Find last edited file in git repo

Question: Is there a way (possibly with git) to get the last edited file in the git repo?
Example: I have a bunch of uncommitted changes from Friday when I left at 4:59pm to go spend the weekend on the lake and don't remember exactly where I left off.
Answer: It sounds like you want to simply find the newest file in your project, which is not something that Git tracks directly.  As far as Git is concerned, you have some staged changes, and some unstaged changes, but it has no idea of recency.
Thankfully, the filesystem does.  And really this is simply a question of which file is the newest, but we might be able to improve on that answer using some knowledge of Git (and also making it cross-platform - the Mac OS X version of find does not have a -printf option).
If you have not yet staged your changes, then they must be newer than the Git index (which stores the staged file information), so we can find all files newer with:
find . -type f -newer .git/index

And we can query the file time using the stat command:
find . -type f -newer .git/index -exec stat -f '%c %N' {} \;

And finally we want to sort the output (to get the newest first) and limit it to a single file:
 find . -type f -newer .git/index -exec stat -f '%c %N' {} \; | sort -r | head -1

Here I've changed a few files in my repository:
% git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   unstaged_edit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    unstaged_add

no changes added to commit (use "git add" and/or "git commit -a")

And if I want to find the newest:
% find . -type f -newer .git/index -exec stat -f '%c %N' {} \; | sort -r | head -1
1466437626 ./staged_edit

Answer: I don't know of a way to find the last edited file, but if you changed your files and saved them, you should be able to see the current state of where you left off with git status.
By default, it will show you what has been changed but not added (working directory) and what has been added but not committed (staging area).

Find last edited file in git repo


How to determine php memory usage on apache?

Question: Sorry if I didn't phrase my question correctly.
I use the php function "memory_get_usage()" to see the amount memory allocated to PHP on a script I am running.
On server A I get around 2 MB. (CentOS 6, Apache 2.2.15)
The same script on server B I get around 1 MB. (Mac OSX, Apache 2.2.22)
Where should I check, or how should I fine tune my apache or php settings on server A to have similar memory footprint on server B?
For example, does the memory foot print affected by the loaded php extensions?
Comment: You can comment on an answer if you want more information or have questions. Or edit your answer if you need to clarify some part of it.
Comment: How did you go with this?
Answer: I guess different environments will cause a script to consume different amounts of memory. I'm fairly sure that the more extensions loaded the more memory PHP will consume, so using only the ones you need is advised. You should start by comparing the versions of PHP and which modules are installed on each server.
If you're looking to increate the amount of memory a script can consume:
http://php.net/manual/en/ini.core.php

memory_limit integer This sets the maximum amount of memory in bytes
  that a script is allowed to allocate. This helps prevent poorly
  written scripts for eating up all available memory on a server. Note
  that to have no memory limit, set this directive to -1.

http://www.hostucan.com/webmaster-tutorials/increase-php-memory-limit

How much memory do I need?
It depends, for wordpress core, 32MB is the default memory limit. For
  drupal6 core, the min memory limit is 16MB, 32MB is recommended. But
  if you install many plugins, especially the plugins deal with image,
  you probably need 128MB or more memory.

There's not too much you can do config wise if you have a script that has inefficient code, if you'd like to get more bang for your buck, you should look to see if you can make the code more efficient so that it consume less memory.