Proper permissions on FreeBSD with nginx and php for multiple users

Question: I am trying to set up a multi-user FreeBSD server using nginx.
Here are my steps so far:
In my /usr/local/etc/nginx/nginx.conf file I set user www www; so that nginx acts as the www user, part of the www group. I also set user = www and group = www in my /usr/local/etc/php-fpm.conf.
What I want to achieve is that I (as the administrator) can add users to my system and create a folder for them (and of course the according server entry in the nginx.conf) in /usr/local/www for them to use, a bit like a shared hosting environment (without any automatic setup).
After installing nginx and php I created a first test user anon, and created a folder for him /usr/local/www/anonsite.
I then performed chown anon:www anonsite to make him the owner, and set the group to www, permission of the folder then looked like this: drwxr-xr-x 3 anon www 4 Apr 11 22:00 anonsite .
Creating a info.php in this folder as anon and pointing a browser to it now works. I then tested downloading and extracting grav, however it will only show a blank page (because of my wrong permission setup, I assume). If I change php-fpm.conf user=anon, it works as intended, or alternatively using chmod -R g+w /usr/local/www/anonsite, after extracting the the downloaded grav folder, will make it work as well.
And this is where I am stuck currently and can't wrap my head around. Both of these "fixes" seem wrong or bad practice to me. If I compare my setup to a shared host provider I use, my webroot folder there only has drwxr-x--- 5 username apache 4096 Apr 2 05:00 username permissions, and after extracting a grav test setup it will work right away (is this because of the way Apache works, maybe?).
Could someone explain to me why that's the case and maybe walk me through the steps to properly set this up, or what I am doing wrong?
Is the approach I tried considered bad practice overall?
Answer: nginx will likely be able to read the files as needed withing being explicitly setting the file's group owner as the nginx user
and for php-fpm the nginx user can simply be set as the listener
php-fpm.d/php-fpm-user1.conf
[grav]
user = user1
group = use1

listen = /var/run/php-fpm-user1.sock

listen.owner = www
listen.group = www

pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

php.ini
cgi.fix_pathinfo=0
nginx.conf
user www;
worker_processes auto;
worker_rlimit_nofile 8192; # should be bigger than worker_connections
pid /run/nginx.pid;

events {
    use kqueue; # No epoll on FreeBSD
    worker_connections 8000;
    multi_accept on;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout 30; # longer values are better for each ssl client, but take up a worker connection longer
    types_hash_max_size 2048;
    server_tokens off;

    # maximum file upload size
    # update 'upload_max_filesize' & 'post_max_size' in /etc/php5/fpm/php.ini accordingly
    client_max_body_size 32m;
    # client_body_timeout 60s; # increase for very long file uploads

    # set default index file (can be overwritten for each site individually)
    index index.html;

    # load MIME types
    include mime.types; # get this file from https://github.com/h5bp/server-configs-nginx
    default_type application/octet-stream; # set default MIME type

    # logging
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    # turn on gzip compression
    gzip on;
    gzip_disable "msie6";
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 5;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types
        application/atom+xml
        application/javascript
        application/json
        application/ld+json
        application/manifest+json
        application/rss+xml
        application/vnd.geo+json
        application/vnd.ms-fontobject
        application/x-font-ttf
        application/x-web-app-manifest+json
        application/xhtml+xml
        application/xml
        font/opentype
        image/bmp
        image/svg+xml
        image/x-icon
        text/cache-manifest
        text/css
        text/plain
        text/vcard
        text/vnd.rim.location.xloc
        text/vtt
        text/x-component
        text/x-cross-domain-policy;

    # disable content type sniffing for more security
    add_header "X-Content-Type-Options" "nosniff";

    # force the latest IE version
    add_header "X-UA-Compatible" "IE=Edge";

    # enable anti-cross-site scripting filter built into IE 8+
    add_header "X-XSS-Protection" "1; mode=block";

}
server {
    #listen 80;
    index index.html index.php;

    ## Begin - Server Info
    root /home/user1/www/html;
    server_name localhost;
    ## End - Server Info

    ## Begin - Index
    # for subfolders, simply adjust:
    # `location /subfolder {`
    # and the rewrite to use `/subfolder/index.php`
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }
    ## End - Index

    ## Begin - Security
    # deny all direct access for these folders
    location ~* /(\.git|cache|bin|logs|backup|tests)/.*$ { return 403; }
    # deny running scripts inside core system folders
    location ~* /(system|vendor)/.*\.(txt|xml|md|html|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ { return 403; }
    # deny running scripts inside user folder
    location ~* /user/.*\.(txt|md|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ { return 403; }
    # deny access to specific files in the root folder
    location ~ /(LICENSE\.txt|composer\.lock|composer\.json|nginx\.conf|web\.config|htaccess\.txt|\.htaccess) { return 403; }
    ## End - Security

    ## Begin - PHP
    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php-fpm-user1.sock;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
    }
    ## End - PHP
}`


WiX Error Determining Previous Package Version

Question: Running into an issue with version numbers.
We used to use Installshield for our client installer.  Since switching to WiX, we've "fixed" a few things such as the version numbering scheme.  We used to use x.x.x-y, but WiX does not allow the "-" to be part of the version #.
When I try to update an older installed product using the "dashed" version #, the installation stops (before it even gets going) with:
[09A4:1908][2016-09-20T09:20:34]e000: Error 0x80070057: Failed to convert version: 5.3.0-7 to DWORD64 for ProductCode: {...}
[09A4:1908][2016-09-20T09:20:34]e151: Detect failed for package: Client_MSI, error: 0x80070057

The error appears to cause the engine to not trigger OnDetectMsiFeature, however, the engine does appear to know, somehow, about the features:
[09A4:1908][2016-09-20T09:20:34]i000: WixWPF: Enter Method: OnDetectPackageComplete
[09A4:1908][2016-09-20T09:20:34]i000: WixWPF: Leaving Method: OnDetectPackageComplete
[09A4:1908][2016-09-20T09:20:34]i101: Detected package: NetFx40Web, state: Present, cached: None
[09A4:1908][2016-09-20T09:20:34]i101: Detected package: vcredist2015x64, state: Present, cached: Complete
[09A4:1908][2016-09-20T09:20:34]i101: Detected package: vcredist2015x86, state: Present, cached: None
[09A4:1908][2016-09-20T09:20:34]i101: Detected package: Client_MSI, state: Unknown, cached: None
[09A4:1908][2016-09-20T09:20:34]i104: Detected package: Client_MSI, feature: FeatureA, state: Unknown
[09A4:1908][2016-09-20T09:20:34]i104: Detected package: Client_MSI, feature: FeatureB, state: Unknown
...
[09A4:1908][2016-09-20T09:20:34]i000: WixWPF: Enter Method: OnDetectComplete

UPDATE:
I tracked the problem with the version numbers to dutil\fileutil.cpp.  I changed the check on line 444 to:
while (wzPartEnd < wzEnd && ((L'.' != *wzPartEnd)
                         &&  (L'-' != *wzPartEnd)))

soas to accept the '-' as a version separator.  Now, the new bootstrapper loads, but is having problems with state detection.  While it "sees" the package and features, it "sees" all of them as "Absent".
[0280:1050][2016-09-20T12:36:05]i103: Detected related package: {...}, scope: PerMachine, version: 18.104.22.168, language: 0 operation: MajorUpgrade
[0280:1050][2016-09-20T12:36:05]i000: WixWPF: Enter Method: OnDetectRelatedMsiPackage
[0280:1050][2016-09-20T12:36:05]i000: WixWPF: Leaving Method: OnDetectRelatedMsiPackage
[0280:1050][2016-09-20T12:36:05]i000: WixWPF: Enter Method: OnDetectMsiFeature
[0280:1050][2016-09-20T12:40:42]i000: WixWPF: Leaving Method: OnDetectMsiFeature
...
[0280:1050][2016-09-20T12:41:04]i000: WixWPF: Enter Method: OnDetectPackageComplete
[0280:1050][2016-09-20T12:41:07]i000: WixWPF: Leaving Method: OnDetectPackageComplete
[0280:1050][2016-09-20T12:41:07]i101: Detected package: NetFx40Web, state: Present, cached: None
[0280:1050][2016-09-20T12:41:07]i101: Detected package: vcredist2015x64, state: Present, cached: Complete
[0280:1050][2016-09-20T12:41:07]i101: Detected package: vcredist2015x86, state: Present, cached: None
[0280:1050][2016-09-20T12:41:07]i101: Detected package: Client_MSI, state: Absent, cached: None
[0280:1050][2016-09-20T12:41:07]i104: Detected package: Client_MSI, feature: LunaClient, state: Absent
[0280:1050][2016-09-20T12:41:07]i104: Detected package: Client_MSI, feature: FeatureB, state: Absent

So, my new questions are:

Why is the old package and selected list of features all marked as Absent?
Can I recover from this?
Answer: Updating this a month after I got it working.  I found that I needed to hook another event, DetectRelatedMsiPackage.  I found this issue and associated code, which the poster was kind enough to share and which I was able to adapt to my needs, as follows:
public override void OnDetectRelatedMsiPackage(WPFBootstrapperEventArgs<Wix.DetectRelatedMsiPackageEventArgs> e)
{
    var existingPackageProductCode = e.Arguments.ProductCode;
    var existingPackageId = e.Arguments.PackageId;

    if (existingPackageId != HSMClientPackageID)
    {
        return;
    }

    InstallData.RelatedOperation = e.Arguments.Operation;
    if (InstallData.RelatedOperation == Wix.RelatedOperation.MajorUpgrade)
    {
        //requires reference to WiX Toolset\SDK\Microsoft.Deployment.WindowsInstaller.dll
        var installedPackage = new ProductInstallation(existingPackageProductCode);
        if (!installedPackage.IsInstalled)
        {
            //Log(string.Format("Migrating Package {0}, which is not installed, so marking it and it's features as Absent", existingPackageId));
            _packageStates[existingPackageId] = Wix.PackageState.Absent;
        }
        else
        {
            //Log(string.Format("Migrating features for MajorUpgrade of Package {0}", existingPackageId));

            _packageStates[existingPackageId] = Wix.PackageState.Present;

            foreach (var currentInstallFeature in installedPackage.Features)
            {
                switch (currentInstallFeature.State)
                {
                    case InstallState.Local:
                        //Log(string.Format("Migrating feature {1} of Package {0} - marking as Present", existingPackageId, currentInstallFeature.FeatureName));
                        _featureStates[currentInstallFeature.FeatureName] = Wix.FeatureState.Local;
                        break;
                    case InstallState.Absent:
                        //Log(string.Format("Migrating feature {1} of Package {0} - marking as Absent", existingPackageId, currentInstallFeature.FeatureName));
                        _featureStates[currentInstallFeature.FeatureName] = Wix.FeatureState.Absent;
                        break;
                    default:
                        //Log(string.Format("Migrating feature {1} of Package {0} - marking as Unknown", existingPackageId, currentInstallFeature.FeatureName));
                        _featureStates[currentInstallFeature.FeatureName] = Wix.FeatureState.Unknown;
                        break;
                }
            }
        }
    }
}

Comment: Yes, I do believe that's it!


Periodic orbits enclosing fixed points in a differential equation

Question: Suppose we have a differential equation $\mathbf{\dot{x}}=\mathbf{f(x)}$ for $\mathbf{x}\in \Bbb{R}^2$.
Suppose further that there are three fixed points, of which one is a saddle and two are sinks. I am not sure how to determine examples of the following scenarios or to prove they do not exist:

There exists a periodic orbit enclosing precisely one sink.
There exists a periodic orbit enclosing all three fixed points.

The index test does not rule out either of these possibilities and I am unsure how to construct examples demonstrating existence.
Any help would be much appreciated!
Comment: Why not pin three points in the plane and draw by hand a vector field with those features. That will be a constructive example. You can then try to guess an algebraic expression for the vector field.
Answer: As you do not have any specific question or attempt to build up upon, here is some general strategy:
The most important insight for this kind of exercise is that you can do almost everything you want once you leave the direct vicinity of the attractors. You can just glue together piecewise dynamical systems, modify the inside of a limit cycle, etc. In this case, you have to take care of not introducing additional fixed points, but that's it.
As a consequence, whenever you can draw a reasonably complete phase-space diagram of such a system, an answer exists. It may be tedious to construct actual ODEs that fulfil it, but depending on whom you need to convince, you do not even need to jump through that hoop. Take care that all the homo- and heteroclinic orbits end somewhere (and be it infinity) and the continuity of the phase-space flow becomes clear (e.g., no adjacent trajectories running in opposite direction).
With that being said, some specific hints:

Note how the Problem 1 can be reduced to finding of finding a periodic orbit enclosing one sink. The presence of the other two fixed points somewhere in phase does not affect it.

Note that when you found a solution to Problem 1, you can solve Problem 2 by finding a configuration that behaves like a sink to the outside (all trajectories ingoing), but has the three fixed points in question on the inside.

Look at some bifurcations. Is there a bifurcation which features a sink within a limit cycle? Is there a bifurcation which turns a sink into two sinks and one saddle?


Page with Arguments

Question: I have a basic page  named as news.
Now i have created a two blocks  from view in page news
Block 1  is block of taxonomy
Block 2 is having Contextual Filter with term id
I have assigned these block to news page
as
news
news/*
Now when i am accessing
example.com/news
only Block 1 is coming which is fine
when i am accessing
example.com/news/1
Block1 is coming and
an error
Requested page could not found
I want block 2
How to do this
Please
Thanks
Comment: Check your settings for the contextual filter, specifically the validation settings. Are you sure you have a taxonomy term with id 1? Check the taxonomy term override view for reference.
Comment: yes, still not working
Answer: Block views don't automatically pull in contextual filter values from the URL.
So to make it work you need to configure the "When the filter value is NOT available" setting to be "Provide default value".
Then you get some more options appearing below, where in the "Type" field you can select "Taxonomy term ID from URL".
Then configure the other settings as required.
[EDIT]
Note that term ID from URL only works from either a node page (node/NID) or a taxonomy term page (taxonomy/term/TID) or an aliased path for one of the pages.
If you want o do it with some other random page use the "Raw value from URL" option for the "When the filter value is NOT available" setting, then select the position in the URL that your parameter is (in your case 2).
Comment: i have done this, i have attached screenshot. still not working
Comment: Your screenshot says you are using the "Load default filter from term page" option. Is your news page a taxonomy term listing page (is it an alias of /taxonomy/term/TID)?
Comment: Edited my answer with info about "Raw value from URL" option.

Page with Arguments


Objective-C read Hex code and output Emoji

Question: I have emoji codes (i.e. Unicode Hex codes) such as U0001f618. I want to be able to enter the code and return the emoji. So far this works:
NSLog (@"\U0001f618");

This returns the emoji, however if I store the code in a file and import it into an NSString and call;
NSLog (unicodeForEmoji);

The code is returned i.e.: \U0001f618, any ideas?
Thanks
D
Comment: You need to scan the text and replace appropriate escape sequences with the equivalent character.
Comment: In addition to the other issues, understand that NSLog often does not render non-ASCII characters correctly.
Answer: The bytes are  unicode encoding:NSUTF32BigEndianStringEncoding.
To read directly from a file use:
NSString *unicodeString = [[NSString alloc] initWithContentsOfFile:path encoding:NSUTF32BigEndianStringEncoding error:&error];

Test:
// The next two lines substiture to reading the data from the file:
unsigned char unicode32Chars[4] = { 0x00, 0x01, 0xf6, 0x18 };
NSData *unicode32Data = [NSData dataWithBytes:unicode32Chars length:4];
NSLog (@"unicode32Data: '%@'", unicode32Data);

NSString *unicode32String = [[NSString alloc] initWithData:unicode32Data encoding:NSUTF32BigEndianStringEncoding];
NSLog (@"unicode32String: '%@'", unicode32String);

Output:
unicodeData: '<0001f618>'
unicodeString: ''

I suspect you might really want to use UTF-8 encoding but that is a different byte string:
unsigned char unicode8Chars[4] = { 0xf0, 0x9f, 0x98, 0x98 };
NSData *unicode8Data = [NSData dataWithBytes:unicode8Chars length:4];
NSLog (@"unicode8Data: '%@'", unicode8Data);

NSString *unicode8String = [[NSString alloc] initWithData:unicode8Data encoding:NSUTF8StringEncoding];
NSLog (@"unicode8String: '%@'", unicode8String);

Output:
unicode8Data: '<f09f9898>'
unicode8String: ''


Where is the Dynamic Language Runtime?

Question: Alright, I have searched everywhere and read several articles on the subject, including MSDN, but I do not quite understand where these assemblies are located.  I know that some are part of the .NET Framework, but I can't find the rest of them, like Microsoft.Scripting, etc.  Could someone elaborate on where all the assemblies are located and give me the namespace in which the DLR types are found.  Do I need to download the project from Codeplex?  If so, what parts of it do I need?
Answer:
Yes you will need to download it from CodePlex
Consult their documentation
Depends on what exactly you will be using it for (see 2)
Comment: Well, I am designing a grammar for Perl, so I am creating a compiler to generate IL for Perl scripts.  What parts of the opensource DLR are not included in .NET framework?  I know some of it is included in System.Dynamic and System.Core but the rest has evaded me.  Thanks for clearing that for me.

Where is the Dynamic Language Runtime?


Google Apps Script: Second e.parameter Not Functioning as Expected

Question: Hello trust you are doing well
My app send a data to a google sheet and it append the data followed by and ID and a timestamp.
I use a key as parameter where if the hkey 1 is inputted the data will go in a one sheet and if the hkey is 2 it will go in another sheet.
for the hkey 1 it works but for the hkey 2 I got errors.
FYI : When appending data, for either hkey 1 or 2, it will append data to 2 sheets for redundancy.
Here is my code :

function doGet(e) {
  var ss = SpreadsheetApp.getActive();
  var StocksSheet = ss.getSheetByName("Stocks");
  var OrdersSheets = ss.getSheetByName("Orders");

  var scriptProperties = PropertiesService.getScriptProperties();
  var counter = Number(scriptProperties.getProperty('counter')) || 0;
  //var counter = scriptProperties.getProperty('counter') || 0;
  var formatDate = Utilities.formatDate(new Date(), "GMT+4", "ddMM");
  var zero = (counter < 365) ? '0' : '';
  var ID = "FS" + formatDate + zero + counter++;
  var IDP = "FP" + formatDate + zero + counter++;
  if (counter > 99) counter = 0;
  scriptProperties.setProperty('counter', counter);

  var TimeStamp = Utilities.formatDate(new Date(), "GMT+4", "dd-MM-yy HH:mm:ss");
  var LinkS = "https://drive.google.com/drive/xxxxxxxxxxxxxxxx";
  var LinkP = "https://drive.google.com/drive/u/xxxxxxxxxxxxxxxx";
  const { Region, Place, Staff, Shop_Representative, Product_Location, Date_Checked, Time_Checked, Remarks, Hkey, RemarksP, Date_Ordered, Time_Ordered, Date_of_Delivery } = e.parameter;
  const rowS = [TimeStamp, ID, Region, Place, Staff, Shop_Representative, Product_Location, Date_Checked, Time_Checked, Remarks, LinkS];
  const rowP = [TimeStamp, IDP, Region, Place, Staff, Shop_Representative, Date_Ordered, Time_Ordered, Date_of_Delivery, LinkP, RemarksP];

  if (Hkey == '123') {
    const res = [...Array(20)].reduce((ar, _, h) => {
      var i = h + 1;
      var Product = e.parameter['Product' + i];
      if (Product) {
        ar.push([...rowS, Product, ...['Filling', 'Quantity', 'Expiry'].map(f => e.parameter[`${f}${i}`] || "")]);
      }
      return ar;
    }, []);

    if (res.length == 0) {
      return ContentService.createTextOutput("Stock not uploaded, please retry");
    }

    try {

      StocksSheet.getRange(StocksSheet.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      var StocksLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Stocks_Logs");
      StocksLogs.getRange(StocksLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      return ContentService.createTextOutput("Your stock had been inputed with ID " + ID);

    } catch (error) {

      var StocksLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Stocks_Logs");
      StocksLogs.getRange(StocksLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
        return ContentService.createTextOutput("Database issue detected for stock with ID:" +ID+  ". Please coordinate with the administrator for resolution.");
    }
  }

  if(Hkey == '456') {
    const res = [...Array(12)].reduce((ar, _, h) => {
      var i = h + 1;
      var Product = e.parameter['Product' + i];
      if (Product) {
        ar.push([...rowP, Product, ...['Filling', 'Quantity', 'Discount'].map(f => e.parameter[`${f}${i}`] || "")]);
      }
      return ar;
    }, []);

    if (res.length == 0) {
      return ContentService.createTextOutput("Order not uploaded, please retry");
    }

    try {

      OrdersSheets.getRange(OrdersSheets.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      var OrdersLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Orders_Logs");
      OrdersLogs.getRange(OrdersLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      return ContentService.createTextOutput("Your order had been inputed with ID " + IDP);

      } catch (error) {

      var OrdersLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Orders_Logs");
      OrdersLogs.getRange(OrdersLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      return ContentService.createTextOutput("Database issue detected for order with ID:" +IDP+  ". Please coordinate with the administrator for resolution.");
    }
  } else {
    return ContentService.createTextOutput("Please check check your app version")
  }
}

function doGet(e) {
  var ss = SpreadsheetApp.getActive();
  var StocksSheet = ss.getSheetByName("Stocks");
  var OrdersSheets = ss.getSheetByName("Orders");

  var scriptProperties = PropertiesService.getScriptProperties();
  var counter = Number(scriptProperties.getProperty('counter')) || 0;
  //var counter = scriptProperties.getProperty('counter') || 0;
  var formatDate = Utilities.formatDate(new Date(), "GMT+4", "ddMM");
  var zero = (counter < 365) ? '0' : '';
  var ID = "FS" + formatDate + zero + counter++;
  var IDP = "FP" + formatDate + zero + counter++;
  if (counter > 99) counter = 0;
  scriptProperties.setProperty('counter', counter);

  var TimeStamp = Utilities.formatDate(new Date(), "GMT+4", "dd-MM-yy HH:mm:ss");
  var LinkS = "https://drive.google.com/drive/xxxxxxxxxxxxxxxx";
  var LinkP = "https://drive.google.com/drive/u/xxxxxxxxxxxxxxxx";
  const { Region, Place, Staff, Shop_Representative, Product_Location, Date_Checked, Time_Checked, Remarks, Hkey, RemarksP, Date_Ordered, Time_Ordered, Date_of_Delivery } = e.parameter;
  const rowS = [TimeStamp, ID, Region, Place, Staff, Shop_Representative, Product_Location, Date_Checked, Time_Checked, Remarks, LinkS];
  const rowP = [TimeStamp, IDP, Region, Place, Staff, Shop_Representative, Date_Ordered, Time_Ordered, Date_of_Delivery, LinkP, RemarksP];

This part below works fine
Reference to question asked : Reduce redundant code by optimizing of appendRow() App script
  if (Hkey == '123') {
    const res = [...Array(20)].reduce((ar, _, h) => {
      var i = h + 1;
      var Product = e.parameter['Product' + i];
      if (Product) {
        ar.push([...rowS, Product, ...['Filling', 'Quantity', 'Expiry'].map(f => e.parameter[`${f}${i}`] || "")]);
      }
      return ar;
    }, []);

    if (res.length == 0) {
      return ContentService.createTextOutput("Stock not uploaded, please retry");
    }

    try {

      StocksSheet.getRange(StocksSheet.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      var StocksLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Stocks_Logs");
      StocksLogs.getRange(StocksLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      return ContentService.createTextOutput("Your stock had been inputed with ID " + ID);

    } catch (error) {

      var StocksLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Stocks_Logs");
      StocksLogs.getRange(StocksLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
        return ContentService.createTextOutput("Database issue detected for stock with ID:" +ID+  ". Please coordinate with the administrator for resolution.");
    }
  }

This part did not work, it did not append data to the google sheets
  if(Hkey == '456') {
    const res = [...Array(12)].reduce((ar, _, h) => {
      var i = h + 1;
      var Product = e.parameter['Product' + i];
      if (Product) {
        ar.push([...rowP, Product, ...['Filling', 'Quantity', 'Discount'].map(f => e.parameter[`${f}${i}`] || "")]);
      }
      return ar;
    }, []);

    if (res.length == 0) {
      return ContentService.createTextOutput("Order not uploaded, please retry");
    }

    try {

      OrdersSheets.getRange(OrdersSheets.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      var OrdersLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Orders_Logs");
      OrdersLogs.getRange(OrdersLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      return ContentService.createTextOutput("Your order had been inputed with ID " + IDP);

      } catch (error) {

      var OrdersLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Orders_Logs");
      OrdersLogs.getRange(OrdersLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      return ContentService.createTextOutput("Database issue detected for order with ID:" +IDP+  ". Please coordinate with the administrator for resolution.");
    }
  }

This part is for error with Hkey
else {
    return ContentService.createTextOutput("Please check check your app version")
  }
}

Comment: I have to apologize for my poor English skill. Unfortunately, I cannot understand your question. In order to correctly understand your question, can you provide the sample input and output situations you expect? And, can you provide whole script? First, I would like to correctly understand your question.
Comment: Hello @Tanaike, the code is base on the previously asked question you helped me

https://stackoverflow.com/questions/76843196/reduce-redundant-code-by-optimizing-of-appendrow-app-script/76844031?noredirect=1#comment135477473_76844031

I just added another parameter to append data to 2 other sheets
eg. if (Hkey == '123'), it will append data to sheets StocksSheet and StocksLogs
where if (Hkey == '456'), it will append data to sheets OrdersSheets and OrdersLogs.

The issue is that only (Hkey == '123') append data but with (Hkey == '456') I got errors and i did not append
Comment: Thank you for replying. I have to apologize for my poor English skill. Unfortunately, I cannot still understand your question. Can you provide whole script? First, I would like to correctly understand your question.
Comment: @Tanaike, I edited the code to clarify the part with issues as  it is the complete code.
Comment: Thank you for replying and updating your question. Your showing script is your whole script. Is my understanding correct?
Comment: Yes it is the whole script
Comment: Thank you for replying. From `Yes it is the whole script`, I would like to try to think of your question. When I find a solution, I would like to post it as an answer.
Comment: ```When I find a solution, I would like to post it as an answer.``` Of course, no issue, thanks in advance.
Comment: Hello, do you have an error returned by Apps Script (if so, please write it in the post) or is it just that it does not behave as expected ?
Comment: Hello @ValLeNain, now  I on got the return only  and the data is not appended

```
else {
    return ContentService.createTextOutput("Please check check your app version")
  }
}
```
Comment: are you sure that `Hkey` has the value you expect ? I don't see in the code where it gets its value from... is it a constant outside the functions ?
Comment: Yes the ```Hkey``` has the expected value.
It is a parameter sent from my app.
Comment: Have yu tried testing it with UrlFetchApp?  Do you have some test data for something like that?  If you could provide that it would make your example reproducible and make it possible for us to test.
Comment: I tested the code with data that I made up and it works for me.
Answer: Cannot confirm problem with code. I'd guess the problem is probably in your data
I modified the code as follows:
This is the test code for sh2
function testDG() {
  var e = { parameter: { A: 1, B: 2, C: 3, D: 4, E: 5, F: 6, G: 7, H: 8, I: "456", J: 9, K: 10, L: 11, M: 12, X: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], Y: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], Z: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], Product1: "one", Product2: "two", Product3: "three", Product4: "four", Product5: "five", Product6: "six", Product7: "seven", Product8: "eight", Product9: "nine" }};
  doGet(e);
}

function doGet(e) {
  Logger.log(JSON.stringify(e));
  var ss = SpreadsheetApp.getActive();
  var sh1 = ss.getSheetByName("Sheet0");
  var sh2 = ss.getSheetByName("Sheet1");
  var scrprops = PropertiesService.getScriptProperties();
  var counter = Number(scrprops.getProperty('counter')) || 0;
  var formatDate = Utilities.formatDate(new Date(), "GMT+4", "ddMM");
  var zero = (counter < 365) ? '0' : '';
  var ID = "FS" + formatDate + zero + counter++;
  var IDP = "FP" + formatDate + zero + counter++;
  if (counter > 99) counter = 0;
  scrprops.setProperty('counter', counter);

  var ts = Utilities.formatDate(new Date(), Session.getScriptTimeZone(), "dd-MM-yy HH:mm:ss");
  var Link1 = "https://drive.google.com/drive/xxxxxxxxxxxxxxxx";
  var Link2 = "https://drive.google.com/drive/u/xxxxxxxxxxxxxxxx";
  const { A, B, C, D, E, F, G, H, I, J, K, L, M } = e.parameter;
  const rowS = [ts, ID, A, B, C, D, E, F, G, H, Link1];
  const rowP = [ts, IDP, A, B, C, D, K, L, M, Link2, J];

  if (I == '123') {
    const res = [...Array(9)].reduce((ar, _, h) => {
      var i = h + 1;
      var Product = e.parameter['Product' + i];
      if (Product) {
        ar.push([...rowS, Product, ...['X', 'Y', 'Z'].map(f => e.parameter[`${f}${i}`] || "")]);
      }
      return ar;
    }, []);

    if (res.length == 0) {
      return ContentService.createTextOutput("Stock not uploaded, please retry");
    }

    try {

      sh1.getRange(sh1.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      //var StocksLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Stocks_Logs");
      //StocksLogs.getRange(StocksLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      //return ContentService.createTextOutput("Your stock had been inputed with ID " + ID);

    } catch (error) {

      //var StocksLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Stocks_Logs");
      //StocksLogs.getRange(StocksLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      //return ContentService.createTextOutput("Database issue detected for stock with ID:" + ID + ". Please coordinate with the administrator for resolution.");
    }
  }

  if (I == '456') {
    const res = [...Array(9)].reduce((ar, _, h) => {
      var i = h + 1;
      var Product = e.parameter['Product' + i];
      if (Product) {
        ar.push([...rowP, Product, ...['X', 'Y', 'Z'].map(f => e.parameter[`${f}${i}`] || "")]);
      }
      return ar;
    }, []);

    if (res.length == 0) {
      return ContentService.createTextOutput("Order not uploaded, please retry");
    }

    try {

      sh2.getRange(sh2.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      var OrdersLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Orders_Logs");
      //OrdersLogs.getRange(OrdersLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      //return ContentService.createTextOutput("Your order had been inputed with ID " + IDP);

    } catch (error) {

      //var OrdersLogs = SpreadsheetApp.openById('xxxxxxxxxxxxxxxx').getSheetByName("Orders_Logs");
      //OrdersLogs.getRange(OrdersLogs.getLastRow() + 1, 1, res.length, res[0].length).setValues(res);
      //return ContentService.createTextOutput("Database issue detected for order with ID:" + IDP + ". Please /coordinate with the administrator for resolution.");
    }
  } else {
    //return ContentService.createTextOutput("Please check check your app version")
  }
}

The code is able to write to both sheets based upon the value in I


What does it mean for a metric space to be isometrically embedded in another space?

Question: I understand the definition of an isometric embedding, (an injective, distance preserving map) but I don't understand what it means for a metric space to be isometrically embedded in another space. Does it mean that all maps from the first space to the other need to be an isometric embedding? Does just one? I tried searching online but I couldn't find any examples.
Comment: Just one. Like how $\mathbb{Q}$ is embedded in $\mathbb{R}$. It doesn't make sense for the condition to be on all maps, because you can define maps arbitrarily.
Comment: Distance preserving map is always injective for metric spaces. There is a need for mentioning a distance preserving map to be injective in cases of pseudo-metric spaces.
Answer: Let $(X,d)$ & $(Y,d_1)$ be a metric spaces. Then $X$ is isometrically embedded to another space means $\exists f:X \to Y$ s.t $d(x,y)=d_1(f(x),f(y)) \forall x,y\in X$
Answer: Let $(S,d)$ and $(T,e)$ be two metric spaces. A function $f$ from $S$ to $T$ is called and isometry iff $e(fx,fy)=d(x,y)$ for all $x,y\in S$. So any isometry $f$ from $S$ onto a subset of $T$ is what is called and embedded into another space. Since $f$ preserves the only given structure on $S$, the metric, we can consider $S$ as a subset of $T$
NB: For any metric space $(S,d)$  there is a complete metric space $(T,e)$ and an isometry $f$ from $S$ onto a subset of $T$, as in the case of the rational numbers, and $T$ is called the completion of $S$. So in this case, $S$ is isometrically embedding in  $T$.


Angular - mat-grid-list not displaying children passed by

Question: I'm using angular material design components and want to create a custom grid-list component. The component would adapt the number of grid-list-columns based on its size. The components template would look like this:
<mat-grid-list [cols]="adaptiveCols">
    <ng-content></ng-content>
</mat-grid-list>

And the component will be used like this:
<my-grid>
    <mat-grid-tile>tile_content</mat-grid-tile>
    <mat-grid-tile>tile_content</mat-grid-tile>
    <mat-grid-tile>tile_content</mat-grid-tile>
</my-grid>

So you would think this would display a <mat-grid-list> with 3 <mat-grid-tile>s in them, and the resulting DOM will contain those elements. But you don't see the tiles nor their content.
Adding more tiles to the grid-list and using a fixed rowHeight doesn't affect the height-attribute of the grid-list (from what I could figure out the grid-list calculates its height based on the number and col-/rowspan of tiles). That makes you think the grid-list doesn't "see" the <mat-grid-tile>-children. Looking at the source code of the grid-list and setting a breakpoint on _layoutTiles() confirmed this, in this method this._tiles is empty.
My guess is the angular ContentChildren-annotation (here) doesn't find the tile-children in the first code snippet because it only works on the first level of DOM, and that first level we only have the <ng-content>.
The desired result is clear, I want to be able to use a custom grid-list component, give it some children and use its features. But for some reason, angular doesn't want me to succeed, or I still have the stuff to learn.
I'm using angular 6.1 and installed the material-components using the official docs.
Answer: mat-grid-list uses content projection in it's own implementation and so it needs to be able to query it's projected content.  Trying to add another level of content projection makes this not possible since the mat-grid-tile's are not actually being projected by the the list, but by your component.
So short answer: this isn't gona work.
There is likely a way to achieve your goal but content projection isn't the way.  If you clarify your intentions a bit, I may be able to help.  You likley need to use templates though.
Here's the simple case: write a wrapper around mat-grid-list that accepts an array of data as input that you can iterate over with ngFor to create the tiles
@Component({
  selector: 'my-grid',
  templateUrl: './my-grid.html'
})
export class MyGrid {
  @Input() data: string[];
}

<mat-grid-list>
  <mat-grid-tile *ngFor="let d of data">{{d}}</mat-grid-tile>
</mat-grid-list>

This provides a reusable wrapper around mat-grid-list that abstracts a bit of it away for you.  however, this is no good if your grid content is more complex than a string (likely) or requires custom templates on every usage.  In this case, you need a somewhat more refined approach, using templates and directives:
@Directive({
  selector: 'gridTemplate'
})
export class GridTemplate {
   @Input() key: string;
   constructor(private elementRef: ElementRef, public template: TemplateRef<any>) {}
}

This directive will identify your custom templates then, in your grid list, you can use these templates to populate the tiles
@Component({
  selector: 'my-grid',
  templateUrl: './my-grid.html'
})
export class MyGrid implements AfterContentInit {
  //structure the data so it can be assigned to a template by key
  @Input() data: {templateKey:string, data: any}[];

  //use content children to find projected templates
  @ContentChildren(GridTemplate)
  gridTemplates: QueryList<GridTemplate>;

  gridTemplateMap: {[key:string]: TemplateRef<any>} = {};

  ngAfterContentInit() {
    //store queried templates in map for use in template
    this.gridTemplates.forEach(t => this.gridTemplateMap[t.key] = t.template);
  }
}

<ng-template #defaultGridTemp let-d="data">{{d}}</ng-template>
<mat-grid-list>
  <mat-grid-tile *ngFor="let d of data">

    <ng-container [ngTemplateOutlet]="(gridTemplateMap[d.templateKey] || defaultGridTemp)"
                  [ngTemplateOutletContext]="{ data: d.data }" ></ng-container>
  </mat-grid-tile>
</mat-grid-list>

Then your usage of your component is like this:
@Component({...})
export class MyComponent {
   data = [
    { templateKey:'MyKey1', data: {message: 'Projected Message', number: 1} },
    { templateKey:'MyKey2', data: {whatever: 'Message', youWant: 'to place'} }
   ];
}

<my-grid [data]="data">
  <ng-template gridTemplate key="MyKey1" let-d="data">

{{d.message}}

{{d.number}}

Any other arbitrary content

  </ng-template>
  <ng-template gridTemplate key="MyKey2" let-d="data">

{{d.whatever}}

{{d.youWant}}

Any other arbitrary content

  </ng-template>
</my-grid>

with this approach, you can add whatever reusable logic to mat-grid-list you want inside your grid component but still maintain the flexibility of providing custom templates to your grid.  The drawback is that you have to take the extra steps to structure your data to take advantage of this approach.
Comment: I wanted to create a component that can be used at multiple places so i don't need to write code for feature X in multiple components that use mat-grid-list. I'm sure that isn't possible without content-projection, maybe with a subclass for the component to inherit the logic from.
Comment: you can write your own wrapper around mat-grid-list that accepts an array of objects as input and then use ngFor to generate the needed mat-grid-tiles using that array.  If you need special / unique templates in each usage, then you can use template projection to accomplish this


What is the quantifier complexity of the function sending an ordinal to the power set iterated that many times?

Question: One use of the Axiom Schema of Replacement in ZFC is that it allows one to prove that for a set $X$ and a limit ordinal $\alpha$, the transfinitely iterated powerset $P^\alpha(X)$ exists (for a successor ordinal $\beta+1$, one only needs $P^\beta(X)$ to exist beforehand). Sometimes people consider restricted versions of Replacement with restricted complexity on the description of the function to which it is being applied. I was wondering what the quantifier complexity of the statement that Replacement is then applied to in order to get $P^\alpha(X)$. This would then put a floor on how low complexity Replacement instances can be before iterated powerset no longer becomes available.
Answer: It looks $\Sigma_2$ to me: we have $y=\mathcal{P}^\alpha(x)$ iff there is a certain sequence of length $\alpha+1$ with first term $x$, last term $y$ and satisfying a certain $\Pi_1$ property (namely that limit terms are the unions of prior terms and successor terms are the powersets of their predecessor terms). So $\mathsf{KP+\Sigma_2Rep+Pow}$ proves the existence of ordinal iterates of the powerset operation.
This isn't as fine-grained as one might hope, though - $\Sigma_2$-$\mathsf{Rep}$ lets us do a lot more than just iterate the particular $\Pi_1$ powerset function, and so seems like overkill. If memory serves, a theory of intermediate strength which shows up a lot here is $\mathsf{KP^{Pow}}$ (maybe $\mathsf{KP^\mathcal{P}}$?). This is the analogue of $\mathsf{KP}$ in the language $\{\in,\mathcal{P}\}$: basically, the powerset operation is taken as primitive here, and we are given $\Delta_1$ separation and $\Sigma_1$ replacement for formulas involving it. This is intermediate between $\mathsf{KP}$ and $\mathsf{KP+\Sigma_2Rep+Pow}$ and, I believe, incomparable with $\mathsf{KP+\Sigma_2Rep}$.
Comment: Thanks for the additional comments, that's interesting.
Comment: @theHigherGeometer Incidentally, Adrian Mathias has a number of papers on systems in this general vicinity - see [here](https://www.dpmms.cam.ac.uk/~ardm/scdlistB.html).
Comment: Thanks Noah, but I'm more interested in the context of ZFC.
Comment: @theHigherGeometer I'm not sure what you mean. Mathias' papers are on subsystems of $\mathsf{ZFC}$, just like $\mathsf{KP}+\Sigma_2\mathsf{Rep}$ or $\mathsf{KP^{Pow}}$ above. Aren't you asking about subsystems of $\mathsf{ZFC}$? (Note that Mathias identifies set theories of MacLane and Bourbaki with corresponding $\mathsf{ZFC}$-fragments.)
Comment: Oh, I'm sorry, I wasn't thinking in terms of $KP \subset ZFC$. In that case, I accept the pointer much more humbly (and I confess I didn't click through, since I think I had looked at some of that work a few months back, as assumed I remembered it accurately)


Difference between calculated inclusion probability and what is returned by sampling function?

Question: I have a (small) population from which I wish to sample. I assign probabilities proportional to $y$. I enumerate the possible samples and then determine the probability of each sample occurring based on the product of the probabilities for each $y_i$ in the sample. I add up the probabilities for the samples that contain the $y_1$ and I believe (incorrectly?) that under the assumption of independence (i.e. with replacement sampling) this gives me the inclusion probability for $y_1$. I look at the inclusion probabilities returned by the inclusionprobabilities function in the sampling package and I get a different answer. I do not understand why, is someone able to explain?
library(survey)
library(sampling)
library(gtools)

set.seed(123)
y <- c(1190,26751,68570,34536)
p <- y/sum(y)

df <- data.frame(permutations(n=length(y), r=2, v=1:length(y), repeats.allowed = T))
df$p <- p[df$X1] * p[df$X2]; df

# X1 and X2 denote the index of the y value that is included
# in the sample.

    X1 X2  p
1   1  1 0.00008245932
2   1  2 0.00185367169
3   1  3 0.00475145854
4   1  4 0.00239312195
5   2  1 0.00185367169
6   2  2 0.04167022794
7   2  3 0.10681198947
8   2  4 0.05379697926
9   3  1 0.00475145854
10  3  2 0.10681198947
11  3  3 0.27378782541
12  3  4 0.13789611111
13  4  1 0.00239312195
14  4  2 0.05379697926
15  4  3 0.13789611111
16  4  4 0.06945282329

samplesSet <- data.frame(df[1 == df$X1  | 1 == df$X2, ])
sum(samplesSet$p)

pik <- inclusionprobabilities(y, 2)
data.frame(pik=pik,name=1:length(y))

Update:
Thanks both @whuber and @StasK. It is clear that the inclusion probabilities reflect sampling without replacement. However, I am uncertain what the inclusion probabilities returned by inclusionprobabilities are. They seem to be calculated as:
$$
n \frac{y_i}{\sum_{i=1}^{N} y_i}
$$
and have an adjustment to ensure that no probability is greater than 1 and also that the sum of the probabilities corresponds to the sample size.
If I assume that my population is $y=\{1,2,3\}$ such that the probabilities of selection are $\frac{1}{6}$, $\frac{2}{6}$ and $\frac{3}{6}$ and then I take a sample of 2, I calculate the inclusion probabilities to be $\frac{5}{12}$, $\frac{11}{15}$ and $\frac{17}{20}$ respectively. Clearly, these are not what is returned by inclusionprobabilities and so my question now is have I calculated the inclusion probabilities incorrectly or is the inclusionprobabilities  function returning something that represents the inclusion probabilities but isn't actually the inclusion probabilities?
myn <- 2
a <- c(1,2,3)
p <- myn * a/sum(a); p
[1] 0.3333333 0.6666667 1.0000000

inclusionprobabilities(a, myn)
[1] 0.3333333 0.6666667 1.0000000

Thanks.
Comment: The help is indeed abysmal. Protect yourself by testing this function on simple arguments with known answers. For instance, `inclusionprobabilities(1:2,2)` returns the vector `1 1`. What does that tell you about the assumed form of sampling?  Could this possibly reflect sampling *with replacement*?  (Such prophylactic testing is essential when learning to use any package--the ultimate arbiter of questions like this is *what the computer does,* not what the help pages seem to say!)
Comment: Sampling with unequal probabilities is really weird, and it does not always give you the answer you expect, although that mostly has to do for sampling with replacement. Selection probabilities, and especially the pairwise selection probabilities, depend on the particular sampling algorithm, see Hanif & Brewer (1983) -- nearly impossible to find -- and [Tille 2006](http://www.amazon.com/Sampling-Algorithms-Springer-Series-Statistics/dp/0387308148). If `inclusionprobabilities()` indeed refer to sampling WOR, you need to filter on `X1!=X2`.
Comment: Thank you @whuber. The return of 1 1 tells me that it does not reflect sampling with replacement. I have done some additional tests and updated my original question. It is still not clear to me what the inclusionprobabilities() function is returning.
Answer: Sampling with replacement is boring. Sampling without replacement is very interesting. That's why the authors of library(sampling) restricted their attention to sampling WOR. So inclusionprobabilities() takes the baseline rates in your y, and figure out what would the inclusion probabilities be should a proper unequal probability WOR sampling algorithm applied to these numbers.
Looking at the source code, I imagine that your snippet of code reproduces the "regular" case of inclusionprobabilities() when none of the inclusion probabilities exceed 1. In that regular case, the inclusion probabilities are simply the input probabilities scaled up so that their sum is equal to the target sample size. Note that inclusion probabilities refer to the units on the frame, rather than the specific samples, as your code does.
For sampling with replacement, I believe your calculations are correct, in that probability of each pair is the product of probabilities. Then what inclusionprobabilities refers to are the sums across all rows where either X1 or X2 are equal to 1, 2, 3 or 4 (the indices of the original units):
for(k in 1:4) {
  print(sum(df$p[df$X1==k|df$X2==k]))
}

This is to say, unit 1 appears in 1.8% of the samples, while unit 3, in 77.3% of the samples. However, these numbers sum up neither to 1 (as base probabilities should) nor to 2 (as correct inclusion probabilities should), and so they are kinda weird, in the end.


Problemas com parâmetro restrict_xpaths num crawler

Question: Não tenho experiência em Python mas decidi tentar fazer qualquer coisa com o Scrapy, para testes. Assim, estou a tentar recolher os artigos existentes numa determinada página, nomeadamente num elemento DIV com um ID devBody.
Nesse sentido, o meu objectivo é obter o título do artigo e o respectivo URL. Assim, estabeleci uma regra para percorrer apenas o conteúdo desse elemento.
Acontece que, por alguma razão, a recolha de links não se está a limitar apenas a esse elemento, o que faz com que estejam a ser recolhidos links não relevantes e que depois "baralham" os pares título-URL quando os tento construir.
Segue o código:
from scrapy import Spider
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import Rule
from stack.items import StackItem

class StackSpider(Spider):
    name = "stack"
    allowed_domains = ["dev.mysql.com"]
    start_urls = ["http://dev.mysql.com/tech-resources/articles/"]

    rules = (Rule(LinkExtractor(restrict_xpaths='//div[@id="devBody"]',), callback='parse'),)

    def parse(self, response):
        entries = response.xpath('//h4')
        items = []
        #usar um contador aqui não será, de certeza, a melhor solução mas foi a única que encontrei para não receber todos os dados recolhidos num único objecto
        i = 0
        for entry in entries:
            item = StackItem()
            item['title'] = entry.xpath('//a/text()').extract()[i]
            item['url'] = entry.xpath('//a/@href').extract()[i]
            yield item
            items.append(item)
            i += 1

Para tentar perceber o que se está a passar, recorri às Developer Tools do Chrome e, através de consultas XPath, tudo parece estar correcto.
No entanto, quando tento replicar a mesma lógica no código, algo corre mal. Segundo os logs, é dito que foram de facto recolhidos 57 links, mas há bastantes que estão fora do âmbito (a tal div com ID devBody).
Não faço ideia o que possa estar a causar este comportamento. Estou a usar a versão 1.0.5 do Scrapy e Python 2.7.
Agradeço desde já qualquer ajuda.
Answer: De acordo com esta resposta, alterou-se a estrutura do código para funcionar como pretendido. Aqui fica o resultado final:
from scrapy.spiders import Spider
from stack.items import StackItem

class StackSpider(Spider):
    handle_httpstatus_list = [403, 404]
    name = "stack"
    allowed_domains = ["dev.mysql.com"]
    start_urls = ["https://dev.mysql.com/tech-resources/articles/"]

    def parse_items(self, response):
        for row in response.xpath('//div[@id="devBody"]/h4'):
            item = StackItem()
            item['title'] = row.xpath('a/text()').extract()
            # get the full url
            item['url'] = response.urljoin(row.xpath('a/@href').extract_first())
            yield item


Prove that when two matrices satisfy $Span(A) \subset Span(B)$, there exists a matrix $C$ such that $A=BC$.

Question: Let $A=(v_1| ... |v_k)$ and $B=(u_1| ... | u_k)$ be two $n \times k$ matrices. If Span$\{v_1,..., v_k\}$ is included in Span$\{u_1,...,u_k\}$, prove that there exists a matrix $C$ such that $A=BC$.
I am a student and want to prove this homework problem. I simply don't know how to start it. Thank you.
Comment: See [this](https://math.stackexchange.com/a/569362/721644) and [this](https://math.stackexchange.com/a/3418416/721644).
Answer: Just go through the definitions. What does "Span{v1,..., vk} belongs to Span{u1,...,uk}" mean? Can you express that in terms of a linear algebra equation? Can you apply that equation to each column of $A$? What do you get when you combine the equations for all the columns of $A$? Can you combine all of the equations into one linear algebra equation?


A Ghost File on my website

Question: We have three IIS 7.5 server running on Windows 2008, R2, 64-bit OS.  Recently we have a problem with the two of them in relation to a one .htm file.  For some reason the site seems to serve an older version of the file even though the file on the local drive is a newer version.  If I were to delete the file and then hit the URL in the browser, I still get the old version displayed instead of a 404 or page not found.
I am new to IIS 7.5 and have delete the folder from the C:\Inetpub\Temp\IIS Temporary Compressed Files and still no luck.
Any idea where this particular file may be cached?  I have searched all drives on the computer and there are no other copies.
Comment: Do you have a proxy in the mix?
Answer: There's Output Caching in IIS 7.5 which you can configure in IIS Manager. I'd check there first to see if there's anything configured.
Make sure that you don't have a proxy hosting service like CloudFlare for the site too, that one's gotten me before.
Another thing to check is for any proxy/caching server on the network. If you're overlooking an ISA server and looking at the site on the network it could actually be gone already.
Open Internet Information Services (IIS) Manager:
On the taskbar, click Start, point to Administrative Tools, and then click Internet Information Services (IIS) Manager.
In the Connections pane, go to the connection, site, application, or directory for which you want to configure page output caching.
In the Home pane, scroll to Output Caching, and then double-click Output Caching.
In the Actions pane, click Add...
In the Add Cache Rule dialog box, type the file name extension you want to cache in the File name extension box, and then select the User-mode caching option, the Kernel-mode caching option, or both.
Select the options that you want to use for caching, and then click OK.
Hope that helps!
http://www.iis.net/configreference/system.webserver/caching
Comment: Our site is straight forward containing the content that is majority static.  The page having issue is a .htm file.  There are no proxy or ISA server in between.  Output caching is not configured at any level, site, application or directory.  I have used procmon utility but for the file in question, I do not see any activity.  All other files show up when accessed by w3wp.exe.  We will use wireshark next to see if and how the file is accessed.  Thanks.
Answer: I'd bet that you're seeing your web browser cache the content, rather than an issue on your server.
Comment: We checked that also.  We made sure that the browser is not caching the content.  Tested it with FireFox and IE 10.  Used F12 to run a debugger in IE10 and cleared the cache but still getting old content.  Thanks.
Answer: Have you confirmed that the content is actually being served by IIS? You can do this by checking the log files. If there is a cache or other proxy between you and the server (which could be transparent proxy, put in place by your ISP, that you can't normally see our a reverse proxy nearer the other end) then it could be handling the request and never sending it to the web server even though your local cache definitely isn't used.
Comment: I have used procmon to capture activity on the server but it does not show file being accessed.  We don't have an ISP.  The entire site is hosted in our datacenter.  I am hitting the site directly using the IP address of the site as a URL in the browser.  There are three hops to the site.  Two gatways and a Firewall.  Unless the firewall is caching the content which I know it does not I don't this that is an issue.   Also, this issue is only happening with one file and not with any others.  Thanks.

A Ghost File on my website


Nginx two servers were invoked when processing 1 request

Question: Seeing this in nginx logs:
"upstream": "126.96.36.199:443, 188.8.131.52:443",
"upstream_response_time": "7.005, 7.016",

The documentation of nginx says:
If several servers were contacted during request processing,
their addresses are separated by commas,
e.g. "192.168.1.1:80, 192.168.1.2:80, unix:/tmp/sock".

Unfortunately it is not clear to me how can two servers be invoked when processing 1 request? This is w/o internal redirects? Was it a retry after the first attempt failed?
The DNS query returns two A records:
xxx.us-east-1.elb.amazonaws.com. 60 IN A 184.108.40.206
xxx.us-east-1.elb.amazonaws.com. 60 IN A 184.108.40.206

Does it mean, than nginx does a retry for every failed request automatically? Or can it be configured? (this is AWS so the IP's of the load balancers are changing constantly)
Answer: I think one situation is:

request invoke first backend but it is failed, then invoke next one. if backend all failed ,it return 502

BTW:this is my case
Comment: I guess it could be the case. Any other cases you might think off. I hope I will have time to test it out.
Comment: i think same case , but if backend connect success ,but return  an error http header, will nginx invoke next backend ? i think this case is extrend .


Does the norm of a (1,1) tensor depend on the conformal factor?

Question: If we have a $(1,1)$ tensor $T$ on a (pseudo-)Riemannian manifold $(M,g)$, its norm is
$$
g(T, T) = g_{AB}g^{CD}T^A_C T^B_D.
$$
If we let $h = \phi^2 g$ be a conformally equivalent metric, then its inverse is $h^{-1} = \phi^{-2}g^{-1}$ and thus the norm of $T$ as measured by $h$ is identical, as the conformal factors cancel out. This only happens when $T$ is a mixed tensor which is covariant in the same number of indices as it is contravariant.
Questions: Is this proof correct? Is there any geometric reason why these types of tensors have their norm preserved by conformal transformations?
Comment: The (1,1) tensors can be seen as linear maps. I think that a good interpretation of the phenomenon you observed should use this.
Answer: The proof is correct. Giuseppe Negro's comment about $(1,1)$-tensors being regarded as linear maps is also right on point. The same works for $(r,r)$-tensors, as $$h_{i_1j_1}\cdots h_{i_rj_r}h^{k_1\ell_1}\cdots h^{k_r\ell_r} T^{i_1\cdots i_r}_{\qquad k_1\cdots k_r}T^{j_1\cdots j_r}_{\qquad \ell_1\cdots \ell_r} = g_{i_1j_1}\cdots g_{i_rj_r}g^{k_1\ell_1}\cdots g^{k_r\ell_r} T^{i_1\cdots i_r}_{\qquad k_1\cdots k_r}T^{j_1\cdots j_r}_{\qquad \ell_1\cdots \ell_r}, $$since cancellations happen in pairs. Linear algebra: a multilinear map $(V^*)^{\times r} \times V^{\times r} \to \Bbb R$ can be seen as a linear map $V^{\otimes r} \to V^{\otimes r}$, so there you have it. If the quantity of covariant indices is not equal to the quantity of contravariant indices, you'll have extra $V$ or $V^*$ factors which will break the conformal invariance.
Comment: Yes, exactly. The thing is that an inner product in $V$ induces an inner product in $V^*$ (the components of the induced inner product in $V^*$ are $g^{ij}$) and also an inner product in ${\rm Hom}(V,V)$. The isomorphism ${\rm Hom}(V,V) \cong V^*\otimes V$ turns out to be an isometry. Another comment that might be enlightening: given a bilinear $B\colon V \times V \to \Bbb R$, of course one can think of it as $\widetilde{B}\colon V \to V$ via $B(v,w) = g(\widetilde{B}(v),w)$, but $B \mapsto \widetilde{B}$ depends on the inner product, while ${\rm Hom}(V,V) \cong V^*\otimes V$ does not.
Comment: In this spirit, and Gisueppe Negro's comment, is the reason I should expect the norm of a linear map $T$ to be conformally invariant just because the operator norm on the normed vector space each metric induces is the same if we scale the norm by a constant factor?


At what steps is Replacement required in definitions by Transfinite Recursion?

Question: I am trying to pin-point where Replacement is required in a specific definition by Transfinite Recurson. In the general proof of the Transfinite Recursion Theorem, it is noted that Replacement is used

to guarantee that $F \restriction \alpha$ (i.e. the range of the function being defined) is always a set (because its domain $\alpha$ is a set),
to prove that $dom (F)$ is all of ON (by reductio)

But suppose our specific definition is something like:
$P(0) = 0$
$P(\alpha+1) = P(\alpha) \cup \wp (P(\alpha))$
$P(\lambda) = \bigcup_{\alpha < \lambda} P(\alpha)$
I don't see where Replacement is necessary since it seems that the existence of
$P(\alpha+1)$ follows by Union and Powerset, and the existence of $P(\lambda)$ follows from that by Union. Suppose, moreover, that we just want to define this function up to some set $\lambda$ (as opposed to a proper class). Then is Replacement being used at all?
Comment: Note also that, the way you've written it, you use the axiom of pairing to obtain $P(\alpha +1)$. (This, however, isn't necessary as defining $P(\alpha+1) = \wp(P(\alpha))$ gives the same sequence.)
Answer:
the existence of $P(\lambda)$ follows from that by Union

This isn't true. To get $P(\lambda)$ from Union we would need, not just each $P(\alpha)$ for $\alpha<\lambda$, but the whole set $\{P(\alpha):\alpha<\lambda\}$. This isn't something we can get without Replacement.
The easiest way to convince yourself that this might be an issue, I think, is to think about the set $V_\omega$. $V_\omega$ is a model of all of ZFC except for the axiom of infinity; in particular, it satisfies Union. And each finite ordinal is in $V_\omega$. But we can't apply Union to get $\omega\in V_\omega$, since we don't have a single set containing all the finite ordinals to apply it to. Now this is a silly example, but hopefully it convinces you that Union on its own doesn't actually have very much power in terms of building "limit stages" of sequences of sets.
Think of it this way: in order to use Union, we need to already have "collected" all the sets whose union we want to take into one single set. And this "collection" is where Replacement (usually) comes in. (On that note, it's worth pointing out that there's an axiom scheme called "Collection" which is in most contexts equivalent to Replacement.)

It's a good exercise at this point to show the following:

$V_{\omega+\omega}$ is a model of Z (= ZF without the Replacement scheme).

(And in fact ZF proves this.) There's nothing particularly hard here, just go through the axioms one by one and show they hold in $V_{\omega+\omega}$.


Why aren't my local tasks shown?

Question: I have the module.routing.yml as
entity.node.ssl_check:
  path: '/node/{node}/check-ssl'
  defaults:
  _form: '\Drupal\ssl_check\Form\SslCheck'
  _title: 'Check SSL Cert'
  requirements:
  _permission: 'access check ssl tab'

and module.links.task.yml as
 entity.node.ssl_check:
  route_name: entity.node.ssl_check
  title: 'Ssl check'
  base_route: entity.node.canonical
  weight: 15

Here when I access node/1104/check-ssl/ the form defined in the form controller is loading. But my other need that is to show a 'Ssl check' tab when I visit path /node/1104/ is not there. I have done the tab thing while dealing with Controller but not working with form controller. Please help. I have checked 'Add tab / local task to a node' this but there was nothing dealing with form controller.
Comment: The module.routing.yml file you are showing is not well-formed; it misses the indentation for `defaults`.
Comment: Also, as for creating local tasks, there isn't any difference between using a form or a controller: What you put in the .links.task.yml file is the same for both the cases.
Answer: Since everything looked just fine in your code, I tried the following in a custom module:
#test.links.task.yml
test.node_test:
   title: 'Wooorking'
   route_name: entity.node.canonical
   base_route: entity.node.canonical

And this is working like a charm. Also when I added a custom routing, which was pointing to a form controller, it showed up! (although not worked, because I didn't handled the node parameter in the routing configuration)
So the problem lies another where.
So I suggest you to do it step by step way (building your routing and task yml) to understand your problem. Clearing the cache each time when you change something in your YML files. If you found the problem, please post the solution, so also we can learn from this.
Answer: Apparently, if there's only one task, it won't appear.
You'll watch the links appear if you add 2 or more links.
In other words, this wont appear alone :
entity.node.ssl_check:
  route_name: entity.node.ssl_check
  title: 'Ssl check'
  base_route: entity.node.canonical

This will show two tabs
entity.node.ssl_check:
  route_name: entity.node.ssl_check
  title: 'Ssl check'
  base_route: entity.node.canonical

entity.node.ssl_check2:
  route_name: entity.node.ssl_check
  title: 'Ssl check again'
  base_route: entity.node.canonical

Answer: Modified the code as
 entity.node.ssl_check:
      title: Ssl check
      route_name: entity.node.ssl_check
      base_route: entity.node.canonical
      weight: 15

Answer: I had copied and pasted my code wrong. Things to check:

The file name is YOUR_MODULE.links.task.yml
The task name is YOUR_MODULE.some_task
The route name is YOUR_MODULE.some_route

The last one got me. It appears that if the route does not exist, the local task will silently not show.

File: YOUR_MODULE.links.task.yml
YOUR_MODULE.some_task
  title: 'Example Title'
  route_name: YOUR_MODULE.manual_email
  base_route: YOUR_MODULE.manual_email

Why aren't my local tasks shown?


How to pass params from a stack navigator to a Material top tab navigator?

Question: I have a Stack named as PaymentStack which consists of two Stack screens. The second screen consists of a MaterialTopTabs which have two other Screens named as 'Net banking' and 'Cash Payment'.
PaymentStack consists of two screens, one screen is a simple Stack screen navigator with a functional component passed in the component attribute of Stack.Screen but for the other screen I have tried to make a Material Top tabs Navigator
const TopTabStack = createMaterialTopTabNavigator();

//this MaterialTopTab is passed as a component in the below PaymentStack tab navigator
const MaterialTopTab = ({ navigation }) => (
    <TopTabStack.Navigator>
        <TopTabStack.Screen
            name={'Net banking'}
            component={ComponentX} //*I want to receive the params in this screen/cmpnt.
        />
        <TopTabStack.Screen
            name={'Cash payment'}
            component={ComponentX}
        />
    </TopTabStack.Navigator>
)

//this PaymentStack is passed as a component in Bottom tab navigator
const PaymentStack = ({navigation}) => {
      <Stack.Navigator>
        <Stack.Screen
            name={'PaymentOption'}
            component={ComponentY} //Suppose I am on this Screen
        />
        <Stack.Screen
            name={'MaterialTopTab'}
            component={MaterialTopTab} //this component is defined above
        />
    </Stack.Navigator>
}

export default function AppStack = () => {
   return (
    <NavigationContainer>
       <PaymentStack /> //defined above
    </NavigationContainer>
  );
}

Now suppose I am in the screen named PaymentOption whose component is ComponentY and I want to navigate to the Screen named as Material top tab whose component is a Material top tab navigator in which two screens are present named as Net banking and Cash payment and receive the params in the Net banking. How can we do this ?
I tried
navigation.navigate('MaterialTopTab', {body : data} )

But when I am accessing the body in ComponentX , it shows undefined.
function ComponentX({route}){
   console.log(route.params.body)
   .......
   .......
}

Comment: Wouldn't you be accessing `route.params` in the `` component instead? Since that's where you're passing it to
Comment: Ok, But MaterialTopTab is a Tab navigator, how am I supposed to access from it and then pass it to one of its screen names as 'Net banking'?
Answer: Ok, hopefully this is what you're looking for. Please ignore the typescript typings if you aren't using Typescript. As you can see, our RootStack component is a stack navigator that contains 1 screen and a TabNavigator. I don't have experience using top tabs but I'm pretty sure it works the same way as bottom tabs. We can navigate to our TabNavigator from the first screen in RootStack (Screen1), passing a prop called helloKitty to the TabNavigator. The TabNavigator then passes the helloKitty prop from route.params to the first screen in the TabNavigator (TabScreen1). I'm not particularly sure if what I did with TabScreen1Props is the proper way to do it, but it works for now. Honestly though, at this point, if this is what your navigation structure looks like, why not consider useContext or a state management library like Zustand?
import React, { useEffect } from 'react'
import { Text, View } from 'react-native'
import { createStackNavigator, StackScreenProps } from '@react-navigation/stack'
import { BottomTabScreenProps } from '@react-navigation/bottom-tabs'
import { Button } from 'react-native'
import { NavigationContainer } from '@react-navigation/native'

type HelloKittyType = string
type RootStackParamList = {
  Screen1: undefined
  SomeCuteTabs: { helloKitty: HelloKittyType }
}
type RootStackNavigationProps = StackScreenProps<RootStackParamList>
type SomeCuteTabsParamList = {
  TabScreen1: undefined
  TabScreen2: undefined
}
type SomeCuteTabsNavigationProps = BottomTabScreenProps<SomeCuteTabsParamList>

const RootStack = createStackNavigator<RootStackParamList>()
const SomeCuteTabs = createStackNavigator<SomeCuteTabsParamList>()

const Screen1 = ({ navigation }: RootStackNavigationProps) => (
  <View>
    <Button title="Go to tabs" onPress={() => navigation.navigate('SomeCuteTabs', { helloKitty: 'I love hello kitties' })} />
  </View>
)

interface TabScreen1Props extends SomeCuteTabsNavigationProps {
  helloKitty: HelloKittyType | undefined
}
const TabScreen1 = ({ route, navigation, helloKitty }: TabScreen1Props) => {
  useEffect(() => {
    console.log(helloKitty)
  }, [])
  return (
    <View>
      <Text>Tab Screen 1</Text>
    </View>
  )
}
const TabScreen2 = () => (
  <View>
    <Text>Tab Screen 1</Text>
  </View>
)

const RootStackComponent = () => (
  <RootStack.Navigator>
    <RootStack.Screen name="Screen1" component={Screen1} />
    <RootStack.Screen name="SomeCuteTabs" component={SomeCuteTabsComponent} />
  </RootStack.Navigator>
)

const SomeCuteTabsComponent = ({ route, navigation }: RootStackNavigationProps) => (
  <SomeCuteTabs.Navigator>
    <SomeCuteTabs.Screen name="TabScreen1">{(props) => <TabScreen1 {...props} helloKitty={route.params?.helloKitty} />}</SomeCuteTabs.Screen>
    <SomeCuteTabs.Screen name="TabScreen2" component={TabScreen2} />
  </SomeCuteTabs.Navigator>
)

export default function App() {
  return (
    <NavigationContainer>
      <RootStackComponent />
    </NavigationContainer>
  )
}

Comment: Ok, I'll give it a try, thanks for answering! And could you also provide some resources to use navigation with useContexf?
Comment: @Madhavmishra `react-navigation` is on v6 already, but this should still help https://stackoverflow.com/questions/62242973/how-to-use-context-api-in-react-navigation-5 But personally, I would go with `zustand` for its simplicity and ease of use in projects of any size, and unless you like writing lots of code, probably `redux-toolkit` (RTK) for very large projects


What methods can be used to overcome the tie-breaking when using majority voting in ensemble?

Question: What methods can be used to overcome the tie-breaking when using majority voting in ensemble? I read that Weighted majority voting can help; however, it wasn't effective on the dataset I am using in terms of the F1 score and it dropped significantly.
Comment: There are about a million options here, but nobody can make a universal pronouncement about 'best'.
Comment: More votes (ie more trees) will lead to less ties. An odd number of votes will eliminate ties in binary classification
Comment: What's wrong with a tie? (I have my thoughts, but it will help to hear yours.)
Comment: @Dave Currently I am doing an Unsupervised binary classification (0,1). For the majority voting, I am using `mode` from `scipy`. Once I have a tie break the `mode` chooses 0 which is affecting my f1 score. I thought of adding an odd number of base learners. However, my base learners are created from two algorithms with different parameter sets. Therefore, I cant choose (favour) one algorithm over the other to break the tie
Comment: Would you prefer that some of the ties be called $1$ instead of $0$?
Comment: @Dave My aim is to use a method that breaks the ties and improve my results
Comment: But why are the ties a problem?
Comment: @Dave I think ties make the results bias
Answer: a) count the votes, select the class with more votes
b) count the votes weighted by the confidence (or probability) the base classifier assigns to its decision. Select the class with higher weighted vote.
c) count the votes. If there is a tie (and only if there is a tie) select the class with higher weighted votes for only the classes tied as best. (I think sklearn uses this at least in their OVO implementation)
None of them is "best". This third one is not equivalent to the 2nd and so it may be worth it to try.
Comment: Usually, classifiers output a degree do confidence (which could be a probability or another number)  for the data in relation to each class. This is the `decision_function()` method output in sklearn. Add such degrees of confidence for ach class and select the most confident.
Comment: Point B is not clear to me/ Could you please explain it a bit more. Point C is an interesting approach, definitely I am going to try it.
Comment: Thank you for the explination.


Probability statement with a confidence interval

Question: Suppose you construct a 95% confidence interval on a mean.  Can you state that there is a 97.5% probability that the true population mean lies below the CI upper bound and that there is a 97.5% probability that the true population mean lies above the CI lower bound?
Comment: [This question](http://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval) may be helpful.
Comment: There are actually two reasons why the answer is 'no' in general. The first is given in Alexis' answer and SeanEaster's link. The second reason is that $1-\alpha$ confidence intervals do not *necessarily* put exactly $\alpha/2$ in each tail. Indeed when dealing with a discrete parameter, a CI for it generally can't.
Answer: No.
The confidence interval (specifically this beast: $\bar{x} \pm z_{(1-\text{CL})/2}\hat{\sigma}_{\mu}$, where CL is the confidence level) is not a statement about the probability of the mean, $\mu$, which either is or is not contained within any specific interval with no possibility of both conditions obtaining. The substantive meaning of the frequentist confidence interval does not really reduce beyond:
If you repeated the study that produced your sample data many, many times so that you gathered many many samples of size $N$ you would expect that the CL% confidence intervals constructed on the means of each of those samples would contain the true population mean CL% of the time.
Of course, there are other kinds of estimation and inference intervals with different substantive meanings.
Comment: @Alexis Here you go: https://stats.stackexchange.com/questions/552783/practical-consequences-of-wrong-interpretation-of-confidence-intervals
Comment: Suppose you have a set of data, compute the sample mean and the CL% confidence interval on the mean. What does the single CI that you have tell you, and what statement can you make with the single CI that you have?
Comment: You can say you are *this confident*, where confident has the meaning I gave in my second paragraph. It's not especially pretty. :\ That said, it is a way of evaluating $\sigma_{\mu}$, which, if $\hat{\sigma}_{\mu} \approx \sigma_{\mu}$ is true, should give you an idea of the trustworthiness of $\bar{x}$ as an estimate of $\mu$... where "trustworthiness" is a synonym for "confidence." Sorry, it just doesn't get much better.
Comment: OK thanks.  So if I understand you correctly, you cannot say that there is a 95% chance that the single confidence interval you have contains the mean?
Comment: That is correct.
Comment: @Alexis is it possible to make a simulation that proves so?
Comment: @mihagazvoda No that is just plain frequentist logic: either your single confidence interval **does contain** $\boldsymbol{\mu}$ or it **does *not* contain** $\boldsymbol{\mu}$ **.** Assume you calculate a mean of 6 and a 95%CI for $\mu$ to be 3, 9. If $\mu$ actually equals 10 there is a 0 probability that your CI contains it. If $\mu$ actually equals 4.5, there is probability 1 that your CI contains it. **Once you have calculated a CI, it either *does* or *does not* contain your population statistic of interest.**
Comment: @Alexis I understand. But - are there any practical consequences if you interpret it as a Bayesian credible interval (because a lot of people do)? Can you give me an example that will show that you come to the wrong conclusion/result if you use the wrong interpretation?
Comment: @mihagazvoda Post that as a separate question!


Expectation of Sufficient Statistic for Beta Distribution

Question: I am looking at question 1b of the following notes: http://www.gatsby.ucl.ac.uk/teaching/courses/ml1/asst1.pdf
In 1a, I have shown that the Beta distribution has a density that can be written in the form $g(\boldsymbol{\theta}) f(\textbf{x}) e^{\phi(\boldsymbol{\theta})^T T(\textbf{x})}$ where:
$g(\boldsymbol{\theta}) = \frac{1}{B(\alpha,\beta)}$
$f(x) = \frac{1}{x(1-x)}$
$\phi(\boldsymbol{\theta}) = \begin{pmatrix} \alpha \\ \beta \end{pmatrix}$
$T(x) = \begin{pmatrix} \log{x} \\ \log{(1-x)} \end{pmatrix}$
Now, according to this thread, I can calculate the expectation of the sufficient statistic, $E[T(x)]$, by rewriting the density as
$f(\textbf{x}) e^{\phi(\boldsymbol{\theta})^T T(\textbf{x})} e^{-A(\boldsymbol{\theta})}$
and then using $E[T(x)]_i = \frac{A'(\boldsymbol{\theta})_i}{\phi'(\boldsymbol{\theta})_i}$
To do this, we define $A(\theta) = - \log{g(\theta)} = \log{B(\alpha,\beta)}$ but then I become uncertain of what to do. My attempt so far looks like:
$A'(\theta) = \frac{1}{B(\alpha,\beta)} \begin{pmatrix} B_\alpha \\ B_\beta \end{pmatrix}$ where $B_\alpha = \frac{\partial B}{\partial \alpha}$ etc. I also find $\phi'(\theta) = \begin{pmatrix} 1 \\ 1 \end{pmatrix}$.
I'm pretty sure I've made a mistake with the calculation of $phi'(\theta)$ since the vector of ones seems wrong, and yet $\phi'(\theta) = \frac{\partial}{\partial \theta_i} \phi(\theta) = \begin{pmatrix} \frac{\partial}{\partial \alpha} \\ \frac{\partial}{\partial \beta} \end{pmatrix} \phi(\alpha, \beta)$. Is this going to become a 2x2 matrix?
I have managed to get this technique to work for distributions such as binomial and Poisson where the sufficient statistic is a scalar. I am getting confused in this case where we have a vector of sufficient statistics.
Comment: Please make the question self-contained rather than expecting the reader to follow a web link.
Answer: The thread you provide only operates for one-dimensional exponential families. In general, when the exponential family is written wrt its natural parameter:
$$f(x) e^{\theta^T T(x)} e^{-A(\boldsymbol{\theta})}$$
the mean of the associated (sufficient) statistic is
$$\Bbb E_\boldsymbol\theta[T(X)] = \nabla A(\boldsymbol{\theta})$$
Since this is the case for the Beta distribution, namely $\phi(
\boldsymbol\theta)=\boldsymbol\theta$,
$$\nabla A(\boldsymbol{\theta})=\frac{1}{B(\alpha,\beta)} \begin{pmatrix} \frac{\partial B(\alpha,\beta)}{\partial\alpha} \\ \frac{\partial B(\alpha,\beta)}{\partial\beta} \end{pmatrix}$$
Comment: You can always reparameterise $\phi(\theta)$ as the default $\theta$.
Comment: It seems that we cannot always have $\phi(\theta) =\theta$ however. For example, with the gamma distribution, I find $\phi(\theta) =( \alpha - 1, - \beta) ^T$. Under these circumstances, if I follow through the algebra, we would have $(\nabla \phi) E[T(x)] = \nabla A$. Is that correct?


Using a mixed effects model vs. just adding dummies for every subject?

Question: I'm a little fuzzy on the exact assumptions needed for mixed/fixed effects models. As an example, let's say we're trying to model the effect of age on a person's 5k time, and we have a dataset of race times by person, by year (so multiple observations per person at different ages).
From what I remember, the naive OLS way would be to regress race time on age. We don't want to do this because there are multiple observations per person, which violates the OLS assumption of independent observations. We can introduce a random effect for person to the model, which "allows" for each person to have their own intercept and lets us see the within-subject effect of age on race time. I believe this is a pretty standard way to deal with multiple observations per subject.
However, what's the difference between:

Using a mixed-effects model as specified, and
Using a fixed-effects model but using dummy variables for each person? In essence, why can't we just regress race time on age + Person A + Person B + ..., where Person [x] is a dummy for a particular person in the data? Isn't this also effectively allowing each person to have their own intercept?
Answer: The difference is, that in the random-effects model, you will have a shrinkage effect: While in the fixed-effects model you get offsets that give the best fit for each person, independent of the offsets of the others, the random-effects model also tries to make those offsets similar. The idea is that the offset of one person is somehow an indicator for the offset of another person to be similar. Because, they are both, after all, "persons". This especially helps e.g. if you have persons with lots of data, where you are quite sure about the fitted offsets, and other persons with only very little data which leads to large uncertainty about the offsets. In this case, the requirement that the offsets of all the persons are similar will draw them together ("shrinking" them together), and draw the uncertain offsets towards the certain offsets.
Answer: Using dummy variables for each person is called the LSDV approach. If the variables stay the same, the coefficient for "age" will be the same for both LSDV and Fixed effects even though the estimation approach is different.
Fixed effects/LSDV are used when the individual effects are not random but are correlated with the independent variables. Conversely, random effects assume that there is no such correlation. This forms the basis of the Hausman test as well. You can use the Hausman Test to determine whether to apply fixed or random effects.
This can help you choose the appropriate approach between fixed and random effects.
As for the Fixed effects, LSDV or dummy variable approach, there are some things to keep in mind when using these. First, these models don't allow you to include any time-invariant independent variables. For example, you can't estimate the effects of gender, race etc. If you are including only age as the independent variable, you don't need to worry about this. But I would still suggest you include some control variables that may affect the 'race time'.
Secondly, the dummy variables take away degrees of freedom because you are estimating more coefficients (one for each individual).
Comment: I'd just like to point out that using a test (e.g. Hausman) to decide on your model is not considered good practice because it creates problems with the validity of inference down the line.
Comment: @Stefan Can you please elaborate more on your point, how does it create problems? Maybe with an example. Thank you for your input.
Comment: The problem is that using the same data for model selection and inference creates problems of data dredging (see e.g. https://stats.stackexchange.com/a/20856/200803), a keyword here is inference after model selection.


Login to website with multiple client certificates

Question: I have users that need to access a website with a different client certificate depending on what function they are trying to perform.
The problem is that once the user logs in with one of the client certificates, the only way they can log in with a different certificate is to close every instance of Internet Explorer that is open first.
Is there any way to force Internet Explorer to try to log in with a different certificate?
I have tried having the users click the "Clear SSL state" button under Internet Options > Content, but that seems to only work about 25% of the time.
Answer: why dont you get a SAN certificate, that would work across multiple domains..by using a single certificate..
Comment: We have no control over the certificates or the website. The website generates the certificates and we can only use the certificates that they generate. They give us one certificate per role but we have users that fill multiple rolls, which means they have to use more than one certificate.


Amazon EC2: Jetty Unaccessible

Question: I'm trying to install jetty in an Amazon EC2 instance running Ubuntu. The installation seems OK
$ sudo apt-get install jetty

After that I made the parameter NO_START=0  in /etc/default/jetty
Finally
$ sudo service jetty start
    Starting Jetty servlet engine. jetty
    Jetty servlet engine started, reachable on http://ip-xx-xxx-xxx-xxx:8080/
    ...

Nevertheless, when I write in the browser (external computer):
 http://ec2-xxx-xx-xx-xx.compute.amazonaws.com:8080

The browser could not connect to the address using the given port. I don't know if something is missing during the installation or the URL is wrong.
Thanks for your time and help :-) ,
- Carolina
Comment: When you are on the instance, does this connect: telnet localhost 8080
Comment: @EricHammond : I do curl http://localhost:8080/ and I get the expected response. Yet, when I try from an external machine it doesn't work. I should note that I also installed an ubuntu VM on a Win7 guest machine, and I have the exact same problem.
Comment: I don't know Jetty, but some applications like MySQL and Apache have to be told to listen on more than just localhost.
Answer: Each instance on amazon has a security group assigned to it. In the AWS console, select the instance and the security group is shown in the lower pane. Then select security groups on the left and select the group for that instance from the table. In the lower tab, select inbound, these are the IPs addresses/ports allowed in. If 8080 isn't allowed, that's your problem, you need to add it. Enter 8080 in the port range box and 0.0.0.0/0 for the source. Then "add rule" and make sure you "apply rule change" lastly.
Comment: I have the exact same problem. However opening the 8080 port on the security group had no effect. What else could be the problem ?
Answer: I found the answer here:
Jetty (mis)configuration: it wont respond from an external machine... why?
You need to change /etc/default/jetty to have JETTY_HOST=0.0.0.0

Amazon EC2: Jetty Unaccessible


Add New Item to Already Bound ListView in ASP Net (Unable to set DataKeys/FieldName)

Question: My overall goal is to add fake/unbound items to a listview control (for final HTML Table output reasons). This is a code behind solution. Users will not be adding items as it will be outputted in a rigid table.
I have looked at several examples and while this is easy for a dropdown it is not for listview.
The code below works without error, but my item is not shown on runtime. I think the class is not setting the item fieldname correctly, but I can't figure out the right syntax to fix it.
ColumnNameAList.DataSource = PeriodDataView
ColumnNameAList.DataBind()

Dim test As New Example1("ColumnNameA")

Dim newItem As New ListViewDataItem(ColumnNameAList.Items.Count, ColumnNameAList.Items.Count)
newItem.DataItem = test

ColumnNameAList.Items.Insert(ColumnNameAList.Items.Count, newItem)
ColumnNameAList.Items.Add(newItem)

Here is the Example1 class that is supposed to set the DataValueField:
Public Class Example1
Public Sub New(ColumnNameA__1 As String)
    ColumnNameA = ColumnNameA__1
End Sub

Private m_ColumnNameA As String

Public Property ColumnNameA() As String
    Get
        Return m_ColumnNameA
    End Get
    Set(value As String)
        m_ColumnNameA = value
    End Set
End Property

End Class

This outputs my original datasource list, but not the added item.
<ItemTemplate>

        <%# Eval("ColumnNameA")%>

</ItemTemplate>

Comment: At what point are you invoking your insertion code? It should be executed as a listener on the [`ListView.DataBound`](http://msdn.microsoft.com/en-us/library/system.web.ui.webcontrols.basedataboundcontrol.databound.aspx) event, otherwise databinding will empty any elements you have populated manually.
Comment: I'm working in someone else's code and they have a custom sub that then has the "ColumnNameAList.DataBind()" called. I placed the manual insertion code in that same sub after the "ColumnNameAList.DataBind()" line. I assumed that since I am modifying the listview after databinding that those changes would be kept.
Comment: The custom sub I mentioned above resides in the Page_PreRender area.
Answer: In the end I could only reliably solve this with a codebehind solution.
I made a copy of the original datasource, modified my copy and then databound to it.
Dim MyOriginalTableSource As Data.DataView = DataManager.example()
Dim ModifiedTable As DataTable = MyOriginalTableSource.ToTable
'do stuff here
Mylistbox.DataSource = ModifiedTable
Mylistbox.DataBind()

Won't work for everyone, but in this case it works fine for me.
Answer: There could be a couple of issues with the way you are approaching this, including that the ListView is already databound and that you are both adding and inserting the newItem.
When we have a scenario like this, we take one of two approaches:
1) Add the new item to the data source before the source is data bound.
2) Remove databinding and manually create each of the list view items, then add your new item at the beginning or end of the loop.
Comment: Looks like I will have to follow approach #2 and add them manually because I am trying to insert items and change the list order.  Do you have any concise examples of how to do the postbacks if you manually create a list of items in a listview?
Answer: Another way to do it would be to inject it into the sql.
select col1, col2, col3 from table1 union select '1','2','3'

this would ensure that the item is always added, and asp.net doesn't need to know or care.
You can add this into the sql query or add it from the behind code before binding query. if you are not binding with sql, you can also do this to any list item with LINQ