How to deal with hierarchical / nested data in machine learning

Question: I'll explain my problem with an example.  Suppose you want to predict the income of an individual given some attributes: {Age, Gender, Country, Region, City}. You have a training dataset like so
train <- data.frame(CountryID=c(1,1,1,1, 2,2,2,2, 3,3,3,3),
             RegionID=c(1,1,1,2, 3,3,4,4, 5,5,5,5),
             CityID=c(1,1,2,3, 4,5,6,6, 7,7,7,8),
             Age=c(23,48,62,63, 25,41,45,19, 37,41,31,50),
             Gender=factor(c("M","F","M","F", "M","F","M","F", "F","F","F","M")),
             Income=c(31,42,71,65, 50,51,101,38, 47,50,55,23))
train
   CountryID RegionID CityID Age Gender Income
1          1        1      1  23      M     31
2          1        1      1  48      F     42
3          1        1      2  62      M     71
4          1        2      3  63      F     65
5          2        3      4  25      M     50
6          2        3      5  41      F     51
7          2        4      6  45      M    101
8          2        4      6  19      F     38
9          3        5      7  37      F     47
10         3        5      7  41      F     50
11         3        5      7  31      F     55
12         3        5      8  50      M     23

Now suppose I want to predict the income of a new person who lives in City 7.  My training set has a whopping 3 samples with people in City 7 (assume this is a lot) so I can probably use the average income in City 7 to predict the income of this new individual.
Now suppose I want to predict the income of a new person who lives in City 2. My training set only has 1 sample with City 2 so the average income in City 2 probably isn't a reliable predictor.  But I can probably use the average income in Region 1.
Extrapolating this idea a bit, I can transform my training dataset as
    Age Gender CountrySamples CountryIncome RegionSamples RegionIncome CitySamples CityIncome
 1:  23      M              4         52.25             3        48.00           2    36.5000
 2:  48      F              4         52.25             3        48.00           2    36.5000
 3:  62      M              4         52.25             3        48.00           1    71.0000
 4:  63      F              4         52.25             1        65.00           1    65.0000
 5:  25      M              4         60.00             2        50.50           1    50.0000
 6:  41      F              4         60.00             2        50.50           1    51.0000
 7:  45      M              4         60.00             2        69.50           2    69.5000
 8:  19      F              4         60.00             2        69.50           2    69.5000
 9:  37      F              4         43.75             4        43.75           3    50.6667
10:  41      F              4         43.75             4        43.75           3    50.6667
11:  31      F              4         43.75             4        43.75           3    50.6667
12:  50      M              4         43.75             4        43.75           1    23.0000

So, the goal is to somehow combine the average CityIncome, RegionIncome, and CountryIncome while using the number of training samples for each to give a weight/credibility to each value. (Ideally, still including information from Age and Gender.)
What are tips for solving this type of problem? I prefer to use tree based models like random forest or gradient boosting, but I'm having trouble getting these to perform well.
UPDATE
For anyone willing to take a stab at this problem, I've generated sample data to test your proposed solution here.
Comment: Hierarchical bayesian models are very natural to exploit structure as described in your data. Check out the classical example on Radon contamination modeling http://mc-stan.org/documentation/case-studies/radon.html
Comment: @VihariPiratla Thanks for the input, but I'm familiar with the Titanic dataset and challenge, and I don't see how it relates to the nested data issue I asked about.
Comment: Take a look at this Kaggle Getting Started tutorial challenge: https://www.kaggle.com/c/titanic. It deals with a similar problem, that is to predict if a person has survived the Titanic disaster given various attributes about the person such as Gender, ticket type etc. The best submitted solutions for this use advanced methods such as Gradient Boosting and Hierarchical Bayesian Models etc.
Comment: Using L2/L1 regularisation for those models eg logistic regression/SVM that perform it should help (poor mans hierarchical bayes). effectively you penalize coefficients, so unless a coefficient significantly ( ie on lots of data) improves error it will be set to close to zero. and you use crossvalidation to decide level of penalisation
Comment: Do you also want to be able to predict the income of a person who lives in city 9?
Answer: I have been thinking about this problem for a while, with inspirations from the following questions on this site.

How can I include random effects into a randomForest?
Random forest on grouped data
Random Forests / adaboost in panel regression setting
Random forest for binary panel data
Modelling clustered data using boosted regression trees

Let me first introduce the mixed-effects models for hierarchical/nested data and start from a simple two-level model (samples nested within cities). For the $j$-th sample in the $i$-th city, we write the outcome $y_{ij}$ as a function of covariates $\boldsymbol x_{ij}$ (a list of variables including gender and age),
$$ y_{ij}=f(\boldsymbol x_{ij})+{u_i}+\epsilon_{ij},$$
where ${u_i}$ is the random intercept for each city, $j=1,\ldots,n_i$. If we assume $u_i$ and $\epsilon_{ij}$ follow normal distributions with mean 0 and variances $\sigma^2_u$ and $\sigma^2$, the empirical Bayesian (EB) estimate of $u_i$ is $$\hat{u}_i=\frac{\sigma^2_u}{\sigma^2_u+\sigma^2/n_i}(\bar{\mathbf{y}}_{i.}-f(\bar{\boldsymbol x}_{i.})),$$ where $\bar{\mathbf{y}}_{i.}=\frac{1}{n_i}\sum_i^{n_i}y_{ij}$, $f(\bar{\boldsymbol x}_{i.})=\frac{1}{n_i}\sum_i^{n_i}f(\boldsymbol x_{ij}).$ If we treat $(\bar{\mathbf{y}}_{i.}-f(\bar{\boldsymbol x}_{i.}))$ as the OLS (ordinary least square)  estimate of $u_i$, then the EB estimate is a weighted sum of 0 and the OLS estimate, and the weight is an increasing function of the sample size $n_i$. The final prediction is $$\hat{f}(\boldsymbol x_{ij})+\hat{u}_{i},$$ where $\hat{f}(\boldsymbol x_{ij})$ is the estimate of the fixed effect from linear regression or machine learning method like random forest. This can be easily extended to any level of data, say samples nested in cities and then regions and then countries. Other than the tree-based methods, there is a method based on SVM.
For random-forest-based method, you can try MixRF() in our R package MixRF on CRAN.
Comment: Could you explain why the intercept is allowed to vary with $i$, but $f$ has a single set of parameters for all $i$? Is it a simplifying assumption to avoid overfitting or make the problem tractable?
Comment: @user20160 Yes, we call $f$ part as fixed-effects part, and $u_i$ as the random effect. We can also have some random slopes like $\mathbf{x}_{ij}^{'} \mathbf{u}_{i},$ but should limit the number of random slopes because this would introduce a large number of variance components in the covariance matrix of the random-effects if it is unstructured.
Answer: Given that you only have two variables and straightforward nesting, I would echo the comments of others mentioning a hierarchical Bayes model. You mention a preference for tree-based methods, but is there a particular reason for this? With a minimal number of predictors, I find that linearity is often a valid assumption that works well, and any model mis-specification could easily be checked via residual plots.
If you did have a large number of predictors, the RF example based on the EM approach mentioned by @Randel would certainly be an option. One other option I haven't seen yet is to use model-based boosting (available via the mboost package in R). Essentially, this approach allows you to estimate the functional form of your fixed-effects using various base learners (linear and non-linear), and the random effects estimates are approximated using a ridge-based penalty for all levels in that particular factor. This paper is a pretty nice tutorial (random effects base learners are discussed on page 11).
I took a look at your sample data, but it looks like it only has the random effects variables of City, Region, and Country. In this case, it would only be useful to calculate the Empirical Bayes estimates for those factors, independent of any predictors. That might actually be a good exercise to start with in general, as maybe the higher levels (Country, for example), have minimal variance explained in the outcome, and so it probably wouldn't be worthwhile to add them in your model.
Comment: I see, agreed that trees can certainly be useful in that situation. In that case sticking with @Randel's suggestion would be a good option. Mboost also has a tree base learner as well which might prove useful in combination with the random effects base learners.
Comment: The *real* data I'm working with has a lot more than two variables which add a lot of real-world mess that isn't picked up by my simple example (e.g. non linearity, codependence, missing values, categorical values, etc.).  In my experience, tree based learners do the best job at handling all the real-world mess, which is why I lean towards using them. (There's a reason XGBoost wins almost all the structured data competitions on Kaggle.)

`mboost` sounds interesting - I'll give it a look.  Thanks
Answer: This is more of a comment or suggestion rather than an answer, but I think you ask an important question here. As someone who works exclusively with multilevel data, I can say that I have found very little about machine learning with multilevel data. However, Dan Martin, a recent PhD graduate in quantitative psychology at the University of Virginia, did his dissertation on the use of regression trees with multilevel data. Below is a link to an R package he wrote for some of these purposes:
https://github.com/dpmartin42/mleda/blob/master/README.md
Also, you can find his dissertation here:
http://dpmartin42.github.io/about.html
Comment: Thanks for the reference, Erik! One thing to keep in mind with my dissertation is that it only evaluated how forest methods perform "out of the box" when exposed to multi-level data structures. Methods that @Randel mentioned are discussed in Chapter 3 as a literature review, but the simulation study investigated mainly Breiman's original algorithm and conditional inference trees/forests.
Answer: You can use a mixed effect model that models the ID variables as random effects. By doing so, you allow for information pooling: you use both data from the global average and from the group averages, and the less data you have per group, the more weight is given to the global average. If you want to use a machine learning model for the fixed effects part, you can, for instance, use tree-boosting.
The GPBoost library with Python and R packages builds on LightGBM and allows for combining tree-boosting and mixed effects models. Simply speaking it is an extension of linear mixed effects models where the fixed-effects are learned using tree-boosting. See this blog post and Sigrist (2020) for further information. Below is some sample R code on how to run the analysis in your case. Note that currently the R package is not yet on CRAN (hopefully soon) but the the Python package is available on PyPI.
Disclaimer: I am the author of the GPBoost library.
library(gpboost)
train <- data.frame(CountryID=c(1,1,1,1, 2,2,2,2, 3,3,3,3),
                    RegionID=c(1,1,1,2, 3,3,4,4, 5,5,5,5),
                    CityID=c(1,1,2,3, 4,5,6,6, 7,7,7,8),
                    Age=c(23,48,62,63, 25,41,45,19, 37,41,31,50),
                    Gender=factor(c("M","F","M","F", "M","F","M","F", "F","F","F","M")),
                    Income=c(31,42,71,65, 50,51,101,38, 47,50,55,23))
# Prepare data
X <- as.matrix(cbind(Gender=train[,"Gender"],Age=train[,"Age"]))# fixed effects data
group_data <- train[,c("CountryID","RegionID","CityID")]# grouping data for random effects
y <- train[,c("Income")]# response variable
# Define a random effects model
gp_model <- GPModel(group_data = group_data)
# Run boosting algorithm (this will not give meaningfull results as the data is too small)
bst <- gpboost(data = X, label = y, gp_model = gp_model, verbose = -1,
               objective = "regression_l2",  nrounds=10, learning_rate=0.1)
# Show estimated variance parameters
summary(gp_model)

# A linear mixed effects model also has problems with this small data and some of the variances are 0
gp_model <- fitGPModel(group_data=group_data, y=y, X=cbind(Intercept = rep(1,length(y)),X))
summary(gp_model)
# Or the same thing using the lme4 package
library(lme4)
mod <- lmer(Income ~ Age + Gender + (1|CountryID) + (1|RegionID) + (1|CityID), data=train, REML=FALSE)
summary(mod)

Answer: The function RFcluster() from the gamclass package for R "adapts random forests to work (albeit clumsily and inefficiently) with clustered categorical outcome data".  The following example is from the help page for RFcluster:
 library(randomForest)
 library(gamclass)
 data(mlbench::Vowel)
 RFcluster(formula=Class ~., id = V1, data = Vowel, nfold = 15,
           tree=500, progress=TRUE, printit = TRUE, seed = 29)

This returns an OOB accuracy (where the "bags" are bags of speakers, not bags of individual speaker samples), that my machine gives as 0.57 .
Comment: To anyone interested in this, [`RFcluster`](https://github.com/cran/gamclass/blob/master/R/RFcluster.R) appears to be just a random forest cross-validated such that the `id` variable is taken into account when constructing the *folds*. It does not in any way take the `id` information into account in the fitting procedure.
Answer: R package glmertree allows for fitting decision trees to multilevel and longitudinal data (which would otherwise be modeled with a mixed-effects model).
It allows for specifying a random effects structure, and partitions the dataset into subgroups using level-I, or higher-level predictors.
For further reference, see the package vignette (tutorial): https://cran.r-project.org/web/packages/glmertree/vignettes/glmertree.pdf.
Fokkema, M., Smits, N., Zeileis, A., Hothorn, T., & Kelderman, H. (2018). Detecting treatment-subgroup interactions in clustered data with generalized linear mixed-effects model trees. Behavior research methods, 50(5), 2016-2034. https://doi.org/10.3758/s13428-017-0971-x
Fokkema, M., Edbrooke-Childs, J., & Wolpert, M. (2021). Generalized linear mixed-model (GLMM) trees: A flexible decision-tree method for multilevel and longitudinal data. Psychotherapy Research, 31(3), 329-341. https://doi.org/10.1080/10503307.2020.1785037
Answer: You may want to have a look at metboost: Miller PJ et al. metboost: Exploratory regression analysis with hierarchically clustered data.arXiv:1702.03994
Quote from the abstract: We propose an extension to boosted decision decision trees called metboost for hierarchically clustered data. It works by constraining the structure of each tree to be the same across groups, but allowing the terminal node means to differ. This allows predictors and split
points to lead to different predictions within each group, and approximates nonlinear group specific effects. Importantly, metboost remains computationally
feasible for thousands of observations and hundreds of predictors that may contain missing values.
It is implemented in the R package mvtboost


Topologically nilpotent elements of a linearly topologized ring

Question: In what follows all rings are commutative topological rings.
An element $x$ of a ring is called topologically nilpotent if $lim_{n\rightarrow \infty} x^n = 0$.
If a ring $A$ has a fundamental system of neighborhoods of $0$ consiting of ideals, $A$ is called linearly topologized.
An open ideal $I$ of a linearly topologized ring is called an ideal of definition if, for every neighborhood $V$ of $0$, there exists an integer $n > 0$ such that $I^n \subset V$.
The following proposition is Lemma 7.1.3 of Grothendieck's EGA I, Ch. 0.
Proposition
Suppose a linearly topologized ring $A$ has an ideal of definition $I$.
Suppose $x$ mod $I$ is nilpotent in $A/I$.
Then $x$ is topologically nilpotent.
The proof of EGA is as follows.
Let $V$ be a neighborhood of $0$.
There exists an integer $n > 0$ such that $I^n \subset V$.
If $x^m \in I$, $x^{mq} \in V$ for all integer $q \ge n$.
Hence $x$ is topologically nilpotent.
My question
Why $x$ is topologically nilpotent?
Answer: It seems like your problem is with the last sentence. This should be just the definition of convergence of the sequence $(x^n)_{n\in \mathbb{N}}$: It convergences against $0$ iff for every neighbourhood $V$ there exists $s_0$ such that for all $s\geq s_0$ we have $x^{s}\in V$. In fact the sentence before should better read $x^s\in I^n\subset V$ for all $s\geq mn$ (since $I$ is an ideal).
Comment: Indeed, the sentence before the last one is misleading.


outside vs inside __main__ variable definition in multiprocessing

Question: I have following code:
import multiprocessing
import time
import os

# WHEN SEMAPHORE IS DEFINED HERE THEN IT IT WORKS
semaphore = multiprocessing.Semaphore(1)

def producer(num, output):
  semaphore.acquire()
  time.sleep(1)
  element = "PROCESS: %d PID: %d PPID: %d" % (num, os.getpid(), os.getppid())
  print "WRITE -> " + element
  output.put(element)
  time.sleep(1)
  semaphore.release()

if __name__ == '__main__':
    """
    Reads elements as soon as they are are put inside queue
    """

    output    = multiprocessing.Manager().Queue()
    pool      = multiprocessing.Pool(4)
    lst       = range(40)

    # WHEN SEMAPHORE IS DEFINED HERE THEN IT DOES NOT WORKS
    # semaphore = multiprocessing.Semaphore(1)

    for i in lst:
        pool.apply_async(producer, (i, output))
        # print "%d Do not wait!" % i
        # res.get()

    counter = 0
    while True:
      try:
        print "READ  <- " + output.get_nowait()
        counter += 1
        if (counter == len(lst)):
          print "Break"
          break
      except:
        print "READ  <- NOTHING IN BUFFER"
        pass
      time.sleep(1)

This code is working as expected and it prints:
READ  <- NOTHING IN BUFFER
WRITE -> PROCESS: 0 PID: 15803 PPID: 15798
READ  <- NOTHING IN BUFFER
READ  <- PROCESS: 0 PID: 15803 PPID: 15798
READ  <- NOTHING IN BUFFER
WRITE -> PROCESS: 1 PID: 15806 PPID: 15798
READ  <- PROCESS: 1 PID: 15806 PPID: 15798
...

Then I have this version which is not working (It is basically the same as first one except the definition of semaphore is in another place):
import multiprocessing
import time
import os

# WHEN SEMAPHORE IS DEFINED HERE THEN IT IT WORKS
# semaphore = multiprocessing.Semaphore(1)

def producer(num, output):
  print hex(id(semaphore))
  semaphore.acquire()
  time.sleep(1)
  element = "PROCESS: %d PID: %d PPID: %d" % (num, os.getpid(), os.getppid())
  print "WRITE -> " + element
  output.put(element)
  time.sleep(1)
  semaphore.release()

if __name__ == '__main__':
    """
    Reads elements as soon as they are are put inside queue
    """

    output    = multiprocessing.Manager().Queue()
    pool      = multiprocessing.Pool(4)
    lst       = range(40)

    # WHEN SEMAPHORE IS DEFINED HERE THEN IT DOES NOT WORKS
    semaphore = multiprocessing.Semaphore(1)

    for i in lst:
        pool.apply_async(producer, (i, output))
        # print "%d Do not wait!" % i
        # res.get()

    counter = 0
    while True:
      try:
        print "READ  <- " + output.get_nowait()
        counter += 1
        if (counter == len(lst)):
          print "Break"
          break
      except:
        print "READ  <- NOTHING IN BUFFER"
        pass
      time.sleep(1)

This version prints:
READ  <- NOTHING IN BUFFER
READ  <- NOTHING IN BUFFER
READ  <- NOTHING IN BUFFER
READ  <- NOTHING IN BUFFER
READ  <- NOTHING IN BUFFER
READ  <- NOTHING IN BUFFER
READ  <- NOTHING IN BUFFER
...

It seems like if producer never writes anything to Queue. I've read somewhere that apply_sync does not print error messages. So I've changed pool.apply_async(producer, (i, output)) to pool.apply(producer, (i, output)) in second code, to see what is going on. It seems that semaphore is not defined, here is the output:
Traceback (most recent call last):
  File "glob_var_wrong.py", line 31, in <module>
    pool.apply(producer, (i, output))
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 244, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
NameError: global name 'semaphore' is not defined

However following code runs correctly and print 10 (value defined inside __main__):
global_var = 20

def print_global_var():
    print global_var

if __name__ == '__main__':
    global_var = 10
    print_global_var()

It seems that in this code global variable can be defined inside __main__ while in previous codes it is not possible. First I was assuming that variables defined inside __main__ are not shared between processes but it only affects semaphore and not output, pool, lst. Why is this happening?
Comment: It's nothing to do with `__main__`. You have to define semaphore _before_ you create the pools.  Your code will work if you just move the line `semaphore = ...` up before creating the pools.
Comment: You mean before `multiprocessing.Pool(4)` ? How is this related?
Comment: @mehtunguh thank you this did the trick. Can you please post your comment as answer and explain the background details (I still do not understand why this worked) so I can accept it. Thank you
Answer: When you create a new process using Multiprocessing.Process (used underneath the hood by Pool, it copies the local scope, pickles it, and sends it to a new process to evaluate.
Because you did not define the variable semaphore before calling Pool(4), the variable is undefined (in those OTHER processes where the code gets evaluated) and the function producer will throw an exception.
To see this, change the definition
def producer(num, output):
    print hex(id(semaphore))
    try:
        semaphore.acquire()
    except Exception as e:
        print e
    time.sleep(1)
    element = "PROCESS: %d PID: %d PPID: %d" % (num, os.getpid(), os.getppid())
    print "WRITE -> " + element
    output.put(element)
    time.sleep(1)
    semaphore.release()

and now your failing code will print out a bunch (40) of errors that looks like
global name 'semaphore' is not defined

This is why semaphore has to be defined BEFORE calling Pool
Answer: It is because you execute the code on Windows.
You will get the expected results on Linux.
That's the difference between fork and spawn.
Comment: I don't see how this is relevant.


How to group facets (Facet API) by day for date fields?

Question: How to group facets (Facet API) by day for date fields?
Currently I have a custom date field indexed by Search API Solr.

I want to group this values by day, for example:

Friday 26 june
-08:30
-08:45
Saturday 27 june
-08:30
Sunday 28 june
-08:30
Answer: Use the numeric_groups module. It is a sandbox project.


Crystal Report: Custom Order By

Question: I have a working crystal report and it called from a VB application. I want the report to be sorted according to specific column values. Consider the following example,
Model-Type     Model-Code     Quantity

A              xxx-123-xA1    1
A              xxx-123-xA2    1
B              xxx-123-xB1    1
B              xxx-123-xB2    1
C              xxx-123-xC1    1
C              xxx-123-xC2    1

Through crystal report, I can sort Model-Type in either ascending order or descending order. Is there any way I can sort it in a customized way like Model-Type "B" first then "A" and then "C" (for business purposes)?? I can do it through SQL using CASE in ORDER BY clause. But how can this be done in Crystal Report.
Many thanks
Answer: if you create a group on the column, in the group options box, on the common tab, select "in a specified order" from the dropdown box
alternatively create a formula like:
if {table.value} = 'B' then 1
else if {table.value} = 'A' then 2
else if {table.value} = 'C' then 3

etc... then you can sort on that formula field instead

Crystal Report: Custom Order By


Android - Listview Items position decreasing

Question: I'm building an App that receives all wifi networks arround my house.
All wifi networks are being listed inside a Listview.
Now I would like to open a new activity when some listview item is clicked. But before that I would like to get the SSID from the selected Item.
I'm using onItemClick to get the position from the item and is working fine. The problem that I'm getting is that when I click at some item, I get a different SSID from the current selected item, the values are being desplayed like that:
WIFI 1 - When I click here I get the SSID from WIFI 3

WIFI 2 - When I click here I get the SSID from WIFI 2

WIFI 3 - When I click here I get the SSID from WIFI 1

Instead of getting the SSID from the wifi 1 when I click at wifi one and etc.
Can you guys check what am I doing wrong?
My code:
    @Override
    public void onCreate(Bundle savedInstanceState)
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        textStatus = (TextView) findViewById(R.id.textView1);
        buttonScan = (Button) findViewById(R.id.button1);

        lv = (ListView)findViewById(R.id.text);

        lv.setClickable(true);

        wifi = (WifiManager) getSystemService(Context.WIFI_SERVICE);
        if (wifi.isWifiEnabled() == false)
        {
            Toast.makeText(getApplicationContext(), "wifi is disabled..making it enabled", Toast.LENGTH_LONG).show();
            wifi.setWifiEnabled(true);
        }
        this.adapter = new SimpleAdapter(MainActivity.this, arraylist, android.R.layout.simple_list_item_1, new String[] { ITEM_KEY }, new int[] { android.R.id.text1 });
        lv.setAdapter(this.adapter);

        registerReceiver(new BroadcastReceiver()
        {
            @Override
            public void onReceive(Context c, Intent intent)
            {
               results = wifi.getScanResults();
               size = results.size();

            }
        }, new IntentFilter(WifiManager.SCAN_RESULTS_AVAILABLE_ACTION));

    }

    @Override
    protected void onResume() {
        super.onResume();
        final Handler handler = new Handler();

        Runnable refresh = new Runnable() {
                @Override
                public void run() {
                   test();
                    handler.postDelayed(this, 8000);
                }
            };

        handler.postDelayed(refresh, 2000);
    }

    private void test() {
    {

        arraylist.clear();
        wifi.startScan();

        Toast.makeText(this, "Scanning...." + size, Toast.LENGTH_SHORT).show();
        try
        {
            size = size - 1;
            lv.setOnItemClickListener(new OnItemClickListener() {

                @Override
                public void onItemClick(AdapterView<?> arg0, View arg1, int arg2,
                        long arg3) {

                    // TODO Auto-generated method stub
                    Log.e("", "" + results.get(arg2).SSID);
                }

            });

            while (size >= 0)
            {
                LinkedHashMap<String, String> item = new LinkedHashMap<String, String>();

                String BSSID = results.get(size).BSSID;

                int frequency = results.get(size).frequency;

                if(!results.get(size).SSID.equals("FET")) {

                item.put(ITEM_KEY, results.get(size).SSID.concat(""));
                arraylist.add(item);

                }

                size--;
                adapter.notifyDataSetChanged();

            }

        }
        catch (Exception e)
        { }

    }
}

}

Answer: Reverse the order in which you add your list items to arraylist
try
{
    size = 0;
    ...
    while (size < results.size())
    {
      ...
      arraylist.add(item);
      ...
      size++;
      ...
    }
}

Comment: I didn't receive any wifi SSID after doing that.


Dropdown navigation menu appears besides the element

Question: I have been trying to create a dropdown menu which appears to be working with some success but the catch is...It presents the dropdown menu on the right side of the element instead of directly under it.
I suspect it has something to do with either position or padding but I wasn't able to figure it out.
Thanks for taking a look, I'm still new to css and it means a lot!
html

Home

Producten

Personaliseren

Over ons

Blog

Contact

FAQ

css
html {
    background: #936A4A;
}

nav {
    position: absolute;
    width: 100%;
    top: 0;
    left: 0;
    background: #F5F5F5;

}

ul {
    display: flex;
    flex-wrap: wrap;
    padding: 0;
}

ul li {
    width: 25%;
}

ul li > ul {
    display: none;
    flex-direction: column;
}

ul li:hover > ul {
    display: flex;
    flex-wrap: wrap;
}

ul li > ul li {
    width: 100%;
    height: 100%;
}

li {
    display: flex;
    flex: auto;

}

li a {
    color: #B85750;
    text-decoration: none;
}

.active {
    pointer-events: none
}

Answer: Please take a look at the following solution.

html {
          background: #fff;
        }

        .parent-container{
            position: relative;
            width:100%;
            height: 100%;
        }

        nav {
            background: #F5F5F5;
        }

        ul {
            list-style: none;
            margin: 0;
            padding-left: 0;
        }

        li {
            background: #936A4A;
            display: block;
            float: left;
            padding: 1rem;
            position: relative;
            text-decoration: none;
            transition-duration: 0.5s;
        }

        li a {
          color: #fff;
        }

        li:hover {
            cursor: pointer;
        }

        ul li ul {
          visibility: hidden;
          opacity: 0;
          min-width: 5rem;
          position: absolute;
          transition: all 0.5s ease;
          margin-top: 1rem;
          left: 0;
          display: none;
          background: lightblue;
        }

        ul li:hover > ul,
        ul li ul:hover {
          visibility: visible;
          opacity: 1;
          display: block;
        }

        ul li ul li {
          clear: both;
          width: 100%;
        }
<!DOCTYPE html>

        <link href="app.css" rel="stylesheet" type="text/css"/>

Home

Producten

Personaliseren

Over ons

Blog

Contact

FAQ/

Answer: Yes, you need to position the submenu absolutely in relation to the parent li.

html {
  background: #936A4A;
}

nav {
  position: absolute;
  width: 100%;
  top: 0;
  left: 0;
  background: #F5F5F5;
}

ul {
  display: flex;
  flex-wrap: wrap;
  padding: 0;
}

ul li {
  width: 25%;
  position: relative;
}

ul li>ul {
  position: absolute;
  left: 0;
  top: 100%;
  width: 100%;
  background: lightblue;
  /* display: none; */
  flex-direction: column;
}

ul li:hover>ul {
  display: flex;
  flex-wrap: wrap;
}

ul li>ul li {
  width: 100%;
}

li {
  display: flex;
  flex: auto;
}

li a {
  color: #B85750;
  text-decoration: none;
}

.active {
  pointer-events: none
}

Home

Producten

Personaliseren

Over ons

Blog

Contact

FAQ


Why the (') of opener and closer in HTML attribute is rendered as ('') in document.documentElement.outerHTML? and how to prevent it?

Question: Why the single quote (') of opener and closer in HTML attribute is rendered as double quote ('') in document.documentElement.outerHTML? and how to prevent it?
I have this element in my html page:

When I tried to get the entire html via document.documentElement.outerHTML, as a result, it becomes like this:

For the ", it doesn't matter at all. But for the opener and closer of single-quotes ('), why it's rendered as double-quote ('') instead of single-quote (') ?
Is there any solution to keep the quotes as it is?
Comment: hardly matters! http://stackoverflow.com/questions/2373074/single-vs-double-quotes-vs
Comment: @ajaykumar Yup, of course it's a matter for the using of quotation. What I mean by "doesn't matter at all" is the rendering of double-quote into `"`. It's not a problem because in the end, it will be rendered back into a double-quote again, but this is not happened for the single-quote unfortunately.
Answer: When the DOM is loaded, it's turned into DOM data structure that represents the logical relationships between all the elements and attributes. It doesn't retain the source code. So it just has the information that the class attribute contains the value {MO-CSS-CLASS value="UsedItems"}. The fact that you used single or double quotes in the original HTML is lost in this translation to the DOM.  When an element is serialized by reading its innerHTMl or outerHTML property, attribute values are always surrounded by double quotes.
Comment: Okay then.
I just want to tell my case, the `{MO-CSS-CLASS value="UsedItems"}` tag is like a syntax, so it will be translated into a particular value at the end. But because in the end, the element becomes `class="{MO-CSS-CLASS value="UsedItems"}"`, the system can't read it because it contains the same quote. Therefore is it possible to retrieve the "original" syntax which has the single-quote instead of the double-quote with javascript?
Comment: I don't understand what you're asking. The quotes used **around** the value don't affect the value of the attribute itself.
Comment: Why are you using `outerHTML` for this? Use `documentElement.className` to get the contents of the `class` attribute.


General Lower Bound of Power in Neyman-Pearson

Question: Let $X$ be an $\sigma$-finite space $(\mathcal{X}, F_{\mathcal{X}}, \nu)$ valued absolutely continuous random variable whose distribution is one of $P_0 = f_0(x)d\nu(x)$ or $P_1 = f_1(x)d\nu(x)$. We test the null hypothesis who says that $X$ has distribution $P_0$ against the alternative hypothesis who says that $X$ has distribution $P_1$. Given size $\alpha \in (0,1)$, the Neyman-Pearson lemma shows that the uniformly most powerful test $T : \mathcal{X} \to [0,1]$ representing rejection probability (more precisely, if we use randomised rule $\delta$ over the two decisions, $T(x) = \delta (x, \{1\})$ is that:
\begin{equation*}
T(X) = \begin{cases}
1  & f_1(X) > c f_0(X)\\
\gamma & f_1(X) = c f_0(X) \\
0 & f_1(X) < c f_0(X)
\end{cases}
\end{equation*}
Here $\gamma \in [0,1]$ and $c \in [0, \infty]$ are known constants chosen so that the size is $\alpha$. Typically, we choose them in the following way. Put, for every $t \in \mathbb{R}$ :
\begin{equation*}
\gamma(t) = P_{0} \{ f_1 > t f_0\} = P_0 (\{x \in \mathcal{X} \mid f_1(x) > t f_0(x)\})
\end{equation*}
Choose $c$ such that $\gamma(c) \leqslant \alpha \leqslant \gamma(c-)$. If $\gamma(c-) = \gamma(c)$, put $\gamma = 0$. Otherwise, put $\gamma = \frac{\alpha - \gamma(c)}{\gamma(c-)-\gamma(c)}$. This assures $\mathbb{E}_0[T(X)]=\alpha$.
I know that $T$ constructed in this way has its power always strictly greater than $\alpha$ (That is, $\mathbb{E}_1[T(X)] > \mathbb{E}_0[T(X)] = \alpha$. ) unless $P_0=P_1$ . But this lower bound is too weak. Is there any general lower bound on power $\mathbb{E}_1[T(X)]$ else than the one I mentionned?
Answer: I don't think you can come up with a tighter bound.
For example, if you have a sample $Y_1,\ldots,Y_n \sim \mathrm{N}(\mu,1)$ and you're testing $H_0:\mu=\mu_0$ versus $H_1:\mu=\mu_1$, you can easily see that the power of the MPT converges to $\alpha$ as $\mu_1$ approaches $\mu_0$.


Is the AIC of an ARIMA model comparable with the AIC of a regression with ARIMA errors?

Question: I already know that models from different "families" cannot be compared through AIC (or other information criteria such as BIC or HQIC), however, I'm not sure about the specific case of ARIMA and regression with ARIMA erros. Can you help me with this doubt?
Also if they aren't comparable, can you tell me what is the correct approach to compare the output of both models?
Thanks in advance.
Comment: I think what the cited textbook describes has to do with failing bullet point 3 (and probably bullet point 2) of my answer, which is a software issue rather than a fundamental problem. I will not elaborate on my view this time (that would take considerable effort), but you may always check out the relevant threads on CV if you wish to get some more information on the subject.
Comment: You are very welcome!
Comment: Why do you think that *models from different "families" cannot be compared through AIC*? There have been some heated debates on the question around here, and my take-away is different than yours.
Comment: @RichardHardy Honestly I don't think I have enough knowledge on the subject to be able to advocate that statement. That is just my current understanding of the use of the AIC from what I have read so far. Apparently, different "families" of models cannot be compared through AIC because their likelihood is computed in different ways (reference: https://otexts.com/fpp2/arima-ets.html). However, I am willing to read your view on why this is a misconception and, I may even agree with you. Thank you for your help.
Comment: OK, thank you Richard.
Answer: ARIMA and regression with ARIMA errors are comparable in terms of AIC as long as

the models are estimated by maximum likelihood,
the dependent variable is exactly the same and the likelihood is evaluated on exactly the same data points of the dependent variable in all competing models, and
the AIC (mainly, the likelihood) is calculated exactly (without scaling, without dropping constants) to ensure comparability across models. (Or the transformations made in AIC calculation are the same across models so that comparability is not compromised.)

The latter point addresses different software implementations which often make the calculated AIC values incomparable.
Alternatively, you can compare the predictive performance of your models using time series cross validation based on rolling or expanding windows.


one behavior about javascript regular expression

Question: On Firefox
"aaa".match(/a?/g)

produces
["a", "a", "a", ""]

just curious why we have a empty string at the end of the array ?
as if the RE engine is to examine every character, the string only has 3 character, why produce 4 matches ?
Comment: ok, so the engine will take 4 tries, a a a and "" .
Answer: The empty string is a valid match. The only reason it didn't match it before, is that ? is greedy. So the match "a" is preffered, but when the engine hits the end of the string the only thing it can match is the empty string.
The engine doesn't count how many matches it has to do. After a match it simply countinues from the next character. After the third a, what's left is the empty string:
>>> "".match(/a?/g)
[""]


What is the proper way to uninstall a module that adds a content type

Question: I heard that when uninstalling a module that is a content type, you should not use node_type_delete('my_module') in hook_uninstall(). Instead, you should let it be handled by Drupal that just disables the content type in the database. Is this right?
Apparently Drupal handles it in hook_node_info().
Wouldn't it be better to delete this node type in the database, or does Drupal handle uninstallation process in an other way?
I am using Drupal 7.
Answer: You should not delete content types in hook_disable(). Instead, you should make sure callbacks are left sane, and all data you will need on hook_enable() is properly archived and secured from modification by other modules. Good idea is to use drupal_set_message() to warn about content type that should no longer be used or displayed, but that's site admin's responsibility.
On hook_uninstall() you are supposed to left the database as if your module was never there. Thus, feel free to delete content types that was created using node_type_save(), if there are any. The ones created with hook_node_info() will be deleted automatically. Again, setting message about orphanted content is good idea, but it's site admin's responsibility not to uninstall module he is still using, after all.
If your hook_node_info() defined content type is not deleted, just disabled, you can decide to explicitly delete it, but you are responsible for cleaning all the mess it may cause, like orphaned content. You risk serious site failure if you don't.
Edited to incorporate information from Clive's comments under other answer
Comment: So you say I should not use node_type_delete in hook_uninstall (in file .install) if I have hook_node_info in .module? Because now I use hook_node_info and I create fields and instances in hook_install but I don't use node_type_save. And in hook_uninstall I take care of removing previously created fields and instances. But the thing is when I uninstall the module, the content type remains in node_type table but it is now disabled. Is this the normal behavior?
Comment: Yes it is - if you have any content of that content type, or if for some other reason Drupal "thinks" it may be unclean to remove it entirely. Of course you are allowed to force-delete it, but then you are responsible for all the cleaning that is needed.
Comment: Ok thank you. I currently remove all nodes of this type in my hook_uninstall but the content type remains in db.
Answer: There are two functions which can help you out of this problem with writing hook_uninstall():

node_type_delete is for deleting the content type
node_delete is for  deleting the associated nodes
Comment: node_type_delete seems to be in conflict with the way hook_node_info handles the node type. I get an error :


Machine learning task with feedback loop

Question: What are the available options if I want to perform a scoring task on a set of observations that: a) have a set of variables connected to them and, b) each round I get new information about the success of the latest round.
What I mean is, I start with a normal scoring task: Using the available training data I want to rank each observation in a new dataset. This part is straightforward and can naturally be performed by a range of different methods. However, the nature of problem is that I get feedback from this ranking when see how observations react. Specifically, when a number of observations are picked from those with the highest score, the feedback consists of these observation either reacting positively or negatively to them being picked. This information can in turn be used for the next round, which is performing a new scoring and picking the next observations with the highest scores.
What mathematical methods can be used to process this feedback information? How should/could the feedback be used in conjunction with the original (background) information? What R packages are there for these types of on-line machine learning problems?
Comment: I am not sure that reinforcement learning is appropriate here. Question for clarification: Do you can get feedback for an observation only once OR can you elect it for feedback multiple times ?
Comment: I get feedback from the chosen observations once for each round. If the observation is not scored to be among the n:th highest in a round, I get no feedback at all for those observation in that round. A observation can be chosen again (and give feedback multiple rounds) if it's scored high multiple times.
Answer: It sounds like you want reinforcement learning. I'm having a little trouble parsing the exact details of your specific problem, but perhaps it could be cast in the framework of a Multi-Armed Bandit problem?
Answer: alto is right, it sounds like a reinforcement learning setting. Although most RL examples involve a robot, RL can be applied to various other interactive problem domains.
You should read this free ebook by Sutton and Barto, which is the standard work for RL.
There have been loads of developments since the original writing of the book though. I recommend that once you can phrase your problem in RL formalities, you come back and ask what kind of algorithms can be used for that.


Pagination by compound key

Question: I am trying to optimize my query which runs on Postgresql version 13 or 14.
Let's say we have table with corresponding index:
CREATE TABLE journal
(
    account_id TEXT NOT NULL,
    event_at TEXT NOT NULL,
    id INTEGER PRIMARY KEY
);

CREATE INDEX journal_account_id_event_at_id_idx
    ON journal (account_id, event_at, id);

INSERT INTO journal VALUES ('A', '2023-01-01', 1);
INSERT INTO journal VALUES ('A', '2023-01-10', 50);
INSERT INTO journal VALUES ('A', '2023-01-30', 15);
INSERT INTO journal VALUES ('A', '2023-03-02', 28);
INSERT INTO journal VALUES ('A', '2023-03-05', 16);
INSERT INTO journal VALUES ('B', '2023-01-01', 101);
INSERT INTO journal VALUES ('B', '2023-01-01', 102);
INSERT INTO journal VALUES ('B', '2023-01-01', 103);
INSERT INTO journal VALUES ('C', '2022-12-01', 2);
INSERT INTO journal VALUES ('C', '2023-01-02', 10);
INSERT INTO journal VALUES ('C', '2023-01-30', 6);
INSERT INTO journal VALUES ('C', '2023-01-30', 20);
INSERT INTO journal VALUES ('C', '2023-02-02', 29);
INSERT INTO journal VALUES ('C', '2023-03-03', 31);

I need to select from said table via pagination. If we were to know account we need to select journal entries for, then we could do something like this:
Query #1:
SELECT *
FROM journal
WHERE
    account_id = 'C' -- account condition
    AND (event_at, id) > ('2022-12-01', 2) -- pagination condition
ORDER BY event_at, id
LIMIT 2;  -- page size for pagination

                                                                          QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.14..8.15 rows=1 width=68) (actual time=0.029..0.036 rows=2 loops=1)
   Output: account_id, event_at, id
   ->  Index Only Scan using journal_account_id_event_at_id_idx on public.journal  (cost=0.14..8.15 rows=1 width=68) (actual time=0.026..0.029 rows=2 loops=1)
         Output: account_id, event_at, id
         Index Cond: ((journal.account_id = 'C'::text) AND (ROW(journal.event_at, journal.id) > ROW('2022-12-01'::text, 2)))
         Heap Fetches: 2
 Planning Time: 0.204 ms
 Execution Time: 0.085 ms
(8 rows)

Which works perfectly fine, postgres uses index and, as you can see from the query plan, in accordance with limit, only 2 journal entries have been read (see 'Index Only Scan' node has rows=2).
However, if we need to select journal entries for multiple accounts and we don't know exact accounts in advance, postgres seem to select all entries and sort them in order to properly limit the output:
Query #2:
WITH accounts AS
(
    -- let's pretend here is some select from 'accounts' table
    SELECT *
    FROM unnest(ARRAY['A', 'C']::TEXT[]) AS account(id)
)
SELECT j.*
FROM journal j JOIN accounts a ON a.id = j.account_id -- account condition
WHERE (event_at, j.id) > ('2022-12-01', 2) -- pagination condition
ORDER BY event_at, j.id
LIMIT 2;  -- page size for pagination
                                                                                QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=12.36..12.37 rows=1 width=68) (actual time=0.120..0.134 rows=2 loops=1)
   Output: j.account_id, j.event_at, j.id
   ->  Sort  (cost=12.36..12.37 rows=1 width=68) (actual time=0.117..0.125 rows=2 loops=1)
         Output: j.account_id, j.event_at, j.id
         Sort Key: j.event_at, j.id
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Nested Loop  (cost=0.14..12.35 rows=1 width=68) (actual time=0.049..0.091 rows=10 loops=1)
               Output: j.account_id, j.event_at, j.id
               ->  Function Scan on pg_catalog.unnest account  (cost=0.00..0.02 rows=2 width=32) (actual time=0.008..0.011 rows=2 loops=1)
                     Output: account.id
                     Function Call: unnest('{A,C}'::text[])
               ->  Index Only Scan using journal_account_id_event_at_id_idx on public.journal j  (cost=0.14..6.15 rows=1 width=68) (actual time=0.020..0.025 rows=5 loops=2)
                     Output: j.account_id, j.event_at, j.id
                     Index Cond: ((j.account_id = account.id) AND (ROW(j.event_at, j.id) > ROW('2022-12-01'::text, 2)))
                     Heap Fetches: 10
 Planning Time: 0.307 ms
 Execution Time: 0.188 ms
(17 rows)

We now see from the query plan, that 'Nested loop' node has rows=10, which means that postgres read all entries that satisfy pagination condition only for them to be omitted later in 'Limit' node. My thought is, since we have compound index (account_id, event_at, id) for the journal table, postgres could have read limit journal entries for each account from the 'accounts' CTE and then merge them without even sorting, since entries alread sorted by (event_at, id), essentially doing something like this:
Query #3:
SELECT *
FROM (
  SELECT *
  FROM journal
  WHERE (account_id = 'A')
    AND (event_at, id) > ('2022-12-01', 2) -- pagination condition
  ORDER BY event_at, id
  LIMIT 2
) j1
UNION ALL
SELECT *
FROM (
  SELECT *
  FROM journal
  WHERE (account_id = 'C')
    AND (event_at, id) > ('2022-12-01', 2) -- pagination condition
  ORDER BY event_at, id
  LIMIT 2
) j2
ORDER BY event_at, id
LIMIT 2;
                                                                                QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.28..16.36 rows=2 width=68) (actual time=0.053..0.070 rows=2 loops=1)
   Output: journal.account_id, journal.event_at, journal.id
   ->  Merge Append  (cost=0.28..16.36 rows=2 width=68) (actual time=0.050..0.063 rows=2 loops=1)
         Sort Key: journal.event_at, journal.id
         ->  Limit  (cost=0.14..8.15 rows=1 width=68) (actual time=0.033..0.038 rows=2 loops=1)
               Output: journal.account_id, journal.event_at, journal.id
               ->  Index Only Scan using journal_account_id_event_at_id_idx on public.journal  (cost=0.14..8.15 rows=1 width=68) (actual time=0.031..0.033 rows=2 loops=1)
                     Output: journal.account_id, journal.event_at, journal.id
                     Index Cond: ((journal.account_id = 'A'::text) AND (ROW(journal.event_at, journal.id) > ROW('2022-12-01'::text, 2)))
                     Heap Fetches: 2
         ->  Limit  (cost=0.14..8.15 rows=1 width=68) (actual time=0.014..0.016 rows=1 loops=1)
               Output: journal_1.account_id, journal_1.event_at, journal_1.id
               ->  Index Only Scan using journal_account_id_event_at_id_idx on public.journal journal_1  (cost=0.14..8.15 rows=1 width=68) (actual time=0.012..0.013 rows=1 loops=1)
                     Output: journal_1.account_id, journal_1.event_at, journal_1.id
                     Index Cond: ((journal_1.account_id = 'C'::text) AND (ROW(journal_1.event_at, journal_1.id) > ROW('2022-12-01'::text, 2)))
                     Heap Fetches: 1
 Planning Time: 0.362 ms
 Execution Time: 0.132 ms
(18 rows)

Please help me understand how to write query like query #2 (for
multiple accounts) in such a way that it would have a query plan like query #3 (which takes advantage of already sorted data and read less data than query #2).
EDIT #1
Thanks to Laurenz Albe for mentioning "skip scan". I was able to find a possible solution  - using RECURSIVE CTE to emulate skip scan. I tried to use this aproach for my data and came up with this:
Query #4:
EXPLAIN (ANALYZE, VERBOSE)
WITH RECURSIVE accounts AS (
  -- lets pretent here is some select from 'accounts' table
  SELECT *
  FROM unnest(ARRAY['A', 'C']::TEXT[]) AS account(id)
  ORDER BY id
), journal_entries AS (
    (
      SELECT *
      FROM journal
      WHERE
        account_id = (SELECT MIN(id) FROM accounts)
        AND (event_at, id) > ('2022-12-01', 2) -- pagination condition
      ORDER BY event_at, id
      LIMIT 2  -- page size for pagination
    )
    UNION ALL
    SELECT chunk.*
    FROM (
      SELECT account_id FROM journal_entries LIMIT 1
    ) prev CROSS JOIN LATERAL (
      SELECT *
      FROM journal j
      WHERE
        j.account_id = (
          SELECT MIN(id)
          FROM accounts
          WHERE id > prev.account_id
          LIMIT 1
        )
        AND (event_at, id) > ('2022-12-01', 2) -- pagination condition
      ORDER BY event_at, id
      LIMIT 2 -- page size for pagination
    ) chunk
)
SELECT *
FROM journal_entries
ORDER BY event_at, id
LIMIT 2; -- page size for pagination;
                                                                                QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=91.42..91.43 rows=2 width=68) (actual time=0.198..0.230 rows=2 loops=1)
   Output: journal_entries.account_id, journal_entries.event_at, journal_entries.id
   CTE accounts
     ->  Sort  (cost=0.03..0.04 rows=2 width=32) (actual time=0.028..0.032 rows=2 loops=1)
           Output: account.id
           Sort Key: account.id
           Sort Method: quicksort  Memory: 25kB
           ->  Function Scan on pg_catalog.unnest account  (cost=0.00..0.02 rows=2 width=32) (actual time=0.009..0.011 rows=2 loops=1)
                 Output: account.id
                 Function Call: unnest('{A,C}'::text[])
   CTE journal_entries
     ->  Recursive Union  (cost=0.19..91.05 rows=11 width=68) (actual time=0.082..0.179 rows=4 loops=1)
           ->  Limit  (cost=0.19..8.21 rows=1 width=68) (actual time=0.080..0.090 rows=2 loops=1)
                 Output: journal.account_id, journal.event_at, journal.id
                 InitPlan 2 (returns $2)
                   ->  Aggregate  (cost=0.04..0.06 rows=1 width=32) (actual time=0.044..0.047 rows=1 loops=1)
                         Output: min(accounts.id)
                         ->  CTE Scan on accounts  (cost=0.00..0.04 rows=2 width=32) (actual time=0.030..0.036 rows=2 loops=1)
                               Output: accounts.id
                 ->  Index Only Scan using journal_account_id_event_at_id_idx on public.journal  (cost=0.14..8.15 rows=1 width=68) (actual time=0.078..0.080 rows=2 loops=1)
                       Output: journal.account_id, journal.event_at, journal.id
                       Index Cond: ((journal.account_id = $2) AND (ROW(journal.event_at, journal.id) > ROW('2022-12-01'::text, 2)))
                       Heap Fetches: 2
           ->  Nested Loop  (cost=0.19..8.26 rows=1 width=68) (actual time=0.027..0.037 rows=1 loops=2)
                 Output: j.account_id, j.event_at, j.id
                 ->  Limit  (cost=0.00..0.02 rows=1 width=32) (actual time=0.004..0.006 rows=1 loops=2)
                       Output: journal_entries_1.account_id
                       ->  WorkTable Scan on journal_entries journal_entries_1  (cost=0.00..0.20 rows=10 width=32) (actual time=0.002..0.002 rows=1 loops=2)
                             Output: journal_entries_1.account_id
                 ->  Limit  (cost=0.19..8.21 rows=1 width=68) (actual time=0.019..0.025 rows=1 loops=2)
                       Output: j.account_id, j.event_at, j.id
                       InitPlan 3 (returns $4)
                         ->  Limit  (cost=0.05..0.06 rows=1 width=32) (actual time=0.008..0.010 rows=1 loops=2)
                               Output: (min(accounts_1.id))
                               ->  Aggregate  (cost=0.05..0.06 rows=1 width=32) (actual time=0.005..0.007 rows=1 loops=2)
                                     Output: min(accounts_1.id)
                                     ->  CTE Scan on accounts accounts_1  (cost=0.00..0.04 rows=1 width=32) (actual time=0.002..0.003 rows=0 loops=2)
                                           Output: accounts_1.id
                                           Filter: (accounts_1.id > $3)
                                           Rows Removed by Filter: 2
                       ->  Index Only Scan using journal_account_id_event_at_id_idx on public.journal j  (cost=0.14..8.15 rows=1 width=68) (actual time=0.007..0.008 rows=1 loops=2)
                             Output: j.account_id, j.event_at, j.id
                             Index Cond: ((j.account_id = $4) AND (ROW(j.event_at, j.id) > ROW('2022-12-01'::text, 2)))
                             Heap Fetches: 2
   ->  Sort  (cost=0.33..0.36 rows=11 width=68) (actual time=0.196..0.199 rows=2 loops=1)
         Output: journal_entries.account_id, journal_entries.event_at, journal_entries.id
         Sort Key: journal_entries.event_at, journal_entries.id
         Sort Method: quicksort  Memory: 25kB
         ->  CTE Scan on journal_entries  (cost=0.00..0.22 rows=11 width=68) (actual time=0.091..0.175 rows=4 loops=1)
               Output: journal_entries.account_id, journal_entries.event_at, journal_entries.id
 Planning Time: 0.506 ms
 Execution Time: 0.422 ms
(52 rows)

As we can see, postgres still sorts journal entries instead of merging them, but there is less journal entires than in Query #2 - we only read limit entries for each account (see CTE Scan on journal_entries rows=4). I guess it's the best i can get out of this.
Answer: An index scan is only efficient if account_id is compared with =. Since PostgreSQL does not have an index "skip scan", you have to rewrite the query like you did.
Not everything that is possible is implemented in the PostgreSQL optimizer.
Comment: Thanks for you feedback! Searching for "skip scan" led me to [this possible solution](https://stackoverflow.com/questions/66893968/select-distinct-is-slower-than-expected-on-my-table-in-postgresql/66894500#66894500), i will try to use RECURSIVE CTE to solve my problem

Pagination by compound key


Will steel strings wear out my nickel frets ?

Question: I have bought some coated strings because I wanted to try that out. After my purchase I noticed that the package says the uncoated strings are stainless steel.
Since my frets are made out of nickel and nickel is softer than steel, this would mean I would see increased fret wear because of those strings.
Am I correct ? But more importantly, if this is correct how much faster approximately will my frets wear out ?
It would be great if people have had any experience with this.
EDIT: Reading some of the answers it seems some people think I am talking about changing nylon strings to "steel" strings.
To quote from wikipedia:
"whereas acoustic guitar and piano strings are usually wound with bronze" and
"Electric guitar strings are usually wound with nickel plated steel; pure nickel and stainless steel are also used"
http://en.wikipedia.org/wiki/Guitar_strings#String_construction
If you read the packaging of Ernie Balls, Martin, DR, etc. you often see them advertise their strings as "nickel", "bronze", "phosphor bronze".
Comment: No, the set in question is an Elixir Nanoweb string set. Only the wound strings are coated. The unwound strings are uncoated (this seems to be the case for both their Nanoweb and Polyweb stes). These unwound uncoated strings are made out of steel to "resist tone-deadening corrosion" in the words of the manufacturer.
Comment: Do you mean that the strings are stainless underneath the coating?
Answer: Frets wear.  It's inevitable with any steel string guitar; the unwound strings are technically made of "music wire" which is an industry-standard alloy.
The degree of wear is dependent on how hard you press  the strings down, whether you bend a lot, etc.
It's pretty typical to see older instruments with the "first position" frets rather badly grooved.  Any luthier can rather easily replace worn frets.
Answer: Stainless steel is a harder metal than your typical fret wire, which can be constructed of nickel/silver, plain steel, or more exotic alloys. The most commonly used is the nickel/silver alloy. Don't be fooled by the name though, nickel/silver is actually an alloy of nickel and brass. You can get some hard scientific data (pun intended) if you look up the Mohs hardness values of nickel (4.0) along with brass(5.5-6.0) and compare it to that of pure stainless steel (5.5-6.0 depending on the grade). The Mohs hardness scale describes a material's resistance toward being scratched by a harder material, which is a pretty solid metric for our purposes here.
So, given that the hardness of stainless steel is a half a point to one and a half points higher than that of typical nickel/silver fret wire given that you average the hardness numbers for brass and silver, then you will likely see more wear on your frets. This wear, however, will be proportional to how much you bend and how hard you fret.
Disclaimer: A more precise description would likely come from comparing the Vickers hardness numbers for nickel/silver to that of the grade of stainless steel that the strings are made of. But, YMMV because each manufacturer likely has different values for the fret wire they use. It's unfortunately an guestimation game and I definitely fudged the numbers a little bit above, but the general concept is solid.
Answer: What were your previous strings made out of?  If they weren't steel, you shouldn't be putting steel strings on instead, coated or not.  Steel and nylon string guitars aren't interchangeable.
Comment: My previous strings were Ernie Ball. According to their website the wound strings have steel cores with an external nickel plated winding, while the unwound strings are made of a tin-plated, steel alloy.
Answer: Ok, so I did some research and found part of my answer.
Normally for acoustic and electric guitars, the wound strings are composed of a steel core on which a wire a wound. This winding is either made of; nickel, bronze, steel or various other metals.
Nickel being often seen on sets for electric guitars and bronze often seen on sets for acoustic guitars.
As for unwound strings, they are made from a steel alloy and sometimes plated with another material or alloy. In the case of Ernie Ball Slinky (a popular string brand) the unwound strings are made from a steel allow plated with a tin allow.
When you buy a set of Elixir coated strings, the only string which are coated are the wound strings. The coating is there primarily to prevent gunk from getting in between the windings. This is why the unwound strings aren't coated.
Elixir also advertises that their uncoated unwound strings on their coated set (remember only the wound ones are coated) are made from stainless steel.
Before reading up on the subject I thought that since Nickel was written on the packaging of my old strings there was probably nickel in all the strings in a pack of those strings. In fact, the word nickel on the packaging refers only to the winding of the wound strings of the pack.
I was therefore thinking my current situation as having all 6 strings made out of nickel playing on nickel frets, which would mean both material already have about the same hardness. I was therefore thinking I was going from nickel strings to steel strings on the unwound ones.
This is not correct as my research as shown me.
There is still some differences between the unwound strings from Elixir compared to other brands so I may see faster fret wear. I will have to try them for myself and see.
The only part of my answer I am missing is if someone could has some experience changing if the slight change in materials on the unwound strings has some noticeable effect.
Answer: Regarding fret wear and string materials on steel string guitars, nickel silver frets are softer than most string and winding materials, except pure nickel. When I take off a set of pure nickel wound strings, I see the "fret marks" worn into the areas where they contacted the frets. I seldom find wear marks on the frets themselves, just on the pure nickel wound strings. As for "nickel" electric strings, the are actual nickel plated steel. They have a thin coat of nickel on the outside, which will wear off as you play them and eventually expose the harder carbon steel wrap wire to the frets. Fortunately, the contact area with wound strings is relatively high, so the contact loads are distributed over a broader area (less contact stress), and they don't lead to excessive fret wear. Knowing this, I recommend changing nickel plated steel wound strings before all the nickel is worn off, for the sake of a clean tone and preserving the frets.
Compared to brass frets are harder than nickel silver frets. I have an old archtop guitar with brass frets, which never showed any strings wear marks after more than 50-plus years. I had to level and re-crown the frets for other age related reasons, and found during the filing processes that brass is much harder and tougher than nickel silver.
Answer: To begin with, it depends on what type of guitar you have. If it's a classical guitar, you shouldn't put steel strings!
Also, it's not only the material of the strings that make frets wear out, but also the the gauge (diameter).
http://en.wikipedia.org/wiki/Strings_(music)#Gauge
Comment: It's an electric. I know the difference between classical and acoustic guitar and my question could also apply to an acoustic. The question stems from the fact that the unwound strings from the coated set seem to be made of a different material than regular unwound strings.
Answer: From my experience I found that strings labeled as steel have a more metallic sound than those that are labeled nickle plated. I used Elixir for a while and noticed the frets on the elixir guitars were wearing out faster. I switched to DR Black Beauties (which are all coated) and have found no unusual wear. While it is easy for a Luthier to put new frets on. In my area they are hard to find due to the fact that most of them were put out of business by corporate music stores, which have technicians who for the most part do not know what they are doing.


Jquery window.open em "success" no ajax está sendo bloqueado

Question: Gostaria que ao salvar um input e atualizar o div com a nova informação, na mensagem de retorno "success" do ajax, fizesse também a impressão do div. Exemplo
success :  function(response){
 $("#containerResultado").html(response);
 var divToPrint = document.getElementById('divresultado');
            var popupWin = window.open('', '_blank', 'width=800,height=600');
            popupWin.document.open();
            popupWin.document.write('' + divToPrint.innerHTML + '');
            popupWin.document.close();
}

Answer: Geralmente os navegadores bloqueiam o método window.open quando não é uma ação direta do usuário. Uma solução seria abrir a janela antes de chamar o AJAX e após a resposta do AJAX manipulá-la, seria algo parecido com isso:
var popupWin = window.open('', '_blank', 'width=800,height=600');

$.ajax({
    type: "...",
    url: ...,
    data: ...,
    contentType: "...",
    dataType: "...",
    success: function(response){
      $("#containerResultado").html(response);
      var divToPrint = document.getElementById('divresultado');
      popupWin.document.open();
      popupWin.document.write('' +
      divToPrint.innerHTML + '');
      popupWin.document.close();
},
    error: function (response) {
      popupWin.close();
    }
});

Comment: Obrigado pelo retorno. Tive a ideia de fazer com iframe que para o meu caso funciona até melhor:

`var newWin = window.frames["printf"];
newWin.document.write('dddd');
newWin.document.close();`


cywgin x86_64: where is diff?

Question: I did a recent cygwin 64 installation.
I cannot find the regular diff tool in either /usr/bin (as answered here), and when I try to reinstall Cygwin and search on diff, all I get are things like:
diffstat
diffutils
bsdiff
colordiff
  etc

No plain diff(1) utility.
Now it turns out I have a CM tool called Serena Dimensions installed, and so when I do a:
 which diff

I get:
 /cygdrive/c/Program Files (x86)/Serena/Dimensions 10.1/CM/prog/diff

I figured I just need to reference the "unix diff" via it's complete path, easy solution ;-)   But where is it!?
Thanks. (please see last comment, thanks)
Comment: Didn't you ask essentially the same question yesterday: http://stackoverflow.com/questions/32238796/cygwin-or-linux-what-is-the-directory-location-of-diff/32238844#32238844
Comment: I couldn't find it where you said, and it wasn't obvious to me from the running the cygwin install 64 setup (as I explained in the question).
Comment: Maybe you want to give [Babun](https://babun.github.io/) a try. It is a Cygwin wrapper project that has a lot of preconfigured stuff in it and is even easier to install than the raw Cygwin itself. I don't know if it has the diffutils preinstalled (I'm curently on a mac), but if not there is even a little package manager (called `pact`) in Babun so you could easily install it with `pact install diffutils`.
Comment: Amazing how much folks hate a simple question on this site (per down votes).   The answer is up-voted twice, but the question down-voted 3 times.  Is the question so unclear?  The guy answering got it in one quick shot.  If you agree, please up-vote the question at least to zero. THANK YOU.
Answer: The diff command is provided by the diffutils package. Install it via setup-x86_64.exe.
Comment: Thank you. Amazing how much folks hate a simple question on this site (per down votes).  They must have a lot of self-fear/hatred for where they see themselves not-knowing/ignorant.

cywgin x86_64: where is diff?


Estimating standard deviation

Question: I read in a book that we can estimate the distribution of the number of heads we get if we flip a coin 100 times(so doing 100 flip sessions multiple times and taking the distribution of the number of heads) with the normal distribution (50,5), where 5 is the standard deviation. But how do we approximate the standard deviation?(i.e. why is it 5 when the number of flips is 100, how are these numbers related)
Comment: If the coin is fair, the distribution is binomial, $n=100$, $p=1/2$. It is a standard result that the variance is $np(1-p)$.
Comment: The easiest way (but it involves some theory) is that the variance of a sum of independent random variables is the sum of the variances. Imagine doing an experiment independently $n$ times, with probability of success $p$ each time. Let $X_i=1$ if we have success on the $i$-th trial, and $0$ otherwise. Then the number $Y$ of successes is $X_1+\cdots +X_n$. The variance of $Y$ is therefore the sum of the variances of the $X_i$, and it is not hard to compute the variance of $X_i$ from the definition of variance. Each $X_i$ has variance $p(1-p)$. There are many other ways.
Comment: @AndréNicolas but how do we arrive at that?
Answer: This is not a simple result that one can just arrive at by looking at it. If you want a proof, take a look at this pdf.

Estimating standard deviation


What does [,...n] mean in T-SQL BNF?

Question: If you look at this page http://msdn.microsoft.com/en-us/library/ms189499 What does the [,...n] mean? I have my suspicions that it's just Microsoft's alternative to using the sequence { } symbols.
Answer: That nice big link right above the Syntax block on those MSDN pages lays out their "conventions." For [,...n], it has this to say:

Indicates the preceding item can be repeated n number of times. The
  occurrences are separated by commas.
Comment: To be fair, it'd be far more obvious (at least to me) if that link were *below* the syntax block, or at least below the "Syntax" heading, so it was actually part of that section.
Answer: It means an optional number of N repetitions.
Square Brackets denote optional parts of the statement. So for example, you can have a SELECT statement without an ORDER BY clause.

What does [,...n] mean in T-SQL BNF?


Why does ckeditor_config make it impossible to update Composer packages?

Question: I've got a Drupal 9.4 site that I've upgraded my database stack (XAMPP dev stack) to the latest version that is using php 8.1, but now have package incompatibilities showing up when I try to run "composer udpdate". Even trying to update with all dependencies fails.
My site runs fine, however I can no longer update anything in composer without fixing whatever dependency issues there are. This is what it tells me are the problems:
Problem 1
- zendframework/zend-feed[dev-master, dev-develop, 2.11.0, ..., 2.13.x-dev] require php ^5.6 || ^7.0 -> your php version (8.1.6) does not satisfy that requirement.
- laminas/laminas-feed[2.12.0, ..., 2.12.x-dev] require php ^5.6 || ^7.0 -> your php version (8.1.6) does not satisfy that requirement.
- symfony/psr-http-message-bridge[v1.1.0, ..., 1.1.x-dev] require php ^5.3.3 || ^7.0 -> your php version (8.1.6) does not satisfy that requirement.
- symfony-cmf/routing[1.4.0-RC1, ..., 1.4.x-dev] require php ^5.3.9|^7.0 -> your php version (8.1.6) does not satisfy that requirement.
- symfony/psr-http-message-bridge[v1.2.0, ..., v1.3.0] require php ^7.1 -> your php version (8.1.6) does not satisfy that requirement.
- laminas/laminas-feed[2.13.0, ..., 2.14.x-dev] require php ^7.3 || ~8.0.0 -> your php version (8.1.6) does not satisfy that requirement.
- drupal/core[8.9.11, ..., 8.9.x-dev] require php ^7.0.8 -> your php version (8.1.6) does not satisfy that requirement.
- drupal/ckeditor_config[3.0.0-beta1, ..., 3.x-dev] require drupal/ckeditor ^8.5 -> satisfiable by drupal/core[8.5.0-alpha1, ..., 8.9.x-dev].
- drupal/core[8.5.0-alpha1, ..., 8.7.7] require symfony-cmf/routing ^1.4 -> satisfiable by symfony-cmf/routing[1.4.0-RC1, ..., 1.4.x-dev].
- drupal/core[8.7.8, ..., 8.9.10] require symfony/psr-http-message-bridge ^1.1.2 -> satisfiable by symfony/psr-http-message-bridge[v1.1.2, 1.1.x-dev, v1.2.0, v1.3.0].
- drupal/core[8.8.0-alpha1, ..., 8.8.x-dev] require zendframework/zend-feed ^2.12 -> satisfiable by laminas/laminas-feed[2.12.0, ..., 2.19.x-dev], zendframework/zend-feed[2.12.0, 2.12.x-dev, 2.13.x-dev].
- Root composer.json requires drupal/ckeditor_config ^3.0 -> satisfiable by drupal/ckeditor_config[3.0.0-beta1, 3.0.0, 3.1.0, 3.x-dev].

My composer.json file looks like this:
    {
    "name": "drupal/recommended-project",
    "description": "Project template for Drupal 8 projects with a relocated document root",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "homepage": "https://www.drupal.org/project/drupal",
    "support": {
        "docs": "https://www.drupal.org/docs/user_guide/en/index.html",
        "chat": "https://www.drupal.org/node/314178"
    },
    "repositories": [
        {
            "type": "path",
            "url": "web/modules/custom/*"
        },
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "^1.7",
        "drupal/acl": "^1.0@beta",
        "drupal/admin_toolbar": "^3.1",
        "drupal/ckeditor_config": "^3.0",
        "drupal/config_inspector": "^1.3",
        "drupal/console": "^1.9",
        "drupal/content_access": "^1.0@alpha",
        "drupal/content_browser": "^1.0",
        "drupal/core-composer-scaffold": "^9.4",
        "drupal/core-project-message": "^9.4",
        "drupal/core-recommended": "^9.4",
        "drupal/csp": "^1.15",
        "drupal/d8w3css": "^2.0",
        "drupal/devel": "^5.0.0-beta1",
        "drupal/devel_entity_updates": "^4.0",
        "drupal/domain": "^1.0@beta",
        "drupal/domain_group": "^2.0",
        "drupal/domain_site_settings": "^1.3",
        "drupal/dropzonejs": "^2.4",
        "drupal/dynamic_entity_reference": "^1.12",
        "drupal/editor_advanced_link": "^1.8",
        "drupal/entity_browser": "^2.5",
        "drupal/entity_embed": "^1.1",
        "drupal/fancy_file_delete": "^2.0",
        "drupal/file_delete": "^1.0",
        "drupal/filefield_paths": "^1.0@beta",
        "drupal/formblock": "^2.0@beta",
        "drupal/gdoc_field": "^1.1",
        "drupal/group": "^1.5",
        "drupal/groupmedia": "^2.0@alpha",
        "drupal/ief_table_view_mode": "^2.2",
        "drupal/inline_entity_form": "^1.0@RC",
        "drupal/item_bucket": "*",
        "drupal/jsonapi_extras": "^3.19",
        "drupal/jsonapi_include": "^1.4",
        "drupal/libraries": "^3.0@beta",
        "drupal/media_bulk_upload": "^3.0",
        "drupal/media_directories": "^2.0",
        "drupal/media_entity_instagram": "^3.0",
        "drupal/multiple_fields_remove_button": "^1.0@alpha",
        "drupal/pathauto": "^1.8",
        "drupal/pdf": "^1.1",
        "drupal/profile": "^1.2",
        "drupal/rabbit_hole": "^1.0@beta",
        "drupal/responsive_menus": "^1.x-dev",
        "drupal/restui": "^1.20",
        "drupal/rules": "^3.0@alpha",
        "drupal/showcase_lite": "^1.3",
        "drupal/taxonomy_manager": "^2.0",
        "drupal/textarea_widget_for_text": "^1.2",
        "drupal/video_embed_spotify": "^1.5",
        "drupal/weight": "^3.3",
        "drush/drush": "^11",
        "kint-php/kint": "^4.2"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true,
        "allow-plugins": {
            "composer/installers": true,
            "drupal/console-extend-plugin": true,
            "cweagans/composer-patches": true,
            "drupal/core-composer-scaffold": true,
            "drupal/core-project-message": true
        }
    },
    "extra": {
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": [
                "type:drupal-core"
            ],
            "web/libraries/{$name}": [
                "type:drupal-library"
            ],
            "web/modules/contrib/{$name}": [
                "type:drupal-module"
            ],
            "web/profiles/contrib/{$name}": [
                "type:drupal-profile"
            ],
            "web/themes/contrib/{$name}": [
                "type:drupal-theme"
            ],
            "drush/Commands/contrib/{$name}": [
                "type:drupal-drush"
            ],
            "web/modules/custom/{$name}": [
                "type:drupal-custom-module"
            ],
            "web/themes/custom/{$name}": [
                "type:drupal-custom-theme"
            ]
        },
        "drupal-core-project-message": {
            "include-keys": [
                "homepage",
                "support"
            ],
            "post-create-project-cmd-message": [
                "<bg=blue;fg=white>                                                         </>",
                "<bg=blue;fg=white>  Congratulations, you've installed the Drupal codebase  </>",
                "<bg=blue;fg=white>  from the drupal/recommended-project template!          </>",
                "<bg=blue;fg=white>                                                         </>",
                "",
                "<bg=yellow;fg=black>Next steps</>:",
                "  * Install the site: https://www.drupal.org/docs/8/install",
                "  * Read the user guide: https://www.drupal.org/docs/user_guide/en/index.html",
                "  * Get support: https://www.drupal.org/support",
                "  * Get involved with the Drupal community:",
                "      https://www.drupal.org/getting-involved",
                "  * Remove the plugin that prints this message:",
                "      composer remove drupal/core-project-message"
            ]
        }
    }
}

I've tried individually updating each listed dependency, (ie composer update zendframework/zend-feed -W), etc, and they seem to run, but no updates are found and nothing is changing. How can I correct these dependency issues?
Comment: Please tell us the command you typed which resulted in the `Problem 1` output above. Much of the output is `drupal/core[8.9.11, ..., 8.9.x-dev] require php ^7.0.8 -> your php version (8.1.6) does not satisfy that requirement.`, which is obvious.
Comment: This is the result of running "$ composer update",  with or without the -W (--with-all-dependencies) flag. Clearly it doesn't like the newer php, but I thought Drupal 9 is compatible with PHP 8+ so its surprising there is an issue or not an update to make everything compatible with Drupal 9, compatible with php8.
Comment: How did this site get into this situation? The `composer.json` you pasted above is not a valid JSON file so I can't test. Drupal 9 is fully compatible with PHP 8 but that doesn't mean the specific set of dependencies in that file and in the lockfile are compatible with PHP 8.
Comment: it's valid, the question formatting code/paragraph cut off the last "}". I'll edit the question.
Answer: The problem package is drupal/ckeditor_config. This is a packaging bug that is being discussed in these issues:

https://www.drupal.org/project/ckeditor_config/issues/3309026
https://www.drupal.org/project/project_composer/issues/3309665
Comment: Yes, this is correct. It has affected many downstream modules that depend on this.
Comment: Yeah, the "drupal/ckeditor_config": "^3.0" was the issue, thanks so much!


What is the correct way to load a JavaScript library?

Question: I am trying to display form data as a line chart.  I am running Drupal 9 in Lando.  I am using Charts 5.0.x w/ the Highcharts submodule. I installed the Highcharts JS library using composer. This is the relevant code.
.libraries.yml
highcharts:
  js:
    /libraries/highcharts/highcharts.js: {  }
  dependencies:
    - core/jquery

.module
function stats_degrees_theme() {
  return [
    'mychart_template' => [
      'variables' => ['chart_var' => NULL],
    ],
  ];
}

.routing.yml
stats_degrees.render_form:
  path: '/statistics/stats-degrees'
  defaults:
    _title: 'Engineering and Physical Science Degrees Earned by Members of Underrepresented Groups'
    _form: '\Drupal\stats_degrees\Form\StatsDegreesForm'
  requirements:
      _permission: 'access content'

.html.twig
{# stats_degrees/templates/mychart-template.html.twig #}
{{ attach_library('stats_degrees/highcharts') }}

  {{ chart_var | raw }}

Submission handler
public function submitForm(array &$form, FormStateInterface $form_state) {
   $form_state->setRebuild();
  }

Form builder
 if ($form_state->isRebuilding()) {
   // get the form data and build the chart
    // get twig template
      $form['#theme'] = 'mychart_template';
      $form['#attached'] = array(
        'library' => array('stats_degrees/highcharts')
      );
      $form['#chart_var'] = $rendered_chart;
  }

The template is rendered without chart. When I look at the HTML markup, I see the rendered chart code in chart_var and this is in the console.

Uncaught ReferenceError: Highcharts is not defined at js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:4396
js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:128 Uncaught TypeError: $(...).highcharts is not a function at HTMLDivElement.
[anonymous] (js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:4373
at Function.each (js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:2)
at S.fn.init.each (js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:2)
at Object.attach (js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:4370)
at js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:139
at Array.forEach ()
at Object.Drupal.attachBehaviors (js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:136)
at js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:365
at HTMLDocument.listener (js_fLkad5FjTCYgrjWIVyhfUxVUTz4-TXopB2EJOzHz820.js:353)

When I drill down into the error I see it is failing here.
(Highcharts)})(function(w){(function(b){function r(b,a){this.init(b,a)} var v=b.CenteredSeriesMixin,u=b.each,n=b.extend,q=b.merge,g=b.splat;n(r.prototype,{coll:"pane",init:function(b,a){this.chart=a;this.background=[];a.pane.push(this);this.setOptions(b)},setOptions:function(b){this.options=q(this.defaultOptions,this.chart.angular?{background:{}}:void 0,b)},render:function(){var b=this.options,a=this.options.background,c=this.chart.renderer;

Am I loading the Highcharts library correctly?
Comment: Have you cleared the registry? Does `highcharts.js` show up in the HTML source? Is the path to the `highcharts.js` correct in the HTML source?
Comment: Yes definitely check the path, the JS files are in a /code folder by default so the path might be wrong. You don't need to register the library yourself BTW, the submodule does that (see charts_highcharts.libraries.yml). Assuming the path to the JS is right and being loaded, I think the next question would be what does `$rendered_chart` contain?
Comment: I have cleared the registry.
highcharts.js is under /libraries/highcharts/.  That's where composer installed it.
When I view source the only reference to highcharts that I see it near the bottom under


Logic Notation question (specifically about logical equivalences)

Question:
$$\text{Equivalence}$$
$p \land T \equiv p\tag{Identity law 1}$
$$p\lor F \equiv p\tag {Identity law 2}$$
$$p\lor T \equiv T\tag{Domination law 1}$$
$$p\land F \equiv F\tag{Domination law 2}$$
So, in the above image, where the T and F are, I assume these represent something like:
T = "any true proposition"
F = "any false proposition"
And so the first row in English would be "If P and any true proposition is true then this is logically equivalent to P" is that right? I could imagine it being something a bit different.
Thanks in advance!
Answer: You're basically right, though I would very slightly change the wording:
Let $T$ be a true proposition. Let $F$ be a false proposition.
$p \land T \equiv p$: We interpret this as:

The proposition "$p$ is true and $T$ is true" is equivalent to the proposition "$p$ is true."

$p \lor F \equiv p$: We interpret this as:

The proposition "$p$ is true or $F$ is false" is equivalent to the proposition "$p$ is true."

$p \lor T \equiv p$: We interpret this as:

The proposition "$p$ is true or $T$ is true" is equivalent to the proposition "$T$ is true."

$p \land F \equiv p$: We interpret this as:

The proposition "$p$ is true and $F$ is false" is equivalent to the proposition "$F$ is false."

To see why this works, write out the truth tables of each proposition and see that the equivalences do hold.
Comment: @papercuts Yes. Those mean the same thing. (If it was useful please upvote! It's how we recognize good answers.)
Comment: Would it be okay to replace "is equivalent to" with "if and only if"?
Answer:
Yes, $T$ = any proposition that is necessarily (or always) true (a tautology).
And $F$ = any proposition that is necessarily (or always) false (a contradiction).
The symbol $\;$"$\;\land\;$" denotes logical AND (conjunction).
The symbol $\;$"$\,\lor\,$" denotes logical OR (disjunction).
The symbol "$\;\equiv\,$" denotes "is logically equivalent to" or if you prefer, it denotes "if and only if".

*It might be helpful to review the truth-tables for the logical connectives $\land,\;\lor,\;\text{and}\;\equiv\;(\text{or}\;\iff)\;$ to understand why te following assertions must be true:

$$p \land T \equiv p$$
  $$p \lor F \equiv p$$
  $$p \lor T \equiv T$$
  $$p \land F \equiv F$$

With respect to your second question.
Yes, for the first identity, we have that $\; p \land T\;$ is logically equivalent to $\; p.\;$Put differently: $(p\,$ AND  $\,T)\;$ if and only if $\;p$.
Since $T$ represents a tautology (true no matter what), then the truth-value of $\;p \land T\;$ depends only on the truth-value of $\;p\;$:
When $p$ is false both sides of the equivalence are false, and when $p$ is true, both sides of the equivalence are true.
So yes,

$p \land T \equiv p$.

This also means $(p \land T \iff p):\quad$ ($p$ and $T$) if and only if $(p)$
Comment: how did this not get any votes? +1


Why do I need to prefix params with out or ref?

Question: If I'm calling a method that uses a out or ref parameter why do you need to mark the parameters with out or ref when calling the method? i.e.
DateTime.TryParse(myDate, out dateLogged)

I had thought this was something to reduce ambiguity when operator overloading but if I create two methods like so:
void DoStuff(int x, out int y)
{
    y = x + 1;
}

void DoStuff(int x, ref int y)
{
    y = x + 1;
}

Then Visual Studio reports "Cannot define overloaded method 'DoStuff' because it differs from another method only on ref and out", so I couldn't do that anyway.
EDIT:
After a bit more research it seems that I would need to specify out or ref when calling a method if it there were two overloaded methods declared like this:
    void DoStuff(int x, out int y)

    void DoStuff(int x, int y)

Comment: you really wouldn't want it to be implicit, its better this way. Just consider the potential for unexpected variable changes.
Comment: This also has the benifit of making passing by ref look awkward and therefore decreasing its popularity and therfore making code more maintainable. (IMO)
Answer: It's there so that you're explicitly consenting to the possibility that the method you're calling could modify the value of the variable itself. This is a C# language requirement; other languages like VB.NET do not require you to decorate the actual method call, just the method signature.
Comment: @Grand: Yep; IL does not distinguish between `ref` and `out`, which is why you can't create an overload that has `ref` and another that is just `out`, but you can have one that has a `ref`/`out` parameter and one that isn't.
Comment: indeed, this is one of the positive features of c#
Comment: Thanks Adam, it seems that overloading does also come into it. If I change the two method signatures in my example to:
        void DoStuff(int x, out int y)
        void DoStuff(int x, int y)
then they can be overloaded.
Answer: Ref and Out are really the same thing.  The difference is that the out keyword makes the compiler check that the variable is assigned before returning from the method.
EDIT
Sorry, misunderstood the question, I thought you were asking why c# wouldn't allow you to have 2 methods with the same signature, with only a difference in parameter modifiers.  See @Adam Robinson's answer; it's to ensure that the caller knows what may happen (or will happen in the case of out) to their variable after returning from the method.
Comment: He's asking why he needs to use `ref` or `out` when *calling* the function (as opposed to only needing to declare them as `ref` or `out` in the method signature), not what `ref` and `out` do.
Comment: Sorry, I've re-worded the question to remove that ambiguity.
Answer: C# makes you mark a parameter as ref / out as a warning / pointer to the programmer that something different is happening with these parameters and that the value passed may change.  For example in this case:
int myInt = 12;
string myString = Fizz(myInt);
// What is the value of myInt at this point?

I know looking at the above example that the value of myInt is 12, however this wouldn't be the case if out and ref parameters were implicit.
Answer: So the people reading your code know what you are doing, it makes it clear that the passed variable could change.
It has additional benefit of stopping you from doing it by accident.
Having maintained code on platforms that pass by ref implicitly, or even by default, I would say that typing 3 charachters is a small price to pay for the clarity (and possibly sanity) it provides.
Answer: Your case could be considered a hack. It's purpose is to allow a method to change the value of an argument.
For example:
void someMethod(ref i)
{
 i = 1;
}

int i = 0;
someMethod(ref i); //i now becomes 1, without the ref it would stay 0.

Comment: He's asking why he needs to use `ref` or `out` when calling the function (as opposed to only needing to declare them as `ref` or `out` in the method signature), not what `ref` and `out` do.
Comment: @TJHeuvel Sorry, my question didn't quite come out as clear as I meant it to!


How do I "kick-start" a jcl stream on the mainframe using FTP inside vb.net 2008

Question: The following is the code that uploades a bytearray into a file DSN on our mainframe.   It works very well.   What I want to do is upload the jcl which should then start to execute.  That's the part I am stuck on.   I used to be able to do it through WININET, but I want to get away from that and use the better FTP commands in vb.net
Public Shared Sub UploadToMainFrame( _
    ByVal ftpHost As String, _
    ByVal ftpMainframeDSN As String, _
    ByVal UserName As String, _
    ByVal Password As String, _
    ByVal DataToUpload As String)

    Dim ftpRequest As FtpWebRequest
    Dim ftpFullMainframePath = String.Format("ftp://{2}//'{3}'", ftpHost, ftpMainframeDSN)

    ftpRequest = WebRequest.Create(ftpFullMainframePath)
    ftpRequest.Credentials = New NetworkCredential(UserName, Password)
    ftpRequest.KeepAlive = True
    ftpRequest.UseBinary = False
    ftpRequest.Method = WebRequestMethods.Ftp.UploadFile

    ftpRequest.

    Dim byteArray() As Byte = StrToByteArray(DataToUpload)

    ftpRequest.ContentLength = byteArray.Length

    Dim ftpStream As Stream = ftpRequest.GetRequestStream()
    ftpStream.Write(byteArray, 0, byteArray.Length)
    ftpStream.Close()
    ftpStream = Nothing
    ftpRequest = Nothing

End Sub

Comment: That string.format line doesn't look right -- it references the 3nd and 4rd parameters ({2}/{3}), but only two are supplied...
Answer: If you want your uploaded file to be treated as a JCL job, it's a simple matter to put:
quote site filetype=jes

into the FTP commands.
This will direct JES to treat the incoming dataset as JCL and automatically submit it.
How you would achieve that with VB is not within my domain but the following works fine from cmd.exe:
ftp -n bigiron.com -s:bigiron.ftp

with bigiron.ftp containing:
user MYNAME MYPASSWORD
quote site filetype=jes
quote site jeslrecl=80
put commands.jcl
bye

If, for some reason, you can't do this from within VB, my next move would be to have a started task running under z/OS which monitored a specific data set and submitted any new members that appeared there.
I'd use the sentinel method to ensure you're not trying to submit half-uploaded jobs:

client uploads member into date dataset (e.g., "DATA.MYJOB").
client then uploads dummy member into sentinel dataset (e.g., "SENTINEL.MYJOB").
started task continuously looks for new members in sentinel dataset.
if found, submits the equivalent data member to JES and deletes both sentinel and data members.

We've actually used this method before when sysprogs weren't keen on allowing those foreign machines direct access to JES (despite passwords).
Update, after a bit of searching:
It doesn't look good for the "quote site" solution. This post from 2006 (but with updates well into 2009) seems to indicate MS won't allow "quote site" to be implemented since it would expose the inner workings of their code. I won't comment of the validity of that argument and their reasoning may have changed, but I can't see anything in the current docs that indicates they've added this feature.
So I guess you're stuck with an external FTP solution.


Dividing sets of services on a single server

Question: A group which I'm part of has grown and is about to move from using DigitalOcean VPSes to a single colocated server. At this point, we don't have a need to distribute our service very highly. This monolithic server will be running pretty much all of our systems, from our public-facing website to our mail service to custom daemons and control panels.
There has been concern, however, about the possibility of one service becoming compromised (we have been a target of intrusions in the past) and causing stability or a security breech in other parts of the server.
What we're looking to do is somehow "sandbox" certain services and processes, restricting the amount of resources (RAM, CPU) they're able to use. It's been suggested running each "set" of services on a different VM, but that seems clunky and difficult to maintain to me. I would think that Linux would have a more elegant way of handling this. Does anyone have any suggestions on the best way to accomplish this?
We're using CentOS 6.4 64 bit.
Comment: Good point, well made, must remember to read every word of new poster's questions :) we get so much crap from most new users - sorry
Comment: "It's been suggested running each set of services on a different VM, but that seems clunky and difficult to maintain to me." - I was asking if that was the best way, or if there was a better way.
Comment: Can't you just run these in separate VMs like everyone else does?
Answer: A project called Docker has recently gotten a lot of attention for attempting to solve this very problem.

Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, OpenStack clusters, public clouds and more.

This technology takes advantage of Linux cgroups and a few other things to build jail-like "containers" that can contain all of the necessary resources and configuration to run a specific service, which can be managed kind of like packages, in that they can be distributed, versioned, and used on a wide variety of environments. (This is a gross simplification.)
See What are the Main Features of Docker to learn more about how it actually works.
Comment: Wow, that looks incredibly awesome, thanks for the answer!
Answer: Traditionally you'd do this with a chroot jail but I strongly recommend putting them each in their own VM. Much better insulation from each other and you get better granularity when upgrading/troubleshooting/changing the services. Being able to restart/patch your web front end(s) with out taking the database server offline can be very nice, especially if you end up scaling out for redundancy.
The overhead of a CentOS box is quite small relative to most apps.


Execute Python with pygtk without terminal appear

Question: I have a little issue using pygtk, i have this project written in python, now the program works correctly, i have already added the string for the execution using terminal:
<code>#! /usr/bin/env python
</code>
Now i want to know how to exec this script hiding the terminal that pop out once you double click the file.
Anyone knows?
Comment: What OS are you interested in?
Comment: i am interested in Linux
Answer: Try saving it as a .pyc and executing that one.
Take a look at this and this.
Basically this:
<code>import py_compile
py_compile.compile('abc.py')
</code>
Comment: I'm don't have access to a linux box to test it right now but I'm pretty sure that has solved it before.
Comment: executing the .pyc pop up an alert window: "No Such program installed for the <> file"
Comment: It works! i choose the python interpreter to open the bytecode python file and now it run with a double click, Thanks!
Comment: @PhantomFav No prob :) Great to hear!!