Data visualisation for binary response data

Question: I'm seeking alternative ways of representing my data.
I have conducted an experiment, with two manipulated variables (v1 from 0 to 3, v2 from -3 to 0), and by factorial design it yields 4*4=16 stimuli. Participants give a 0/1 response for each stimulus. So for each individual I have a following response matrix (Table 1). After I collected data from multiple individuals, I can calculate the percentage of "1" response for each stimuli, leading to a summary results for all individuals (Table 2).
Participants are recruited from two countries. So there are two groups of people. They faced the same stimuli. My ultimate goal is to compare the different response pattern between the two groups.
Table 1. Individual response matrix
+-----+---+----+----+----+
|v1/v2| 0 | -1 | -2 | -3 |
+-----+---+----+----+----+
| 0   | 0 | 0  | 1  | 0  |
+-----+---+----+----+----+
| 1   | 0 | 0  | 1  | 0  |
+-----+---+----+----+----+
| 2   | 0 | 0  | 1  | 1  |
+-----+---+----+----+----+
| 3   | 0 | 1  | 1  | 1  |
+-----+---+----+----+----+

Table 2. Summary results (percentage of "1" responses)
+-----+---+----+----+----+
|v1/v2| 0 | -1 | -2 | -3 |
+-----+---+----+----+----+
| 0   |20%| 39%| 45%| 50%|
+-----+---+----+----+----+
| 1   |34%| 53%| 78%| 78%|
+-----+---+----+----+----+
| 2   |43%| 85%| 95%| 99%|
+-----+---+----+----+----+
| 3   |80%| 86%|100%|100%|
+-----+---+----+----+----+

In my story, v1-v2 is always a crucial predictor. For one group of participants, the absolute v2/v1 is also a crucial predictor. For another group, only v1-v2 matters. The idealised pattern is like follows:

My colleagues found these figures not very intuitive. So I'm seeking alternative ways of presenting the data. Here is some thoughts I have, but not entirely sure how to achieve this.

I'm thinking of a more 'direct' way of translating the summary table into graphs, like colormap (not sure if this is the right name). In this way I could use v1 and v2 as two axis (instead of v1-v2), and marking the value through color. Is this implementable in R?
Another way of revising my current figure is that, I keep using v1-v2 as x-axis, but not using v2 as grouping variable, but show the variance at each v1-v2 level, which also helps distinguish the two groups.
For individual responses (Table 1) there seems to be a boundary line that divides 0 and 1 (downright more likely to be 1, and upleft more likely to be 0), and this boundary line may be different for each individuals. The summary data loses this information on individuals. I wonder if I could estimate a boundary line for each individual (like a regression), and compare the distribution of this boundary line between two groups.

Or if you have other thoughts in presenting the data, I'll appreciate your help!
Comment: May be a 3D plot is your solution? v1 is x-axis, v2 is y-axis, and the entries of table 2 are heights on the z-axis.
Answer: Regarding question 1, here an example with your data implemented in R:
library(tidyverse)

df = data.frame(v1 = c(rep(c(0,-1,-2,-3),4)), v2 = c(rep(0,4), rep(1,4), rep(2,4), rep(3,4)),
                percentage = c(20, 39, 45, 50, 34, 53, 78, 78, 43, 85, 95, 99, 80, 86, 100, 100) )

p = ggplot(df)
p = p + geom_raster(aes(x = v1, y = v2, fill = percentage))
p = p + scale_fill_gradient(low = "#56B1F7", high = "#132B43")
p = print(p)

Comment: Thanks! It is probably the best I could get!


Is there the empty topological space?

Question:
May the empty set be a topological space?
Is the set $\{\emptyset\}$ its topology?
What about the empty metric space?
Comment: Does it satisfy the axioms of a topological space?
Comment: @Black Not including the empty topological space means that its not a complete or cocomplete category. We need the empty topological space, it's very important.
Comment: My professor defined a metric/topological space as a **nonempty** set equipped with ... .  I must therefore thank her for having  saved us from the present fruitless discussions!
Comment: @Black You have been made a considerable disservice, because you'll have to make caveats on the existence of coproducts of families of spaces, the existence of products will have to be unnaturally subordinated to the axiom of choice and Tychonov theorem will be way less elegant.
Answer:
Definitely yes and yes. In point of fact, it's the initial object of the category of topological spaces. It is compact, but as far as I know it is currently a matter of debate whether it should be considered connected or not.

I would say yes (like wikipedia does). It's homeomorphic to the empty topological space and it still is the initial object in all the categories I can think of putting on metric spaces (namely: the category with continuous maps, or the category of uniformly continuous maps, or the category of Lipschitz maps, or the category of non-expansive maps).
Comment: @NurlanAbiev It's just the empty function (the unique function $\emptyset \times\emptyset\rightarrow\mathbb{R}$, defined by "").
Comment: Thank You!  What should we mean by distance in the empty metric space?
Comment: Every continuous function $\emptyset\to\{0,1\}$ is constant. A space $X$ is disconnected if and only if there are nonempty open sets $A,B$ such that $A\cup B=X$; the empty space doesn't satisfy this property. These seem strong enough reasons to declare that the empty space is connected.
Comment: @egreg I think [there is no standard convention](https://math.stackexchange.com/questions/503442/empty-space-connected-if-so-then-what-are-its-components/503446) about whether the empty space is connected.
Comment: @egreg Some people have [arguments for the contrary](https://ncatlab.org/nlab/show/empty+space). I'm not a mathematician, so I don't quite manipulate the theory at a level where I can see the difference.

Is there the empty topological space?


IMCE module: list only image files

Question: I have a Drupal 7 with the IMCE module installed.
It works good but it shows me every files in my media directory.
I need to limit the listing to only image files, for example to only jpg, png and svg file.
Anybody know a way to do it?
Thanks a lot.
Answer: You can restrict by placing file extensions which you want to display in the permitted file extensions field. It will appear in the configuration of IMCE profiles. See below image for your reference.

IMCE module: list only image files


How to generate a rolling mean for a specific date range and location with pandas

Question: I have a large data set with names of stores, dates and profits.
My data set is not the most organized but I now have it in this df.
df
Store   Date           Profit
ABC     May 1 2018     234
XYZ     May 1 2018     410
AZY     May 1 2018     145
ABC     May 2 2018     234
XYZ     May 2 2018     410
AZY     May 2 2018     145

I proudly created a function to get each day into one df by itself until I realized it would be very time consuming to do one for each day.
def avg(n):
  return df.loc[df['Date'] == "May" + " " + str(n) + " " +str(2018)]

where n would be the date I want to get.  So that function get me just the dates I want.
What I really need is to have a way to get all dates I want in a list and to append them to a pd for each day. I tried doing this but did not work out.
def avg(n):
    dlist= []
    for i in n:
        dlist= df.loc[df['Date'] == "May" + " " + str(i) + " "  +str(2018)]
        dlist=pd.DataFrame(dlist)
        dlist.append(i)
        return dlist

df2=avg([21,23,24,25])

My goal there was to have all the dates of (21,23,24,25) for the May
into its own series of df.
But it was a total fail got this error

cannot concatenate object of type ""; only pd.Series, pd.DataFrame, and pd.Panel (deprecated) objs are valid

I am not sure if it's also possible to add a rolling average or mean, to columns for each day of (21,23,24,25), but that's where analysis will conclude.
output desired
 Store   Date           Profit   Rolling Mean
  ABC     May 1 2018     234     250
  XYZ     May 1 2018     410     401
  AZY     May 1 2018     145     415

where the rolling mean is for the past 30 days. Above all, I would like to have each day into its own df where I can save it to csv file the end.
Comment: Perhaps unrelated, but in the `avg()` function, the `for` loop has a `return` statement inside it, which terminates the function immediately.  That loop will only ever run once.
Answer: Rolling Mean:

The example data given in the question, has data in the format of May 1 2018, which can't be used for rolling.  Rolling requires a datetime index.

Instead of string splitting the original Date column, it should be converted to datetime, using df.Date = pd.to_datetime(df.Date), which will give dates in the format 2018-05-01
With a properly formatted datetime column, use df['Day'] = df.Date.dt.day and df['Month'] = df.Date.dt.month_name() to get a Day and Month column, if desired.

Given the original data:
Original Data:
Store        Date  Profit
  ABC  May 1 2018     234
  XYZ  May 1 2018     410
  AZY  May 1 2018     145
  ABC  May 2 2018     234
  XYZ  May 2 2018     410
  AZY  May 2 2018     145

Transformed Original Data:
df.Date = pd.to_datetime(df.Date)
df['Day'] = df.Date.dt.day
df['Month'] = df.Date.dt.month_name()

Store       Date  Profit  Day Month
  ABC 2018-05-01     234    1   May
  XYZ 2018-05-01     410    1   May
  AZY 2018-05-01     145    1   May
  ABC 2018-05-02     234    2   May
  XYZ 2018-05-02     410    2   May
  AZY 2018-05-02     145    2   May

Rolling Example:

The example dataset is insufficient to produce a 30-day rolling average
In order to have a 30-day rolling mean, there needs to be more than 30 days of data for each store (i.e. on the 31st day, you get the 1st mean, for the previous 30 days)
The following example will setup a dataframe consisting of every day in 2018, a random profit between 100 and 1001, and a random store, chosen from ['ABC', 'XYZ', 'AZY'].

Extended Sample:
import pandas as pd
import random
import numpy as np
from datetime import datetime, timedelta

list_of_dates = [date for date in np.arange(datetime(2018, 1, 1), datetime(2019, 1, 1), timedelta(days=1)).astype(datetime)]

df = pd.DataFrame({'Store': [random.choice(['ABC', 'XYZ', 'AZY']) for _ in range(365)],
                   'Date': list_of_dates,
                   'Profit': [np.random.randint(100, 1001) for _ in range(365)]})

Store       Date  Profit
  ABC 2018-01-01     901
  AZY 2018-01-02     540
  AZY 2018-01-03     417
  XYZ 2018-01-04     280
  XYZ 2018-01-05     384
  XYZ 2018-01-06     104
  XYZ 2018-01-07     691
  ABC 2018-01-08     376
  XYZ 2018-01-09     942
  XYZ 2018-01-10     297

df.set_index('Date', inplace=True)
df_rolling = df.groupby(['Store']).rolling(30).mean()
df_rolling.rename(columns={'Profit': '30-Day Rolling Mean'}, inplace=True)
df_rolling.reset_index(inplace=True)

df_rolling.head():

Note the first 30-days for each store, will be NaN

Store       Date  30-Day Rolling Mean
  ABC 2018-01-01                  NaN
  ABC 2018-01-03                  NaN
  ABC 2018-01-07                  NaN
  ABC 2018-01-11                  NaN
  ABC 2018-01-13                  NaN

df_rolling.tail():
Store       Date  30-Day Rolling Mean
  XYZ 2018-12-17           556.966667
  XYZ 2018-12-18           535.633333
  XYZ 2018-12-19           534.733333
  XYZ 2018-12-24           551.066667
  XYZ 2018-12-27           572.033333

Plot:
import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(8, 6))
g = sns.lineplot(x='Date', y='30-Day Rolling Mean', data=df_rolling, hue='Store')
for item in g.get_xticklabels():
    item.set_rotation(60)
plt.show()

Alternatively: A dataframe for each store:

It's also possible to create a separate dataframe for each store and put it inside a dict
This alternative makes is easier to plot a more detailed graph with less code

import pandas as pd
import random
import numpy as np
from datetime import datetime, timedelta

list_of_dates = [date for date in np.arange(datetime(2018, 1, 1), datetime(2019, 1, 1), timedelta(days=1)).astype(datetime)]

df = pd.DataFrame({'Store': [random.choice(['ABC', 'XYZ', 'AZY']) for _ in range(365)],
                   'Date': list_of_dates,
                   'Profit': [np.random.randint(100, 1001) for _ in range(365)]})

df_dict = dict()
for store in df.Store.unique():
    df_dict[store] = df[['Date', 'Profit']][df.Store == store]
    df_dict[store].set_index('Date', inplace=True)
    df_dict[store]['Profit: 30-Day Rolling Mean'] = df_dict[store].rolling(30).mean()

print(df_dict.keys())

>>> dict_keys(['ABC', 'XYZ', 'AZY'])

print(df_dict['ABC'].head())

Plot:
import matplotlib.pyplot as plt

_, axes = plt.subplots(1, 1, figsize=(13, 8), sharex=True)

for k, v in df_dict.items():
    axes.plot(v['Profit'], marker='.', linestyle='-', linewidth=0.5, label=k)
    axes.plot(v['Profit: 30-Day Rolling Mean'], marker='o', markersize=4, linestyle='-', linewidth=0.5, label=f'{k} Rolling')

axes.legend()
axes.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.ylabel('Profit ($)')
plt.xlabel('Date')
plt.title('Recorded Profit vs. 30-Day Rolling Mean of Profit')
plt.show()

Get a dataframe for a specific month:

Recall, this is randomly generated data, so the stores don't have data for every day of the month.

may_df = dict()

for k, v in df_dict.items():
    v.reset_index(inplace=True)
    may_df[k] = v[v.Date.dt.month_name() == 'May']
    may_df[k].set_index('Date', inplace=True)

print(may_df['XYZ'])

Plot: May data only:

Save dataframes:

pandas.DataFrame.to_csv()

may_df.reset_index(inplace=True)
may_df.to_csv('may.csv', index=False)

Answer: A simple solution may be groupby()
Check out this example :
import pandas as pd

listt = [['a',2,3],
        ['b',5,7],
        ['a',3,9],
        ['a',1,3],
        ['b',9,4],
        ['a',4,7],
        ['c',7,2],
        ['a',2,5],
        ['c',4,7],
        ['b',5,5]]

my_df = pd.DataFrame(listt)
my_df.columns=['Class','Day_1','Day_2']

my_df.groupby('Class')['Day_1'].mean()

OutPut :
Class
a    2.400000
b    6.333333
c    5.500000
Name: Day_1, dtype: float64

Note : Similarly You can Group your data by Date and get Average of your Profit.
Comment: thanks. I did my_df.groupby(["Store", "Date"])['profit'].mean() but it included stores that were closed and thus had profits of 0. The good news is if the store is closed its not in the data. But it still has it because that date exits and just assignments it a value of 0.


Why are images from newly created folder not showing up?

Question: In my app i have an option to take a picture, which will be saved in the folder:
Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES) + "/my-images";

Images are being saved fine, and i also have an option to select image which uses this code:
Intent intent = new Intent();
intent.setType("image/*");
intent.setAction(Intent.ACTION_GET_CONTENT);
startActivityForResult(Intent.createChooser(intent, "Select Picture"), SELECT_PICTURE);

here initially it used to show 2 galleries
1. Camera
2. my-images
picking any of them etc.., all worked fine.
but for no reason i deleted the my-images folder and ran the application again. The same folder is created again and images are getting saved as expected.
The problem is:
my-images gallery is not showing up now, when i click the select images button.
Except for deleting the my-images folder and restarting the application, i haven't changed any code since it used to work well.
why is it so?
Thank you
EDIT:
my activity code, which allows user to take a picture and store on SD card, and browse image gallery:
public class PictureFromAppActivity extends SherlockActivity {
    private File dir;
    private Bitmap photo;
    private String encodedString;
    private InputStream is;
    private ImageView imagePreview;
    private String selectedImagePath;
    private static final int CAMERA_REQUEST = 1888;
    private static final int SELECT_PICTURE = 1;

    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        imagePreview = (ImageView) findViewById(R.id.image_preview);
        Button takePicture = (Button) findViewById(R.id.take_a_picture);
        Button selectImage = (Button) findViewById(R.id.select_picture);

        takePicture.setOnClickListener(new View.OnClickListener() {

            public void onClick(View arg0) {

                if (android.os.Environment.getExternalStorageState().equals(
                        android.os.Environment.MEDIA_MOUNTED)) {

                        Intent cameraIntent = new Intent(
                                android.provider.MediaStore.ACTION_IMAGE_CAPTURE);
                        startActivityForResult(cameraIntent, CAMERA_REQUEST);
                    }
            }
        });
        selectImage.setOnClickListener(new View.OnClickListener() {

            @Override
            public void onClick(View arg0) {

                Intent intent = new Intent();
                intent.setType("image/*");
                intent.setAction(Intent.ACTION_GET_CONTENT);
                startActivityForResult(
                        Intent.createChooser(intent, "Select Picture"),
                        SELECT_PICTURE);
            }
        });

    }

    protected void onActivityResult(int requestCode, int resultCode, Intent data) {

        super.onActivityResult(requestCode, resultCode, data);

        if (resultCode == RESULT_OK) {
            if (requestCode == CAMERA_REQUEST) {
                if (data != null) {

                    photo = (Bitmap) data.getExtras().get("data");
                    Bitmap bitmapOrg = photo;
                    ByteArrayOutputStream bao = new ByteArrayOutputStream();
                    bitmapOrg.compress(Bitmap.CompressFormat.PNG, 100, bao);
                    byte[] ba = bao.toByteArray();

                    try {
                        encodedString = Base64.encodeBytes(ba, 0);
                    } catch (IOException e1) {
                    }

                    if (android.os.Environment.getExternalStorageState()
                            .equals(android.os.Environment.MEDIA_MOUNTED)) {

                        File imagesFolder = new File(
                                Environment
                                        .getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES),
                                "/my-images");
                        if (!imagesFolder.exists()) {

                            imagesFolder.mkdirs();
                        } else {

                        }
                        String fileName = "image_" + count + ".jpg";
                        f = new File(imagesFolder, fileName);
                        while (f.exists()) {
                            count++;
                            fileName = "image_" + String.valueOf(count)
                                    + ".jpg";
                            f = new File(
                                    Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES)
                                            + "/my-images", fileName);
                        }
                    } else {

                    }
                    f.createNewFile();

                    FileOutputStream fo = new FileOutputStream(f);
                    fo.write(ba);
                    o.flush();
                    fo.close();
                    try {
                        File imageFile = new File(
                                Environment
                                        .getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES)
                                        + "/my-images" + "/image_" + count + ".jpg");
                        Bitmap bitmap = BitmapFactory.decodeFile(imageFile
                                .getAbsolutePath());
                        imagePreview.setImageBitmap(bitmap);
                        Log.d("Image Width", "+" + imagePreview.getWidth());
                        Log.d("Image Height", "+" + imagePreview.getHeight());
                    } catch (Exception e) {

                    }
                } else {

                    Intent intent = new Intent(PictureFromAppActivity.this,
                            PictureFromAppActivity.class);
                    startActivity(intent);
                }
            } else if (requestCode == SELECT_PICTURE) {

                if (data != null) {
                    Uri selectedImageUri = data.getData();
                    selectedImagePath = getPath(selectedImageUri);
                    try {
                        File imageFile = new File(selectedImagePath);
                        Bitmap bitmap = BitmapFactory.decodeFile(imageFile
                                .getAbsolutePath());
                        imagePreview.setImageBitmap(bitmap);
                    } catch (Exception e) {

                    }

                } else {

                }
            }
        }
    }

    private String getPath(Uri selectedImageUri) {

         String[] projection = { MediaStore.Images.Media.DATA };
            Cursor cursor = managedQuery(selectedImageUri, projection, null, null, null);
            int column_index = cursor
                    .getColumnIndexOrThrow(MediaStore.Images.Media.DATA);
            cursor.moveToFirst();
            return cursor.getString(column_index);
    }
}

Answer: May be that is because of the gallery is associated with your application and its keeping the view with it.
So what is happening when you delete image/folder it been deleted from the system but the media gallery is not updated which is associated with your application.
Please use below code and let me know if this solves your problem
sendBroadcast(new Intent(Intent.ACTION_MEDIA_MOUNTED,
                Uri.parse("file://"
                        + Environment.getExternalStorageDirectory())));

Please use this code to open images from gallery
intent = new Intent(
                    Intent.ACTION_PICK,
                    android.provider.MediaStore.Images.Media.INTERNAL_CONTENT_URI);
            startActivityForResult(intent, PICK_PHOTO);

And in your on activity result
@Override
    public void onActivityResult(int requestCode, int resultCode, Intent data) {
        String path = "";
switch (requestCode) {
        case PICK_PHOTO:
            if (resultCode == RESULT_OK) {
               Uri uri = data.getData();
               path = pathFromUri(uri); // Now you can get the path of your selected image from here
           }
        break;
        }
    }

getting path from URI code
/**
     * Method to get path from uri
     *
     * @param Uri
     *            uri
     * @return String path
     * */
    private String pathFromUri(Uri uri) {
        String[] filePathColumn = { MediaStore.Images.Media.DATA };
        Cursor cursor = getContentResolver().query(uri, filePathColumn, null,
                null, null);
        cursor.moveToFirst();
        int columnIndex = cursor.getColumnIndex(filePathColumn[0]);
        String filePath = cursor.getString(columnIndex);
        cursor.close();
        return filePath;
    }

Comment: You have to use this code while opening the library to view images or folder created by you. This piece of code just send broadcast to refresh your sdcard and library to view your folder/images
Comment: so before calling the intent to open gallery??
Comment: its still showing images only from the default camera gallery which is SD/DCIM/Camera. But not the SD/Pictires/my-images folder. which also contains few images
Comment: done. Please check the edited portion of my question.thank you
Comment: Please check my edited code and use it in your selectImage button
Comment: i should use this "sendBroadcast(new Intent(Intent.ACTION_MEDIA_MOUNTED, Uri.parse(Environment
 .getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES) + "/my-images")));" before the intent in my selectImage button onClick??
Comment: i did this. but nothing seems changed :(
Comment: okay. i used the exact code you posted. It works now. Initially it shows only the default images gallery, then says "Loading.." and shows my-images folder.
Comment: actually intent of opening gallery is little bit wrong and its just taking to you a specific folder if you you the edited code that I have supplied you will get all the images available in the device in the gallery and you can select it and get the required image
Comment: fine.thanks a lot. is it refreshing(loading) images only or the whole sd card content and then fetch only images?
Comment: we are not specifying anything like, refresh only image galleries. it that fine?
Comment: Well you can do that but I used this code to refresh my gallery every time when it is being opened in my project. I found this code while googling, you can modify and check it according to your needs


re-tire elastic search multi table/index search

Question: I' trying to figure out what would be the best way to do a multi table search with elastic.co.
In particular, I was wondering if I could add more indexes to this search method.
Chapter.rb
def self.search(params)
      fields = [:title, :description, :content ]
      **tables** = [Chapter.index_name, Book.index_name]
      tire.search(**tables**, {load: true,page: params[:page], per_page: 5}) do
        query do
          boolean do
            must { string params[:q], default_operator: "AND" } if params[:q].present?
          end
        end

        highlight *fields, :options => { :tag => '' }
      end

The above example works without the Tables. How to make it work with the tables ?
Answer: If you're adding more indexes then you are moving away from it being a model-centric search. That's probably fine as I guess you'll be handling the search results differently on account of them being from different indexes.
In which case I think you can do:
Tire.search([Chapter.index_name, Book.index_name],
  page: params[:page],
  ... etc ...
) do
  query do
    ... etc ...
  end
end

It does mean that you won't be able to do stuff like load: true because you've moved outside of knowing what model to load the results for.
From digging around in the code (here) it looks like you might be able to specify multiple indexes even for a model-centric search. Something like:
tire.search({
  index: [Chapter.index_name, Book.index_name],
  load: true,
  ... etc ...

I haven't tried it though and I'm doubtful as to whether it will work - again because of not being able to load the results into a specific model once multiple indexes are involved.


How Is This Design a Randomized Complete Block Design?

Question: I recently started learning about one-way ANOVA, multiple comparisons, and contrasts. At the moment, I have an introductory-level knowledge of these concepts.
I'm analysing an experiment that compares the effects of two preparation of hormone A (hormone treatments 1 and 2), two preparations of hormone B (hormone treatments 3 and 4), and a control (hormone treatment 5) on the growth of some food-producing plant. These plants are grown in a glasshouse, in individual pots, and the pots are arranged in three sand trays. Each treatment occurs once in each tray.
The experimenters of this experiment recorded the plant height at flowering in cm and the yield for each plant.
My introductory-level understanding of a Randomized Complete Block Design (RCBD) is as follows:
We have a non-uniform group of subjects. We then split this non-uniform group of subjects into "blocks" of uniform subjects. We now have 2 groups of subjects, each group being uniform. Call these two groups $A$ and $B$. We then select from each of these two groups at random, splitting each of $A$ and $B$ into 2 subgroups. So after splitting $A$ randomly, we have the two subgroups $A_1$ and $A_2$. And after splitting $B$ randomly, we have the two subgroups $B_1$ and $B_2$. Now, let's say we have only 2 treatments for this example. We would now apply treatments 1 and 2 to one of the randomly-selected subgroups: so, say, $A_1$ gets treatment 1, $A_2$ gets treatment 2, $B_1$ gets treatment 1, and $B_2$ gets treatment 2.
Is this understanding correct?
What I'm confused about is how the experiment I mentioned earlier is a Randomized Complete Block Design. I don't see the connection between it and the example I just gave.
I would greatly appreciate it if people could please take the time to clarify this.
Answer: You have one factor (independent variable) that is the treatment condition. There are five levels of this factor (four treatment and one control). You have a second factor that is a blocking factor that is the sand trays. It is a block randomized design because randomization into the five conditions is occurring within each tray (you could not have balance within trays if you did not do this). Because randomization is constrained in this way, you must take that into account when you analyze, including both the blocking factor and treatment factor in the analysis.


node package for file attachments and image resizing

Question: I am looking for a node package that handles file attachments and image resizing. Something similar to the paperclip and carrierwave gems.
Comment: File attachments, to e-mails?
Answer: Did you search the Node Modules Wiki Page?  I searched for "upload" and found 2 candidates.
https://github.com/felixge/node-formidable
https://github.com/rootslab/formaline
For image resizing, gm or node-imagemagick will probably suffice.
Comment: [cloudinary](https://github.com/cloudinary/cloudinary_npm) is pretty cool option for people looking for cloud storage
Answer: For image upload and resize, you can check papercut,
which is simular to carrierwave.


How to implement impersonation using SignInManager for Blazor WebAssembly?

Question: I am trying to implement impersonation for a Blazor WebAssembly Solution. I chose to use the SignInManager because it seemed like the best (and only ?) solution for that.
I found some documentation and applied their solution but the newly created cookie which should contains the info for the impersonated user still contains the info from the initial user even though it is a new cookie...
For further information, I use a customized Identity model (based on Microsoft guidelines) to match my database structure. The cookies are used with a MemoryCacheTicketStore to store user info in memory and not in the cookie (too much data to send them with every requests). The cookie only contains the user's email and an expiration date.
So I tried to implement the examples I found in a Controller:
<code>[HttpPost]
        [Authorize(Roles = "Data_User_Substitute")]
        public async Task<IActionResult> Impersonate(string email)
        {
            var user = await _userManager.FindByEmailAsync(email);
            if (user == null)
            {
                return Problem();
            }

            await _signInManager.SignOutAsync();

            var userPrincipal = await _signInManager.CreateUserPrincipalAsync(user);
            await this.HttpContext.SignInAsync(IdentityConstants.ApplicationScheme, userPrincipal);

            return Ok();
        }
</code>
The code goes smoothly through it, in the HttpContext.Request.Cookies the old cookies is set to expire by _signInManager.SignOutAsync and a new one is created by this.HttpContext.SignInAsync.
But the next requests send by the application, when entering the RetrieveAsync Method from the TicketStore, use a key that still contains the info of the old user. And thus the old user is still authenticated.
<code>public async Task<AuthenticationTicket?> RetrieveAsync(string key)
</code>
The thing is that in this Controller I also have an endpoint for Login only and one for Logout only et they work perfectly fine when called by the application.
I think there must be something that is kept in memory and that is written in the cookie instead of the new user. But I could not find where.
What I expect is to have a new cookie containing the information about the impersonated user.
Answer: I found a roundabout solution that consists in using several requests to make the impersonation happen :

Request an impersonation Token

Logout the current user

Impersonate using the token

<code>        class ImpersonationModel
        {
            public string Email { get; set; }
            public DateTimeOffset Timestamp { get; set; }
        }

        [HttpGet]
        [Authorize(Roles = "Data_User_Substitute")]
        public string ImpersonationToken([FromQuery] string email, [FromServices] CustomerAreaServerSettings settings)
        {
            using (var encryptor = SymmetricEncryptor.FromString("key")
            {
                var model = new ImpersonationModel { Email = email, Timestamp = DateTimeOffset.UtcNow };
                string token = encryptor.Base64Encrypt(JsonSerializer.Serialize(model));
                return token;
            }
        }

        [HttpPost]
        [AllowAnonymous]
        public async Task<IActionResult> Impersonate([FromQuery] string token, [FromServices] CustomerAreaServerSettings settings)
        {
            if (string.IsNullOrWhiteSpace(token)) throw new ArgumentException($"'{nameof(token)}' cannot be null or whitespace.", nameof(token));

            string? email = null;
            using (var encryptor = SymmetricEncryptor.FromString("key")
            {
                string json = encryptor.Base64Decrypt(token);
                var model = JsonSerializer.Deserialize<ImpersonationModel>(json);
                if (model is null) throw new ArgumentException("Invalid token");
                if (model.Timestamp.AddMinutes(15) < DateTimeOffset.UtcNow) throw new Exception("Authentication token has expired");
                email = model.Email;
            }

            var user = await _userManager.FindByEmailAsync(email);
            if (user == null)
                return BadRequest("User does not exist");

            await _signInManager.SignInAsync(user, false);
            var newUser = this.HttpContext.User;

            return Ok();
        }
</code>


can insert data to mysql in php call from node js? WITHOUT EXPRESSJS

Question: i am new in nodejs
i want create app use nodejs and php
the scheme like this..
nodejs-->id-->php(insert data to mysql)
nodejs send value id to php page (php receive data from nodejs).in php i want to insert data to mysql
my question is how i send value to php from nodejs so php can receive the data?
how i achieve this?any tutorial explain about this?
thanks
Comment: Why don't you talk with MySQL directly from Node?
Comment: @bfavaretto : i know node can talk with mysql,but that is not the answer..,i just want send value to php from node...
Answer: So I guess you essentially are wanting to move data from a node.js server on one running PHP? In that case you will need to use a web request from within node.js to the URL that your PHP instance is running on. I would recommend Mikeal's excellent request package.
You can use it like so:
var request = require('request');

//so we assume this function is called with some
//string value for id, and some string representing
//the URL for your PHP endpoint, e.g.:
//>sendIdToURL('foo', 'http://mydomain.com/myendpoint.php')
function sendIdToURL(id, url) {
    request(url, {
        method: 'POST',
        body: id
    }, function(err, response) {
        if(err) throw err;
        else {
            //do something with the response if you want
        }
    });
}

Then on the PHP end your $HTTP_RAW_POST_DATA, or file_get_contents("php://input") will have the value 'foo'.
Does that answer the question? It's an interesting use case that you have to use php for the  mysql connection...
Answer: Node side make a post to your php file. Like:
var postRequest = {
    host: "mydomain.com",
    path: "/myfile.php?do=insert&field1=somedata",
    port: 80,
    method: "POST",
    headers: {
        'Content-Type': 'application/x-www-form-urlencoded'
    }
};

and php side recive these stuff like:
$_POST['...']

Then do db actions.
Heres more info: node.js POST request fails
Hope it helps


ROC curve with multiple points

Question: In a typical ROC curve all I do is a line from (0, 0) to (tpr, fpr) and a line from (tpr, fpr) to (1, 1). Now I see ROC curves with more than one points. Can someone explain what these extra points represent?
Comment: More than one points is unclear. Perhaps you would upload a picture of what you are asking to clarify, please? Welcome to the site.
Answer: That pair TPR, FPR you put in your curve was obtained using a threshold in the continuous output (most likely probabilities).
Other points are obtained by simply changing said threshold.

A clarification: Sampling many points in the ROC curve is the norm, I don't remember ever seeing a ROC with such few samples. You can select different strategies to sample your TPR and FPR, but sampling enough points is essential, otherwise you wouldn't need the ROC at all.
Comment: I just plotted such an ROC curve two days ago. There was one threshold involved, on a predicted probability. I varied that threshold from 0 to 1 in steps of 0.01, classified the sample for each threshold, calculated TPR and FPR for each threshold value, and plotted a 101-point ROC curve.
Comment: @DimChtz the full ROC curve can be obtained using every possible threshold, though that is usually unnecessary. Just try to sample uniformly through the curve, or do as Stephan says.
Comment: So, is there any typical threshold range or step which is most often used (in case the output is probabilities)? I mean probably a threshold of 0.001 wouldn't make any sense.

ROC curve with multiple points


example verification partial order

Question: Let A= { 1, 2, 3}  then R= { (1, 1),  (2, 2) , (3, 3) , (1, 2), (2, 3) }  ,the relation R is reflexive and anti-symmetric,i get it, but how is it following the transitive property for it to become partial order relation of set A?,really need to clear this up,is it not needed that a partial relation follows the transitive property??
Answer: This is NOT transitive because even though (1,2) and (2,3) are in the relation but (1,3) is not in the relation. Hence this is NOT a partial order.

example verification partial order


How to update/edit SharePoint classic pages content/aspx using CSOM

Question: We did successfully migrate the content from MediaWiki into the SharePoint classic page. However, some of the content that came from the MediaWiki has extra information that we don't need
Is there a way through CSOM to update all the ASPX pages we generated when migrated content from MediaWiki to SharePoint?
All we need is to remove some footers from all the pages that we generated.
Thank you in advance for everyone's help
Comment: can you post a snapshot of the page and mark on it what you want to remove, so that will be clear to think about a solution.
Answer: Here is the code that worked for me
<code>    static void Main(string[] args)
    {
        try
        {
            ClientContext ctx = GetClientContext("https://tenant.sharepoint.com/sites/sitename",
                "firstname.lastname@example.com", "password");
            var listTitle = "Site Pages";
            var list = ctx.Web.Lists.GetByTitle(listTitle);
            var items = list.GetItems(CamlQuery.CreateAllItemsQuery());
            ctx.Load(items, icol => icol.Include(i => i["WikiField"], i => i["CanvasContent1"], i => i["FileRef"], i => i.ContentType));
            ctx.ExecuteQuery();
            foreach (var item in items)
            {

                switch (item.ContentType.Name)
                {
                    case "Site Page":
                        if (item["CanvasContent1"] != null)
                        {
                            if (item["CanvasContent1"].ToString().Contains("certain pattern"))
                            {
                                ProcessSiteHTML(item["CanvasContent1"].ToString(), item["FileRef"].ToString(), item, ctx, "CanvasContent1");
                            }
                        }
                        break;
                    case "Wiki Page":
                        if (item["WikiField"] != null)
                        {
                            if (item["WikiField"].ToString().Contains("certain pattern"))
                            {
                                ProcessSiteHTML(item["WikiField"].ToString(), item["FileRef"].ToString(), item, ctx, "WikiField");
                            }
                        }
                        break;

                }
            }

        }
        catch (Exception e)
        {
            Console.ForegroundColor = ConsoleColor.Red;
            Console.WriteLine("Main " + e.Message);
            WriteLog(fileName, "Main " + e.Message);
        }
        finally
        {
            Console.ForegroundColor = ConsoleColor.White;
        }
    }

    private static void ProcessSiteHTML(string page, string pageName, ListItem item, ClientContext ctx, string pageType)
    {
        string pattern = "Regular expression pattern";
        System.Text.RegularExpressions.Regex rg = new System.Text.RegularExpressions.Regex(pattern, System.Text.RegularExpressions.RegexOptions.Multiline);
        System.Text.RegularExpressions.MatchCollection matched = rg.Matches(page.ToLower());

        if (matched.Count > 0)
        {
            System.Text.RegularExpressions.Match m =
                System.Text.RegularExpressions.Regex.Match(page, pattern, System.Text.RegularExpressions.RegexOptions.IgnoreCase);

            string updatedField = System.Text.RegularExpressions.Regex.Replace(page,
                m.Value, string.Empty, System.Text.RegularExpressions.RegexOptions.Multiline | System.Text.RegularExpressions.RegexOptions.IgnoreCase);
            item[pageType] = updatedField;
            item.Update();
            ctx.ExecuteQuery();
            Console.WriteLine(pageName + " has been updated");
        }
    }
</code>


Could not set Use Local IIS URL Web Server for SharePoint Web Server project in Visual Studio 2010

Question: I have a custom SharePoint ASP.NET Web Service project.
I've setup it to debug as described in http://www.sharepoint-tips.com/2007/08/web-services-on-sharepoint-making-f5.html
it used to work as long as I was working with Visual Studio 2005.
but now I want to setup Visual Studio 2010 environment,
and I'm stuck trying to set 'Use Local IIS Web server' setting in VS, the error says: "The local IIS URL ... specified for web project ... has not been configured."
Tried in both: Windows2003 with MOSS 2007 and Windows 2008 with SharePoint server 2010. All the same.
What I'm missing? thanks in advance.
Answer: For Visual Studio 2010 and SharePoint 2010, F5 should work right out of the box. There's no need for any special tricks. Have a try without that outdated blog posting.
Comment: hm, I'm in doubt, how that possible?
Comment: hm, I'm in doubt, how that possible? look, I have to define WEB page of project properties anyway. So, my Start URL is http://server:port/_layouts/MyService.asmx and if I don't use Local IIS, I should leave the 'Use Visual studio development server'. This combination obviously do not allow me to hit a breakpoint.
Comment: @vladimirleontyev try following http://www.thesharepointblog.net/Lists/Posts/Post.aspx?List=815f255a%2Dd0ef%2D4258%2Dbe2a%2D28487dc9975c&ID=67 .


JavaScript from custom module not detected

Question: I have created a custom module, which is working. Except the JavaScript, which is not being detected. I have been following the description at https://www.drupal.org/docs/8/api/javascript-api/javascript-api-overview and I have checked other modules.
What have I done to implement JavaScript? I have created a file with the following JavaScript content (location: MODULE-ROOT/js/info.js):
/**
 * @file
 * Info behaviors on node edit form.
 */

(function ($, window) {

  'use strict';

  Drupal.behaviors.nodeEditForm = {
    attach: function () {
      console.log('Hi there!');
    }
  };

})(jQuery, window);

Then I have added the following content into my MODULE-NAME.libraries.yml file:
nodeEditForm:
  version: 'VERSION'
  js:
    js/info.js: {}

I expect to see "Hi there!" in my webdeveloper tools inside Google Chrome browser, after I flush the entire Drupal 8 and browser cache. I do not get any output there.
Is it a "must" to attach the JavaScript to any existing behavior? If so: how can I learn which behavior is tied to an node edit form?
Thanks for your attention.
Comment: But did you load the library in code anywhere? It must be attached to be included in the page. In this example it sounds like you'd want to do a hook_form_alter, and attach your library ex: `$form['#attached']['library'][] = 'mymodule/nodeEditForm';`
Answer:
Drupal uses a high-level principle: assets (CSS or JS) are still only
  loaded if you tell Drupal it should load them. Drupal does not load
  all assets (CSS/JS) on all pages, because this is bad for front-end
  performance.

You need to inform Drupal to attach your library to the page.
In your case, you can use hook_form_alter and check the $form_id or hook_form_FORM_ID_alter for a single case and add this:
$form['#attached']['library'][] = 'mymodule/nodeEditForm';
Then clear the cache. Search the core folder for $form['#attached'] for more examples. You should then see your js file listed in the source of the page as well as console log output.
Comment: Thanks that worked. Is there also an "object oriented" way to add the library from a controller or service? I have not found anything about that in the Drupal.org documentation at https://www.drupal.org/docs/8/creating-custom-modules/adding-stylesheets-css-and-javascript-js-to-a-drupal-8-module


Phpmyadmin - Mysql no privileges

Question: I installed wamp 2.1 on windows 7. However when i open phpMyAdmin, I get the error, Mysql 'No Provileges'. I uninstalled wamp and reinstalled it a few times, but it doesn't help. Does anyone know how to solve this issue?
Also, when i tried creating a database from mysql console, i am getting the following error:
ERROR 1044 <42000>: Access denied for user ''@'localhost' to database 'a_db_name'
Comment: Don't connect with chrome.

See : http://stackoverflow.com/questions/4770348/phpmyadmin-reports-no-privileges
Comment: Then have you tried connecting specifically from 127.0.0.1 instead of "localhost" ?
Comment: @Jerska yes, the problem still persists.
Comment: @Jacob if you give me the solution to my previous issues, i will definitely accept :)
Comment: What happens when you login from command line? Execute SHOW GRANTS;
Comment: I got:
Grants for @localhost
GRANT USAGE ON *.* TO ''@'localhost'
1 row in set
Answer: Thank God and to all helped.
Its simple.

Must! click logout icon in phpadmin page
In login page, type:

username:root password: (blank)
surprise. now you can happily create your database.
Answer: Are you logging into MySQL as root? You have to explicitly grant privileges to your "regular" MySQL user account while logged in as MySQL root.
First set up a root account for your MySQL database.
In the terminal type:
mysqladmin -u root password 'password'

To log into MySQL, use this:
mysql -u root -p

Edit:
To set the privileges manually start the server with the skip-grant-tables option, open mysql client and manually update the mysql.user table and/or the mysql.db tables. This can be a tedious task though so if what you need is an account with all privs I would do the following.
Start the server with the skip-grant-tables option
Start mysql client (without a username/password)
Issue the command
flush privileges;

which forces the grant tables to be loaded.
Create a new account with the GRANT command something like this (but replacing username and password with whatever you want to use.
GRANT ALL on *.* to 'username'@'localhost' identified by 'password';

Restart the server in normal mode (without skip-grant-tables) and log in with your newly created account.
Refer this MySQL docs.
Comment: I am actually unable to login as root. When i go to localhost/phpmyadmin, there are two databases there: information_schema and test. Is there anything missing?
Comment: What error are you getting now? Where are you trying to login? If you are getting 1045 error, refer this: http://www.webyog.com/faq/content/23/18/en/error-no-1045-connection-denied.html
Comment: Still the same error, "No privilege". But my guess is that mysql database is missing as well as performance_schema.
Comment: Please tell whether you are using phpmyadmin or using mysql command line now?
Comment: phpmyadmin. mysql schema is missing there as well as performance_schema.
Comment: Did you try my answer? mysqladmin -u root password 'password'
To log into MySQL, use this: mysql -u root -p
Comment: Jacob, thanks for you info above, it helped me piece together my issues in mamp.
Answer: Take a look at my topic regarding this issue, which takes some of the above.
MAMP mysql broken root user
You need to shutdown your mysql install and restart it from the command line properly like is indicated above.  In my topic I have full clear instructions on how to do so.  My instructions are for MAMP but you should be able to adapt it for your install.
Answer: Simple solution. Just find the icon right to "home" in PhpMyAdmin and click to logout. Then login using username "root" and password ""(blank). This will work accordingly.
Answer: username ought to be root and keep the password null(keep the password field blank)
Answer: right click on the wamp icon and goto mysql console.
Login with password if you have set any.By default the password is blank and username is 'root'
Once you are in mysql prompt execute
FLUSH PRIVILEGES;
now quit the command prompt and you are good to go.
Answer: in step 1 .inter your databasevery important.
in step 2 .select your database via all tables .
in step 3. backup type=replace & export.
in step 4 import database in my sql.
attention please:in import your database...
all table must view & selected in step 4.

Phpmyadmin - Mysql no privileges


What are some FS2 Error Hanlding Practices?

Question: I am new to FS2, Cats Effect, etc., but I have been using Scala since 2005 and Akka since 2010...
I tuned up the FS2 example code to play with error-handling ideas, but I wondered if there are better idiomatic ways to deal with this.
I tried the various Stream mechanisms, but, it was very troublesome, and the best I could do was collapse the whole stream into a single output value. I had a lot of trouble finding good documentation on error handling in FS2.
import cats.effect.{IO, IOApp}
import fs2.{Stream, text}
import fs2.io.file.{Files, Path}

import scala.util.{Failure, Success}

object Converter extends IOApp.Simple {

  private val converter: Stream[IO, Unit] = {
    def fahrenheitToCelsius(f: Double) =
      (f - 32.0) * (5.0/9.0)

    def convert(string: String) =
      try
        Success(fahrenheitToCelsius(string.toDouble).toString)
      catch
        case cause: Exception => Failure(cause)

    Files[IO].readUtf8Lines(Path("testdata/fahrenheit.txt"))
      .filter(s => s.trim.nonEmpty && !s.startsWith("//"))
      .map(line => convert(line) match
        case Success(value) => value
        case Failure(cause) => s"Failed to convert $line: ${cause.getMessage}"
      )
      .intersperse("\n")
      .through(text.utf8.encode)
      .through(Files[IO].writeAll(Path("testdata/celsius.txt")))
  }

  def run: IO[Unit] =
    converter.compile.drain
}

Comment: So, what exactly would you like to do when you get an error? Skip the record? Log and slip? re-try the operation? Stop the whole stream? Start the whole process once again? Run an alternate process?
Comment: Well there is no better or more idiomatic way. Error handling is probably one of the most subjective and context dependant topics. `Stream` allows you to do all that I said and more, including what you did. All of them are useful to some folks and in some specific contexts. Thus, my question is, what do you want to do? What you did is perfectly fine if what you wanted was to replace errors with _"valid values"_ (I would have used `toDoubleOption` and `Either` instead, but those are minor details). - Now, not sure if your end user wanted to see both success and errors in the same output file.
Comment: You can do `string.toDoubleOption.toRight(left = "SOME ERROR MESSAGE")` or maybe a custom exception if you prefer that.
Comment: In the example given, my goal was to eat the error, and replace it with a string in the stream representing the error. So basically I am asking if I am doing this the best way, or is there a better, more idiomatic way?
Comment: Yes, that is helpful. `toDoubleOption` helps. You are clearly more intelligent than GitHub Copilot ;-) Hunting through the IntelliJ alternate methods can lead to madness. There does not seem to exist `toDoubleEither` so I will have to play with that. I totally agree "Error handling is probably one of the most subjective and context dependant topics." It also seems to be one of the most ignored subjects as most people seem more interested in the *happy path* than robust reliability and diagnostics. Years ago, I played with Akka Streams, but PS2 seems a quantum leap better.
Comment: Yes, but I want to capture the *cause* of the Exception...
Comment: If you want to, you may create your own `NmberFormatException` or whatever it is. I really have never ever cared about that exception, it just means the `str` is not a number.
Comment: I found what I wanted... see my answers...
Answer: Thanks to Luis Miguel Mejia Suarez, I tweaked this to become
import cats.effect.{IO, IOApp}
import fs2.{Stream, text}
import fs2.io.file.{Files, Path}

object Converter extends IOApp.Simple {

  private val converter: Stream[IO, Unit] = {
    def fahrenheitToCelsius(f: Double) =
      (f - 32.0) * (5.0/9.0)

    Files[IO].readUtf8Lines(Path("testdata/fahrenheit.txt"))
      .filter(s => s.trim.nonEmpty && !s.startsWith("//"))
      .map(line => line.toDoubleOption match
        case Some(value) => fahrenheitToCelsius(value).toString
        case None => s"Failed to convert $line to a number."
      )
      .intersperse("\n")
      .through(text.utf8.encode)
      .through(Files[IO].writeAll(Path("testdata/celsius.txt")))
  }

  def run: IO[Unit] =
    converter.compile.drain
}

If toDoubleOption did not exist, the right thing to do would have been to implement it. It might be worth implementing toDoubleEither or toDoubleTry as those do not seem to exist.
I had tried playing with Stream methods like handleError, HandleErrorWith, evalTap, evalFilter, etc., but I could not achieve what I wanted. I still have a lot to learn.
Answer: After more experimentation, I find the following more satisfying...
import cats.effect.{IO, IOApp}
import fs2.{Stream, text}
import fs2.io.file.{Files, Path}

import scala.util.{Failure, Success, Try}

object Converter extends IOApp.Simple {

  private val converter: Stream[IO, Unit] = {
    def fahrenheitToCelsius(f: Double) =
      (f - 32.0) * (5.0/9.0)

    Files[IO].readUtf8Lines(Path("testdata/fahrenheit.txt"))
      .filter(s => s.trim.nonEmpty && !s.startsWith("//"))
      .map(line => Try {line.toDouble} match
        case Success(value) => fahrenheitToCelsius(value).toString
        case Failure(cause) => s"Failed to convert $line: $cause"
      )
      .intersperse("\n")
      .through(text.utf8.encode)
      .through(Files[IO].writeAll(Path("testdata/celsius.txt")))
  }

  def run: IO[Unit] =
    converter.compile.drain
}

Sometimes, I forget how powerful Scala is, even without Cats Effect and FS2...


Redux Saga's race won't trigger

Question: I tried to make a 'simple' saga where I race a 'chain' of sagas and a CANCEL action. The race is going down smoothly, and the task is 'winning', BUT the race will return an empty object (so my if(task) won't get called, even though the race went down properly).
How it's works:
task:
I call the allActivity function.
allActivity calls the uploadData function with an array of objects.
After that I basically just upload each item one by one with the sage.
Code:
function* handleSingle(item) {
    const respone = yield call(myFetch, item)
    return response
}

function* uploadData(data) {
    yield all(data.map(item=>handleSingle(item)))
}

function* allActivity() {
    const data = yield select(state=>state.data)
    yield call(uploadData, data)
}

function* uploadJobWorker() {
    const { task, cancel } =  yield race({
        task: call(allAcivity),
        cancel: take(ON_CANCEL)
    })

    if(task)
        console.warn('done')
    if(cancel)
        console.warn('cancelled')

    console.warn('end')
}

So after the race is finished I can see the 'end' logged out, but not the 'done'. And if the cancel get triggered I can see the 'cancelled' logged out. (so the only part what never get triggered is the 'done')
Comment: `call` doesn't return a task but instead returns the return value of a function/generator. Return value of `allActivity` is undefined and so the `task` variable is resolved to undefined as well.
Comment: Ohh this make sense.
Answer: I was able to fix this by a quite simple way.
const delay = (ms) => new Promise(res => setTimeout(res, ms))

function* handleSingle(number) {
    const response = yield call(delay, 1000)
    console.warn(`${number} is finished`)
    return response
}

const mock = [1,2,3,4,5,6,7,8,9,10]

function* uploadData() {
    for(const number of mock) {
        yield call(handleSingle, number)
    }
    yield put({ type: FINISHED })
}

function* allActivity() {
    yield call(uploadData)
}

function* uploadJobWorker() {
    yield fork(allActivity)
    const { task, cancel, error } =  yield race({
        task: take(FINISHED),
        cancel: take(CANCEL),
        error: take(ERROR),
    })

    if(task)
        console.warn('done')
    if(cancel)
        console.warn('cancel')
    if(error)
        console.warn('error')

    console.warn('end')
}

export default function* rootSaga() {
    yield all([
        takeEvery(EXAMPLE, uploadJobWorker)
    ])
}

So instead of calling it. I will fork it so it's won't block the saga. And instead waiting for a FINISHED action.

Redux Saga's race won't trigger


How to display 'Welcome [first_name] [last_name]' in a custom views block? D8

Question: I would like to create a custom block / view to display a welcome message to the currently logged in user in Drupal 8. I followed this answer but it isn't working for me. Maybe things changed in D8. Is there a way to do it in D8 and / or am I doing something wrong?
I've added a bunch of Fields and Contextual Filters trying to get it to work but nothing shows up.
Answer: Changes:

Remove all the contextual filters, that you added. They are not necessary. You have already a filter criteria for the current user.
Change format from table to an unformatted list.
Remove all the fields except first name, last name.
Click on first name and exclude it from display.
Click on last name. Find the fieldset to override results, expand it.
Write your message there. It should look like: Welcome {{field_first_name}} {{field_last_name}}. You can look up the exact tokens in a fieldset below the text field.
Save the views.
Place your block in some region at /admin/structure/block.
Comment: Thank you! That worked perfectly. Much appreciated.


Too many vehicles in Veins simulation

Question: I've adapted the basic example of Veins (Erlangen example) and extended the number of vehicles to 2500 in the rou file. Additionaly, I implemented some more accidents and rerouting.
When I execute the simulation, there are not 2500 but 2510 vehicles over the course of the simulation. Why does this happen and how can I prevent that?
Comment: Possible duplicate of: [Why the number of vehicles are not increased in veins?](https://stackoverflow.com/questions/68181848/why-the-number-of-vehicles-are-not-increased-in-veins), (But not enough info give to be sure ... )
Comment: No, that is something different. As I've written, I increased the number of vehicles to 2500 (in that line of code mentioned in the question you linked). But instead of 2500 vehicles, 2510 vehicles will be spawned in the simulation
Answer: Change number to 2500:
<routes>
    ...
    <flow id="flow0" ... number="2500"/>
    ...
</routes>

Then use duarouter utility to generate your routes:
duarouter -n erlangen.net.xml -r erlangen.rou.xml --randomize-flows -o erlangen.new_rou.xml

Finally replace erlangen.rou.xml with erlangen.new_rou.xml in erlangen.launchd.xml
Comment: Thanks for the reply. That is a way to fix it. However, I still don't understand **why** this happens in the first place.
Comment: I experienced that as well, that's why I always prefer to generate individual trips with duarouter instead of defining a flow, if I have a specific amount of vehicles in my scenario.

Too many vehicles in Veins simulation


iOS on mobile not showing CSS 1px borders correctly

Question: I have a Bootstrap header customized with CSS borders to give it a 2px and 1px shadow effect, like this:

And the CSS is:
.navbar{
 border-bottom: 1px solid #C7C7C7;
}
.header{
 border-bottom: 2px solid #EFEFEF;
}

The HTML tags of the page are:
<meta name="viewport" content="width=970" />

For some strange reason, when viewing the page in iOS on a mobile device like iPad or iPhone, it doesn't show the borders correctly. Initially when loading it, it shows only 1px of the line and when zooming the page the other 2px are visible but somehow distorted. The full line is not shown.
What could be wrong here? Is this a iOS issue or..?
Comment: I forgot to mention, it's happening on both Safari and Chrome.
Answer: I found the solution. I changed the meta tags to:
<meta name="viewport" content="width=device-width, initial-scale=1">


How to set Form API's password_confirm in/visible depending on prior select list input?

Question: Neither visible nor invisible seems to work on password_confirm. Am I doing something wrong? What else can I try? This is my current code for a subadministrator form, where I'ld like to display the password_confirm only after a user is chosen:
$form['user_select'] = array(
  '#type' => 'select',
  '#options' => array('' => '- ' . t('None') . ' -') + _get_users_by_role('Subadministration'),
  '#required' => TRUE,
  '#description' => t('Chose account'),
);

$form['confirm_password'] = array(
  '#type' => 'password_confirm',
  '#required' => TRUE,
  '#description' => t('Type same password in both fields'),
  '#states' => array(
    'invisible' => array(
      ':input[name="user_select"]' => array('value' => ''),
    ),
  ),
);

Answer: A container around the password_confirm did the trick! Fieldset works too.
$form['user_select'] = array(
  '#type' => 'select',
  '#options' => array('' => '- ' . t('None') . ' -') + _get_users_by_role('Subadministration'),
  '#required' => TRUE,
  '#description' => t('Chose account'),
);

$form['pass_wrapper'] = array(
    '#type' => 'container',
    '#states' => array(
      'invisible' => array(
        ':input[name="user_select"]' => array('value' => ''),
      ),
    ),
  );

$form['pass_wrapper']['confirm_password'] = array(
  '#type' => 'password_confirm',
  '#required' => TRUE,
  '#description' => t('Type same password in both fields'),
);

Answer: is the value ever getting "",try the following
 'invisible' => array(
   ':input[name="user_select"]' => array('value' => t('None')),
 ),

Comment: Doesn't work :(


continued fraction of $\sqrt{41}$

Question:
Show that $\sqrt{41} = [6;\overline {2,2,12}]$

here's my try:
$$\sqrt{36}<\sqrt{41}<\sqrt{49}\implies6<\sqrt{41}<7\implies\lfloor\sqrt{41}\rfloor=6$$
$$\sqrt{41}=6+\sqrt{41}-6=6+\frac{1}{\frac{1}{\sqrt{41}-6}}$$
$$\frac{1}{\sqrt{41}-6}=\frac{\sqrt{41}+6}{41-36}=\frac{\sqrt{41}+6}{5}=\frac{12+\sqrt{41}-6}{5}=2+\frac{\sqrt{41}-4}{5}$$
So far,
$$\sqrt{41}=6+\frac{1}{2+\frac{\sqrt{41}-4}{5}}=6+\frac{1}{2+\frac{1}{\frac{5}{\sqrt{41}-4}}}$$
But, $$\frac{5}{\sqrt{41}-4}=\frac{5(\sqrt{41}+4)}{41-16}=\frac{\sqrt{41}+4}{5}=\frac{6+\sqrt{41}-2}{5}=\color{red}{1}+\frac{\sqrt{41}-1}{5}$$
It suppose to be $2$ and not $1$.
Where is the mistake? (I triple-checked and it seems fine to me)
Comment: $$\sqrt{41}-1>5$$ Accordnig to the method you introduced here, you should have: $$\sqrt{41}+4=10+\sqrt{41}-6$$ and go from there
Answer: $$\frac{6+\sqrt{41}-2}{5}=\frac{10+\sqrt{41}-6}{5}=2+\frac{\sqrt{41}-6}{5}$$
with $0<\sqrt{41}-6<5$
Comment: Oh I see, thanks. you should always break it to some $a_i$ and a fraction in $(0,1]$
Answer: Method described  by Prof. Lubin  at Continued fraction of $\sqrt{67} - 4$
$$  \sqrt { 41} = 6 +     \frac{  \sqrt {41} - 6 }{ 1 }  $$
 $$    \frac{ 1 }{   \sqrt {41} - 6 }  =  \frac{    \sqrt {41} + 6 }{5 } = 2 +  \frac{    \sqrt {41} - 4 }{5 } $$
 $$    \frac{ 5 }{   \sqrt {41} - 4 }  =  \frac{    \sqrt {41} + 4 }{5 } = 2 +  \frac{    \sqrt {41} - 6 }{5 } $$
 $$    \frac{ 5 }{   \sqrt {41} - 6 }  =  \frac{    \sqrt {41} + 6 }{1 } = 12 +  \frac{    \sqrt {41} - 6 }{1 } $$
Simple continued fraction tableau:
 $$
 \begin{array}{cccccccccccccccc}
 & & 6 & & 2 & & 2 & & 12 & & 2 & & 2 & & 12 & \\
 \\
  \frac{ 0 }{ 1 }   &   \frac{ 1 }{ 0 }   & &   \frac{ 6 }{ 1 }   & &   \frac{ 13 }{ 2 }   & &   \frac{ 32 }{ 5 }   & &   \frac{ 397 }{ 62 }   & &   \frac{ 826 }{ 129 }   & &   \frac{ 2049 }{ 320 }    \\
  \\
 & 1 & & -5 & & 5 & & -1 & & 5 & & -5 & & 1
 \end{array}
 $$
$$
 \begin{array}{cccc}
  \frac{ 1 }{ 0 }   & 1^2 - 41 \cdot 0^2 = 1 &     \mbox{digit}  &  6  \\
  \frac{ 6 }{ 1 }   & 6^2 - 41 \cdot 1^2 = -5 &     \mbox{digit}  &  2  \\
  \frac{ 13 }{ 2 }   & 13^2 - 41 \cdot 2^2 = 5 &     \mbox{digit}  &  2  \\
  \frac{ 32 }{ 5 }   & 32^2 - 41 \cdot 5^2 = -1 &     \mbox{digit}  &  12  \\
  \frac{ 397 }{ 62 }   & 397^2 - 41 \cdot 62^2 = 5 &     \mbox{digit}  &  2  \\
  \frac{ 826 }{ 129 }   & 826^2 - 41 \cdot 129^2 = -5 &     \mbox{digit}  &  2  \\
  \frac{ 2049 }{ 320 }   & 2049^2 - 41 \cdot 320^2 = 1 &     \mbox{digit}  &  12  \\
 \end{array}
 $$
Answer: [This old question popped up in the feed today (years later), and I can see that there's an accepted answer. Case closed. Though, reading the question and its answers, I can't help but feeling that the problem could (should?) have been tackled differently. I'm leaving this answer in the hope it might inspire and help some future readers looking for solutions to this type of problems.]
Given that the problem statement gives the result upfront and only asks for a proof, I would have just gone the other way around. Starting from the continued fraction $X = [6;\overline {2,2,12}]$ (we remark here that it's a positive number), one can see without too much difficulty that
$$ X+6=12+\cfrac{1}{2+\cfrac{1}{2+\cfrac{1}{X+6}}} $$
And after a few mildly tedious but very easy steps, we derive
$$
\iff X+6=12+\cfrac{1}{2+\cfrac{X+6}{2X+13}} \\
\iff X+6=12+\cfrac{2X+13}{5X+32} \\
\iff X+6=\cfrac{62X+397}{5X+32} \\
\iff 5X^2+62X+192=62X+397 \\
\iff 5X^2-205=0 \\
\iff X^2-41=0 \\
$$
Since we know $X$ is positive, the result $X=\sqrt{41}$ follows.
Answer: Given $(\sqrt{41}+4)/5$, render $6<\sqrt{41}<7$ which you used at the start.  Then your fraction lies between $(6+4)/5$ and $(7+4)/5$ showing the integral part is $2$.
You should be able to get the integer part at every stage just from $6<\sqrt{41}<7$.

continued fraction of $\sqrt{41}$


Getting product stock quantity isn't working

Question: On single product page I'm trying to get stock quantity or atleast stock status from woocommerce product but it seems not working properly, even if it's simple product (manage stock status is checked) or variable product the button is not showing. And strange thing happens - I've got Yith Product Bundle plugin and even if every item is in stock it shows that button but whatever, that's not main problem.
So here is code I'm using
add_action('woocommerce_before_add_to_cart_button', 'show_wishlist');

function show_wishlist() {
    global $product;
    $stockQ=$product->get_stock_quantity();

    if ( $stockQ < 1 ) {
        echo "<button>Hello</button>";
    }
}

I've also tried but completely nothing happens.

add_action('woocommerce_before_add_to_cart_button', 'show_wishlist');

function show_wishlist() {
    global $product;

    if ( ! $product->is_in_stock() ) {
        echo "<button>Hello</button>";
    }
}

Answer: First check if the hook is firing properly.
add_action( 'woocommerce_before_add_to_cart_button', 'show_wishlist' );

function show_wishlist(){
    echo '
Some custom text before
';
}

If you are able to confirm that the hook is working then use the below code.
    function show_wishlist() {
        global $product;
        // Change In Stock Text
        if ( $product->managing_stock() && $product->is_in_stock() ) {
    //add your code here for Products in Stock
    }
else{
//add your code here for Products out of Stock
}
    }

Comment: Ooh, I've just checked if the hook is firing properly, well it is but only on products that are available in stock but it's not firing on products that are out of stock (and I want to fire function on products that are out of stock). So it looks like I should use different hook.
Comment: Why do you want to check stock on out of stock products? You can just add an else part to the above If condition and do what you want. I have updated my answer and added else part. Do accept the answer if it answers the question


What is the finger position of this tab?

Question: I don't know how to mute the first string using my left hand.

I looked at a video on youtube. It looks like the guy doesn't use fret 11 at all. Why? Sorry I don't know too much music theory.
Here's his finger position:

Should I strictly follow the tab? Or I need to make adjustments?
Comment: Tabs are notoriously inaccurate - often written by people who really don't understand. Sorry, but sometimes true!
Comment: The chord pictured is an E chord voiced E-B-E-G# starting on the A string with the first and last string ignored -- not by muting but by only playing the fingered strings. The tab is an E chord voiced G#-B-E-G#, same deal with the other strings. Listen and use the one you think sounds right. Hearing the difference is good ear training!
Comment: @Tim Hmm so I should read staff instead and figure out the best finger position by myself in this case? This sounds time-consuming.
Comment: Possibly time-consuming initially. Sanity-saving in the long run. Believe me. And use those ears.
Answer: He does look like he is using the 11th fret.  In the TAB the lowest line is the low E string.  In other words the strings are "upside down" in comparison to how they would be if you are looking at the guitar in someone else's hands.  As you look down at your guitar neck while you hold it the strings should appear in the order shown in the TAB.
The "first" string is the high e string and for a chord like this I don't think you mute it with your left hand.  Rather, you just don't attack the string with your picking hand.
Answer: I think the best answer to your question is that because ultimately it's the same triad chord E maj (E, G# and B). The same note (G#) he ignored on the 11 fret is still there an octave higher on the 2nd string fret9. But, although he's playing the same chord, it's a different voicing of it. The guy looks like playing 2nd, 3rd and 4th strings on fret 9 which is the first inversion 5th (B) - 3rd (G#) - root (E). The tab adds the 3rd in the bass. You can get away with playing either voicings because they sound pretty similar, but if you want to be faithful to the song, pay attention to the bass note, that will help you figure out which one is it.
Take it from a guy who used tabs for years, never "strictly" follow tabs, they're made by normal people with normal ears who often make mistakes. Use your own ears as much as you can until you no longer need tabs. Keep rocking \m/


Dockerfile CMD for taking bash commands from host

Question: I've created a dockerfile with various compile and build tools. The goal of the dockerimage is to standardize our development tools, and make it easy and consistent for developing.
Everything is installed.
What I am stuck on, is how to make the docker container keep running, and be able to have a bash shell to that container so that I can run, for example, make etc. ?
If I use  ENTRYPOINT /bin/bash my container exits immediately. How to keep the container running?
Answer: You should use the command at run time. You run your Docker container in interatice mode (-i) and set the command to "/bin/bash":
docker run -it myDockerImage myCommandToExecuteInteractively

For instance:
docker run -it myDocker /bin/bash

Here is a real life example:
a) Pulling the most basic image
docker pull debian:jessie-slim

b) Let's have a bash there:
docker run -it debian:jessie-slim /bin/bash

c) Enjoy:
Answer: A docker container will run as long as the CMD/Entrypoint from your Dockerfile takes.
You can run your Docker container in interactive mode using switch i
sudo docker run -it --entrypoint=/bin/bash <imagename>

Example : docker run -it --entrypoint=/bin/bash ubuntu:14.04
This will start an interactive shell in your container. Your container will exit as soon as you exit that shell.


Can an Angular JavaScript reduce method be used to find a moving average

Question: Would like to convert my current Angular reduce function below to calculate a moving average based on a period of every ~2-3 items. Is this possible to do w/ a standard Angular Reduce function?
This is what I currently have to compute a basic average, but would like to convert it to a moving average w/ a settable period if possible, but not sure how/if it's possible to do w/ a standard Angular JS Reduce function.
const data = [
    { val: 20 },
    { val: 15 },
    { val: 10 },
    { val: 12 },
    { val: 15 },
    { val: 09 },
    { val: 14 },
    { val: 17 },
    { val: 11 },
    { val: 15 }
]

var avg = data.reduce((accum, curVal) => accum + curVal.val, 0) / data.length;
console.log("avg= " + avg);

Answer: You can use the slice function to get a portion of the array
let startIndex = 2;
let endIndex = 5;
let length = endIndex - startIndex;
var avg= data.slice(startIndex , endIndex ).reduce((accum, curVal) => accum + curVal.val, 0) / length;
console.log("avg= " + avg);

Answer: If I've understood the question correctly you want an array with a set of averages based on a window that moves one entry at a time across the data. Here's an example using a for loop, please see the comments for how this works:

const data = [{ val: 20 }, { val: 15 }, { val: 10 }, { val: 12 }, { val: 15 }, { val: 9 }, { val: 14 }, { val: 17 }, { val: 11 }, { val: 15 }];

const get_moving_averages = (data, window_size = 3) => {
  const sums = [];

  for(let i = 0; i < data.length; i++) {
    if(i < window_size) {
      // add everything to the first slot for the first items up to window size
      sums[0] = (sums[0] ?? 0) + data[i];
    } else {
      // calculate value by taking the previous value, adding the new data
      // and taking off value that shouldn't be included in this sum from the previous sum
      sums[i + 1 - window_size] = sums[i - window_size] + data[i] - data[i - window_size];
    }
  }

  return sums.map((sum) => sum / window_size);
};

console.log(get_moving_averages(data.map(({val}) => val)));

Same idea but with a reduce function:

const data = [{ val: 20 }, { val: 15 }, { val: 10 }, { val: 12 }, { val: 15 }, { val: 9 }, { val: 14 }, { val: 17 }, { val: 11 }, { val: 15 }];

const get_moving_averages = (data, window_size = 3) =>
  data.reduce((acc, val, i, all) => {
    const first_few = i < window_size;
    const sum_index = first_few ? 0 : i + 1 - window_size;

    acc[sum_index] = (acc[sum_index] ?? 0) + val;

    if(!first_few)
      acc[sum_index] += acc[sum_index - 1] - all[i - window_size];

    return acc;
  }, [])
  .map( (sum) => sum / window_size );

console.log(get_moving_averages(data.map(({val}) => val)));


How to scan values in from keyboard- JAVA

Question: I'm trying to make a code where the user enters to numbers, and the if statement compares the values to give an output. I feel the error is in how I am scanning the numbers. Any suggestions? (also sorry for the noob question, just starting to learn java)
public class MarvinsRoom {
    public static void main(String[] args) {
        Scanner scan = new Scanner(System.in);
        System.out.println("Enter Integer Value");
        firstNumber = scan.nextInt();

        Scanner scan = new Scanner(System.in);
        System.out.println("Enter Integer Value");
        secondNumber = scan.nextInt();

        if (firstNumber > secondNumber) {
            System.out.println("Emotions");
        } else {
            System.out.println("no feelings");
        }
    }
}

Answer: Error: Two scan variables and variables firstNumber and secondNumber doesnt have types.
public class MarvinsRoom {
    public static void main(String[] args) {
        Scanner scan = new Scanner(System.in);
        System.out.println("Enter Integer Value");
        int firstNumber = scan.nextInt();

        System.out.println("Enter Integer Value");
        int secondNumber = scan.nextInt();

        if (firstNumber > secondNumber) {
            System.out.println("Emotions");
        } else {
            System.out.println("no feelings");
        }
    }
}

Comment: Have you imported Scanner?
Comment: As in `import java.util.Scanner;` before the `public class MarvinsRoom` .
Comment: You should include the entire program in your answer for someone who's beginning to learn as they may not use an IDE so they may not know where imports are required.
Comment: How do we know if he hadn't imported the necessary libraries?
Comment: Sometimes answers can be made better through comments.
Comment: Marking as correct or best answer will show some appreciation @Tushar Joshi
Answer: I believe your error is because you have two Scanner objects in your code, You only need one as it is in the same method. I think that should be the only issue you have as far as I can see
Answer: Forgot to
import java.util.Scanner;
and at the end
scan.close();
Comment: I posted that under my comment...did you appreciate that?


How to solve equations with unknown bases?

Question: My friend dared me to find rational solutions for these equations:
$$x+11y=100$$
$$xy=2$$
And he keeps claiming it's possible but even wolfram alpha disagrees.

Perhaps the base is wrong?
$$x+(b+1)y=b^2$$
$$xy=2$$
But then we have $2$ equations and $3$ unknowns, and I don't know how to tackle this.
Wolfram shows the solution $(x,y,b) = (1,2,3)$
But how do I get to them step by step?
Also, notice when I plug in the base, the wolfram misses the irrational $x,y$ solutions for base $10$ even thought I did not put any restrictions on the solutions. Why is that?
It only shows $-1$ and $3$ as solutions for $b$?
Comment: As shown in your Wolfram|Alpha link, your problem can be reduced to finding bases $b$ such that $\sqrt{b^4 - 8b - 8}$ is a rational number. I don't know how to solve it, though.
Answer: You need
$$   (b+1)y^2 - b^2 y + 2 = 0  $$
to have a rational root, that is
 $$  b^4 - 8 b - 8 $$
a square. This is smaller than $$ \left( b^2 \right)^2. $$
It is larger than $$ \left( b^2 - 1 \right)^2 = b^4 - 2 b^2 + 1 $$
as soon as
$$  2 b^2 - 8b - 9 > 0. $$
As  $$  2 \cdot 5^2 - 8 \cdot 5 - 9 = 1, $$ and this increases with $b,$
it is impossible for $  b^4 - 8 b - 8 $ to be a square when $b \geq 5.$
Your remaining possibilities are $b = 2,3,4.$
Comment: and $b=2$ of course is impossible since there is a 2 in the original equation :-)


How to derive the covariance matrix of $\hat\beta$ in linear regression?

Question: I just read this very insightful post about ridge regression, where the author stated that the variance of $\hat\beta$ is:
$$\text{var}(\hat\beta) = \sigma^2(\textbf{X}^\prime \textbf{X})^{-1}.$$
I couldn't figure out why it is like this. Can anyone elaborate a bit?
Answer: The covariance result you are looking at occurs under a standard regression model using ordinary least-squares (OLS) estimation.  The OLS estimator (written as a random variable) is given by:
$$\begin{equation} \begin{aligned}
\hat{\boldsymbol{\beta}}
&= (\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} (\boldsymbol{x}^{\text{T}} \boldsymbol{Y}) \\[6pt]
&= (\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} (\boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt]
&= \boldsymbol{\beta} + (\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} \boldsymbol{\varepsilon}.
\end{aligned} \end{equation}$$
In the standard linear regression model we have $\mathbb{E}(\boldsymbol{\varepsilon}) = \boldsymbol{0}$ and $\mathbb{V}(\boldsymbol{\varepsilon}) = \sigma^2 \boldsymbol{I}$  so that the estimator is unbiased with covariance matrix given by:
$$\begin{equation} \begin{aligned}
\mathbb{V}(\hat{\boldsymbol{\beta}})
&= \mathbb{V}((\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} \boldsymbol{\varepsilon}) \\[6pt]
&= ((\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} ) \mathbb{V}(\boldsymbol{\varepsilon}) ((\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} )^{\text{T}} \\[6pt]
&= \sigma^2 ((\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} ) \boldsymbol{I} ((\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} )^{\text{T}} \\[6pt]
&= \sigma^2 ((\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} ) ((\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} \boldsymbol{x}^{\text{T}} )^{\text{T}} \\[6pt]
&= \sigma^2 (\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1} (\boldsymbol{x}^{\text{T}} \boldsymbol{x}) (\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1}  \\[6pt]
&= \sigma^2 (\boldsymbol{x}^{\text{T}} \boldsymbol{x})^{-1}.
\end{aligned} \end{equation}$$
Note that this is the conditional covariance of the estimator given the design matrix $\boldsymbol{x}$.
Answer: Four things to note:
$\hat{\beta} =(\textbf{X}^\prime\textbf{X})^{-1}\textbf{X}^\prime\textbf{Y}$
$\text{var}({\textbf{A}\textbf{Y}})=\textbf{A}\text{var}(\textbf{Y})\textbf{A}^\prime$
$\text{var}(\textbf{Y}|\textbf{X})=\sigma^2\textbf{I}$ (actually, everything is conditioned on $\textbf{X}$)
$(\textbf{X}^\prime\textbf{X})^{-1}$ is symmetric.
Comment: $\text{var}(Y) \neq \sigma^2\textbf{I}$, unless $Y$ is the error term (or $\beta = 0$).
Comment: @jbowman thank you, I edited to condition on $\textbf{X}$


Introductory Statistics, Hypothesis testing vs CI

Question: Can someone please explain the difference between confidence interval and hypothesis tests.
For example in relation to estimating the mean, it seems to me that the main difference is that in a one sample test (α=0.05) you find the 2.5th percentile and 97.5th percentile of the distribution centered around the null mean and see if the alternative mean lies within it. This is in contrast to confidence intervals where you instead find the percentiles of the alternative hypothesis and see if the null mean is present in it. Is this correct?, this seems a little redundant to me. Is there some benefit of one over the other?
Comment: You don't ask any specific questions; your post simply makes a series of statements. Please identify something you specifically want answered about this situation.
Comment: I think I made it a little clearer, I was just asking about the difference between CI and one sample test. They both seem very similar.
Answer: Your simple explanation of a hypothesis test works, however for the CI, the interval is also constructed assuming the null hypothesis to be true, at least initially. Only when we have rejected the null hypothesis do we construct a CI around the effect. Smithson elaborates on this in the reference below.
Smithson, M. (2001). Correct Confidence Intervals for Various Regression Effect Sizes and Parameters: The Importance of Noncentral Distributions in Computing Intervals. Educational and Psychological Measurement, 61(4), 605–632. https://doi.org/10.1177/00131640121971392
Confidence intervals are another way to conduct hypothesis tests. In this instance, there is no "versus". Some researchers also use them to communicate the uncertainty about the effect obtained from their single study. In this instance, as Smithson elaborates, CI are used differently.
Answer: All values that fall into 95% confidence interval are considered to be plausible values for the parameter being estimated. If the value of the parameter lies within this interval, the null hypothesis cannot be rejected but if the values lies outside then we can reject the null-hypithesis. Lets say our 95% confidence interval lies between 10 and 20. The value of the parameter of the null-hypothesis is 21. We reject the null-hypothesis at the significance level of 5%.
Comment: Why is it though you have a CI centered around our estimated parameter and we see if null hypothesis falls within in it. In the one sample test isn't it the opposite, a CI around the null hypothesis and testing if the estimated parameter falls within in.
Comment: @RahimAhmed One sample test: Null-hypothesis is that loading time of a page is 5s. You have a sample with the mean of 9s and 95% confidence interval of [7,10]. 5 does not fall into the CI, meaning we can reject the Null-hypothesis, the loading time for this sample is not 5s. As Saurav mentioned in his answer, CI is always about your sample, whereas hypothesis testing is about the sample and population.
Answer: The hypothesis tests that we conduct are never about the sample statistics but always about the true population statistics. I'll try to make it as clear as possible through this example:
Suppose we have a huge population and we have to find out the average time one spends on the internet, lets call it u. But we only have a sample of size 50 of the same population from which we found out the fact that- the average time one person among these 50 people spends on the internet is x(our observed value).
So, my population average is u which is unknown to me.
And my sample average is x which we just found out.
So now suppose we have to perform a hypothesis test if my population average is suppose N.(this is my hypothesis since it is has been assumed and N is an arbitrary number).
i.e [Ho: u = N] and so my alternative shall be [Ha: u != N].
So u can see that it is always about the true population parameter(i.e average in our case) and never ever about the sample parameter or our observed value.
On the other hand, creating a confidence interval of this sample, suppose of 95%, what we really mean to say is this- If we take repeated samples of size 50 from this population and create their respective confidence intervals, we are 100% sure that 95% of these intervals will contain my true population mean i.e u .
CI = (x +  Margin Of Error, x -  Margin Of Error)
You can also see this: if my null value i.e  N (check the null hypothesis) doesn't fall within this interval, we can reject our null hypothesis right away.
This is the concrete idea behind these 2 methods.Don't mix them up. I hope this was helpful.
Answer: I think the key here is the effect. A hypothesis test should simply be an up/down vote, but a confidence interval gives us an idea of how different (or not) our results are from what we expected under the null hypothesis.


How to flatten multiple row values into concatenated string in postgres?

Question: I have a simple set of three tables . 2 "source" tables and a "join" table that allows one to many relationship.
I have this query:
          select data.mph1.data ->> 'name' as name, data.mph1.data ->> 'tags' as Tags,
          data.mph2.data ->> 'School' as school from data.mph1
          join data.mph1tomph2 on data.mph1tomph2.mph1 = data.mph1.id
          join data.mph2 on data.mph2.id = data.mph1tomph2.mph2

The output appears as:
            Name             Tags                                School
           "Steve Jones"    "["tag1", "tag2"]"                  "UMass"
           "Steve Jones"    "["tag1", "tag2"]"                  "Harvard"
           "Gary Summers"   "["java", "postgres", "flutter"]"   "Yale"
           "Gary Summers"   "["java", "postgres", "flutter"]"   "Harvard"
           "Gary Summers"   "["java", "postgres", "flutter"]"   "UMass"

What I'm looking for is
            Name             Tags                               School
           "Steve Jones"    "["tag1", "tag2"]"                  "UMass", "Harvard"
           "Gary Summers"   "["java", "postgres", "flutter"]"   "Yale, Harvard, UMass"

How would I get this result in single query? Possible?
Answer: Use the aggregate function string_agg()
select
    data.mph1.data ->> 'name' as name,
    data.mph1.data ->> 'tags' as tags,
    string_agg(data.mph2.data ->> 'School', ', ') as school
from data.mph1
join data.mph1tomph2 on data.mph1tomph2.mph1 = data.mph1.id
join data.mph2 on data.mph2.id = data.mph1tomph2.mph2
group by data.mph1.id  -- if id is a primary key
-- group by 1, 2       -- otherwise

Comment: another quick question. If the root table (data.mph1) had another multi-join to another table (data.someothertable) via another join table (data.mph1tosomeothertable), could I use string_agg() to create a list of those values as a new column?
Comment: Basically yes but the matter is more complex than it may seems. You should join the other table to already aggregated data. If this is not clear for you ask a new question (a full answer is too broad for a comment).
Comment: I am at the early stages of db design for new app, which requires support for extreme dynamic data structure. Based on your deep expertise, I'm wondering if you'd be open to paid session of db architecture review? we are struggling to figure out best way to model for such dynamic data. Would think 1-2 hours would be all that is needed to get guidance on direction.
Comment: Unfortunately, I won't find time. Currently I'm also  struggling with a project and look here for a change.
Comment: undertand, thx for replying. In regards to my question above, generally speaking, if a given table has N number of external multi-value joins like the example I provided, is it possible to dynamically construct a single query to returned those "N" attributes in a separate column with concatenated values? Would love to see the syntax for that.


Tratamento de dados em formato JSON com RegEx em JavaScript

Question: Estou com um problema em um projeto com Backend em PHP e front em HTML, JavaScript e CSS.
O backend está mandando os dados para o front através de JSON, porém, em um dos trechos dos dados há textos com aspas, exemplo:
{
   "descrição": "Eu faço trabalhos "fáceis" porém cansativos"
}

O JSON por padrão reconhece as aspas de "fáceis" como final do dado, e gostaria que não fosse assim.
Posso dar um:
myJson.replace(/"/g,'"');

Mas aí ele irá trocar todas as aspas do JSON, gerando um input assim:
{
   "descrição": "Eu faço trabalhos "fáceis" porém cansativos"
}

O que quebra tudo.
Estou buscando escrever uma regex que substitui as aspas por " apenas no caso de estarem dentro de um campo de dados do JSON. Ou há alguma outra forma de tratar este caso?
Comment: Se o JSON está com as aspas exatamente assim, então o erro está no backend, que está gerando um JSON inválido, e é lá que deveria ser corrigido. Tentar arrumar com regex não é simples ([não use regex para manipular JSON](https://hkotsubo.github.io/blog/2019-04-13/como-ler-e-manipular-um-json#regex) - para um válido [já é difícil](https://pt.stackoverflow.com/a/357086/112052), para um inválido, é pior ainda). Provavelmente vc vai ter que manipular a string manualmente mesmo, já que geralmente os parsers dão erro qdo o JSON está inválido (ou corrija onde ele é gerado, que é o mais indicado)
Comment: Saulo, seria interessante você adicionar na pergunta o código que gera o JSON, ou pelo menos informar qual linguagem você está usando para gerar este JSON.
Answer: Como eu já disse nos comentários, se o JSON está vindo com as aspas exatamente dessa maneira, então ele não é um JSON válido. Nesse caso, o melhor é corrigir no backend, para que ele gere o JSON corretamente - no caso, com as aspas escapadas (\"):
{
    "descrição": "Eu faço trabalhos \"fáceis\" porém cansativos"
}

Tratar o problema na origem garante que quem for receber esses dados não precisará se preocupar em arrumá-los, até porque não é uma tarefa tão simples assim.
Como você disse que o backend é PHP, verifique se está usando corretamente a função json_encode (que é a maneira mais simples para gerar JSON no PHP). Ou, caso esteja usando outro mecanismo/API/framework, verifique se tudo está corretamente configurado, se os parâmetros estão corretos, etc, pois é mais provável que o problema esteja aí.
O restante da resposta abaixo é somente para mostrar como usar regex pode ser uma solução ruim, mais complicada e até desnecessária, caso você corrija o problema na origem.

Veja por exemplo como ficaria uma regex para o seu caso (não vai funcionar em todos os browsers):

let s = `{
   "descrição": "Eu faço trabalhos "fáceis" porém cansativos",
   "teste": "Aqui não tem aspas a mais",
   "teste2": "Aqui " tem várias " aspas a mais""
}`;
let r = /(?<!"\s*:\s*)"(?![\n\r,:]|[^"]+":)/g;
console.log(s.replace(r, '"'));

Basicamente, ela pega as aspas, levando em consideração vários fatores:
O negative lookbehind (?<!"\s*:\s*) verifica se algo não existe antes das aspas. No caso, temos uma aspas, seguida de \s* (zero ou mais espaços), dois pontos, zero ou mais espaços. Ou seja, não pode ser a primeira aspas logo depois do :.
Obs: o lookbehind atualmente só funciona no Chrome. Mas mesmo que você use outra linguagem - que não JavaScript - que suporte este recurso, ainda sim vale ler o restante da resposta.
O negative lookeahead (?![\n\r,:]|[^"]+":) verifica se algo não existe depois das aspas. No caso, é [\n\r,:] (uma quebra de linha, ou vírgula, ou dois pontos). Assim, eu não considero as aspas de fechamento. Mas o | (que significa ou) admite outra possibilidade: [^"]+: - um ou mais caracteres que não sejam aspas, seguido de dois pontos (sem isso, a regex também pegava a primeira aspas de cada linha).
Basicamente, todas essas regras são para desconsiderar as aspas de abertura e fechamento legítimas. Mas esta regex não cobre todos os casos.

Por exemplo, se tivermos um array, não funciona mais:

let s = `{
   "lista": [ "Eu faço trabalhos "fáceis" porém cansativos" ]
}`;
let r = /(?<!"\s*:\s*)"(?![\n\r,:]|[^"]+":)/g;
console.log(s.replace(r, '"'));

Nesse caso, ele substitui todas as aspas dentro do array. Então precisamos colocar mais condições na regex para indicar a nova condição. Por exemplo, eu poderia indicar que deve ignorar as aspas logo depois do [ ou logo antes do ]:

let s = `{
   "lista": [ "Eu faço trabalhos "fáceis" porém cansativos" ]
}`;
let r = /(?<!"\s*:\s*|[\[,]\s*)"(?![\n\r,:]|[^"]+":|\s*\])/g;
console.log(s.replace(r, '"'));

Mas ainda há casos em que ela pode falhar. Por exemplo, se dentro da string tivermos algum desses caracteres, como : ou []:

let s = `{
   "fad": "fasdfa "fa" : "fasdfasd"sdfa",
   "xyz": [ "af [ "xyz " ad", " fasd "fasf" dfs"]
}`;
let r = /(?<!"\s*:\s*|[\[,]\s*)"(?![\n\r,:]|[^"]+":|\s*\])/g;
console.log(s.replace(r, '"'));

Agora, como dentro das strings eu tenho os caracteres : e [] (que eu tinha usado como os pontos de referência para saber se estou no início ou fim da uma string), a regex se perde, pois ela não está verificando se deveria estar dentro de uma string ou não.
Eu até acredito que seja possível prosseguir e incluir esta alteração, mas acho que a regex já está complicada o suficiente e não vale mais a pena.

Tudo isso é para mostrar que talvez não valha a pena tentar corrigir o JSON com regex. Tente arrumar o JSON onde ele é gerado, em vez de criar um problema maior ao tentar resolvê-lo com regex.
Comment: Exato, a resposta é boa, meu medo é que seja usada no lado negro da força. :P
Por via das dúvidas vou perguntar para o AP qual é o código que gera o JSON, pode evitar que uma boa alma use regex pra isso no futuro.
Comment: Cara, duvido que uma biblioteca nativa de qualquer linguagem de backend gere um JSON com um erro tão básico destes... Enfim, se o AP tiver interesse em uma solução para o problema real dele, só vai depender dele agora. :D Toma meu +1
Comment: A resposta é muito boa, mas acho que a pergunta ainda não merecia uma resposta, pois do jeito que está, o AP vai usar regex pra resolver um problema que deveria ser feito no backend (mesmo você tendo advertido para não fazer isso)...
Comment: @fernandosavio Minha intenção era mostrar como a solução com regex é complicada e não vale a pena, mas o que vc falou faz sentido. Se bem que, seguir ou não a recomendação já é uma decisão do AP, pois o aviso foi dado... Mas creio que basta surgir um JSON um pouquinho mais complicado para ele perceber que de fato não vale a pena...
Comment: @fernandosavio É, esqueci de perguntar isso... Mas se o backend é PHP, eu acho que ele não deve estar usando `json_encode`, pois não vejo uma situação em que ele gera um JSON inválido assim... De qualquer forma, coloquei mais alguns avisos no início da resposta..


Difference between Means is the Mean of the Differences

Question: Suppose you have a sequence $a_{1}, \dots, a_{n}$ and another sequence $b_{1}, \dots, b_{m}$, where $n$ and $m$ may differ. I want to know whether the difference between the arithmetic means of these sequences equals the arithmetic means of the differences between the terms, in the following sense:
$${1 \over n} \sum \limits_{i=1}^n a_{i} - {1 \over m} \sum \limits_{j=1}^m b_{i} = {1\over n} \sum \limits_{i=1}^n {1 \over m} \sum \limits_{j=1}^m a_i - b_j$$
I'm not sure that I expressed that right. But here is an example of what I have in mind. Suppose you have two lists of numbers:
$$A = (1, 2, 3)$$
$$B = (1, 3)$$
The difference between the means is $${1+2+3 \over 3} - {1+3 \over 2}=0.$$ What I had in mind as the mean of the differences was something like $${{(1-1)+(1-3) \over 2}+{(2-1)+(2-3) \over 2} +{(3-1)+(3-3) \over 2} \over 3} = 0$$
I'm guessing that these two things are always equal. But I'm wondering how to prove it. (I assume the answer is really simple.) Thanks!
Answer: It'is always the case as:
$${1\over n} \sum \limits_{i=1}^n {1 \over m} \sum \limits_{j=1}^m (a_i - b_j)={1\over n} \sum \limits_{i=1}^n {1 \over m} (\sum \limits_{j=1}^m a_i - \sum \limits_{j=1}^m b_j)={1\over n} \sum \limits_{i=1}^n {1 \over m} (m a_i - \sum \limits_{j=1}^m b_j)={1\over n} \sum \limits_{i=1}^n (a_i - {1 \over m}\sum \limits_{j=1}^m b_j)={1 \over n} \sum \limits_{i=1}^n a_{i} - {1 \over m} \sum \limits_{j=1}^m b_{i}$$
Comment: Ah, that makes sense. Thanks for this!