Common strategy in handling concurrent global 'inventory' updates

Question: To give a simplified example:
I have a database with one table: names, which has 1 million records each containing a common boy or girl's name, and more added every day.
I have an application server that takes as input an http request from parents using my website 'Name Chooser' . With each request, I need to pick up a name from the db and return it, and then NOT give that name to another parent. The server is concurrent so can handle a high volume of requests, and yet have to respect "unique name per request" and still be high available.
What are the major components and strategies for an architecture of this use case?
Answer: From what I understand, you have two operations: Adding a name and Choosing a name.
I have couple of questions:

Qustion 1: Do parents choose names only or do they also add names?
Question 2 If they add names, doest that mean that when a name is added it should also be marked as already chosen?

Assuming that you don't want to make all name selection requests to wait for one another (by locking of queueing them):
One solution to resolve concurrency in case of choosing a name only is to use Optimistic offline lock.
The most common implementation to this is to add a version field to your table and increment this version when you mark a name as chosen. You will need DB support for this, but most databases offer a mechanism for this. MongoDB adds a version field to the documents by default. For a RDBMS (like SQL) you have to add this field yourself.
You havent specified what technology you are using, so I will give an example using pseudo code for an SQL DB. For MongoDB you can check how the DB makes these checks for you.
<code>NameRecord {
  id,
  name,
  parentID,
  version,
  isChosen,

  function chooseForParent(parentID) {

    if(this.isChosen){
      throw Error/Exception;
    }

    this.parentID = parentID
    this.isChosen = true;
    this.version++;
  }
}

NameRecordRepository {

  function getByName(name) { ... }

  function save(record) {

    var oldVersion = record.version - 1;

    var query = "UPDATE records SET .....
                 WHERE id = {record.id} AND version = {oldVersion}";

    var rowsCount = db.execute(query);

    if(rowsCount == 0) {
      throw ConcurrencyViolation
    }
  }
}

// somewhere else in an object or module or whatever...

function chooseName(parentID, name) {

   var record = NameRecordRepository.getByName(name);

   record.chooseForParent(parentID);

   NameRecordRepository.save(record);
}
</code>
Before whis object is saved to the DB a version comparison must be performed. SQL provides a way to execute a query based on some condition and return the row count of affected rows. In our case we check if the version in the Database is the same as the old one before update. If it's not, that means that someone else has updated the record.
In this simple case you can even remove the version field and use the isChosen flag in your SQL query like this:
<code>    var query = "UPDATE records SET .....
                 WHERE id = {record.id} AND isChosend = false";
</code>
When adding a new name to the database you will need a Unique constrant that will solve concurrenty issues.


Creating a class for checking if the value is null or not in sqlserver

Question: I will do a lot of thing like this, can someone help me to convert this into a class?
private void CheckingForCostID()
{
    //if CostID is exist then error.
    newConnection.ConnectionM();
    SqlCommand cmd = SqlConnectionOLTP.cn.CreateCommand();
    cmd.CommandText = "SELECT CostCategoryID FROM CostCategory WHERE CostCategoryName =@costcategoryname AND Description =@description";
    cmd.Parameters.AddWithValue("@costcategoryname", format.SetText(textBoxCostName));
    cmd.Parameters.AddWithValue("@description", format.SetText(textBoxCostDescription));
    var costid = cmd.ExecuteScalar();
    if (costid == null)
    {
        CreateCostCategoryData();
    }
    else
        MessageBox.Show("Cost Category already exist", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
    SqlConnectionOLTP.cn.Close();
}

Answer: create 2 classes first for the entity model (CostCategoryModel)
the second class CostCategoryRepository.
In the first your define the structure of you object CostCategory
in the second your define CRUD methods and your utilities methods like
public bool Exist(string name, string description){...}

then you can use this sample code pattern
CostCategoryRepository repo = new CostCategoryRepository();
if(repo.Exist(name,description))
{
...
}else{
...
}

Comment: Newing up repositories? Someone hasn't heard of unit testing, dependency injection, transactions, units of work, rollbacks. That is poor advice. It should be an injected `IRepository`, at the very least.
Comment: he must change this piece of code, he not refactoring all project structure my solution is not the best way but resolve his question.


Go to definition not working in VS Code version 1.47.2

Question: I'm using VS Code with Angular 8 and TypeScript 3.9.6 and suddenly when I'm on a template, ctrl+click or F12 to go to definition doesn't work any more. I got the following pop up message:
]
I tried to uninstall and reinstall but was not able to solve the issue.
These are actually installed extensions:
<code>Angular.ng-template
Anjali.clipboard-history
byi8220.indented-block-highlighting
CoenraadS.bracket-pair-colorizer-2
cyrilletuzi.angular-schematics
dbaeumer.vscode-eslint
eamodio.gitlens
EditorConfig.EditorConfig
eg2.vscode-npm-script
esbenp.prettier-vscode
FallenMax.mithril-emmet
infinity1207.angular2-switcher
johnpapa.angular-essentials
johnpapa.Angular2
johnpapa.vscode-peacock
johnpapa.winteriscoming
ms-azuretools.vscode-docker
msjsdiag.debugger-for-chrome
msjsdiag.debugger-for-edge
nick-rudenko.back-n-forth
nrwl.angular-console
PKief.material-icon-theme
</code>
Answer: I found in July 2020 that downgrading the vs-code 'angular language service' to version 0.1000.3 fixes the "Go to definition" functionality.
Here is an image showing where to downgrade:
Comment: I'm using version 0.901.11 of angular language service ( june 10 2021 ). With angular 11.


Question about the equality of variances of two populations

Question: When finding the confidence interval for the mean difference between two groups of sample, under which assumption about the equality of their population variances should we take if no information about their population variances is given?
Is it correct that if 1 is within the 98% confidence interval for the ratio of variance, then we can assume equal population variance when finding the confidence interval for the mean difference? At least that is what I understand from my lecture notes:]2
Comment: Additionally, deciding which test to conduct based on the result of an equality of variance test leads to incorrect p values. So if you doubt equality of variance, just use a method that is robust to such assumptions. Google 1. Zimmerman t test variance or 2. Lakens Welch test for references for my first claim.
Comment: Without knowing the nature of your data or the sample sizes involved, it is impossible to answer. But ordinarily, you shouldn't use methods that assume equal variance, if you have doubts about that assumption. Methods for 'detecting' unequal variances generally have poor power, so it may be risky to use your confidence interval as assurance of equal variances.
Comment: @BruceET Could you please kindly read the edited post with added information and explain?
Comment: @HeteroskedasticJim Thank you for the references.
Comment: See Addendum to my Answer.
Answer: In R, the procedure var.test implements the standard F-test for equality of variances for data from two normal populations.
Here is an example, using samples of size $n = 20$ from two normal populations with respective variances $\sigma_1^2 = 49$ and $\sigma_2^2 = 25.$  The null hypothesis of equal variances is not rejected at the 5% level.
set.seed(2019)
x = rnorm(20, 50, 7);  y = rnorm(20, 50, 5)
var.test(x, y)

        F test to compare two variances

data:  x and y
F = 1.8786, num df = 19, denom df = 19, p-value = 0.1785
alternative hypothesis:
   true ratio of variances is not equal to 1
95 percent confidence interval:
 0.7435627 4.7461266
sample estimates:
ratio of variances
          1.878575

The figure below shows the density function of $\mathsf{F}(19, 19),$ with vertical dotted lines at the critical values for a two-sided
test at the 5% level. The observed variance ratio $F = 1.8786$
lies between them.

This is not really an unusual case. According to the simulation below, the power of such a test is only about 30%.
set.seed(1234)
pv = replicate(10^5,
       var.test(rnorm(20, 50, 7), rnorm(20,50,5))$p.value)
mean(pv < .05)
[1] 0.29359

Addendum: I'm not sure what is left to explain at this point,
but I will make a few comments and show some output from R.
(a) As I have hinted and @HeteroskedasticJim has documented: Whenever the goal is to test whether two normal population means are equal and there is doubt about equality of population variances, the recommended procedure is to do a Welch two-sample t
test of the null hypothesis that population means are equal.
[Thus I suppose both of us disagree with the boxed instruction with the "!" in you Question.]
For your data, that null hypothesis cannot be rejected. You can find formulas for the Welch test in you text, class notes, or online.
x = c(12,11,7,13,8,9,10,13);  y = c(13,11,10,6,7,4,10)
t.test(x,y)

        Welch Two Sample t-test

data:  x and y
t = 1.1583, df = 10.771, p-value = 0.2718
alternative hypothesis:
   true difference in means is not equal to 0
95 percent confidence interval:
 -1.503134  4.824562
sample estimates:
mean of x mean of y
10.375000  8.714286

(b) If you had done a pooled two-sample t test (which requires the assumption that population variances are equal), then you would have $T = 1.1851, df = 13,$ and P-value = 0.2572, so again no rejection. (However as mentioned above, it is not recommended to do a 'composite test', first testing for equal variances; and if failing to reject, going on to do the pooled two-sample tests for equal means. The overall significance level of this rwo-test process is unclear.)
(c) If you use R's var.test procedure, mentioned in this Answer above, you can request a 98% CI, which agrees with what you say in your Question. The fact that
this CI includes $1$ does lead one to believe that population variances are equal (but it would be more direct to test the hypothesis that the ratio of population variances is 1.)
var.test(x,y, conf.lev=.98)

        F test to compare two variances

data:  x and y
F = 0.51743, num df = 7, denom df = 6, p-value = 0.4087
alternative hypothesis:
    true ratio of variances is not equal to 1
98 percent confidence interval:
 0.06264264 3.72103337
sample estimates:
ratio of variances
         0.5174279

The overall conclusion is that the data do not show convincing
evidence of a difference between the two fertilizers--neither in terms of their
population means nor in terms of their population variances.
The lack of significance is unsurprising on account of the small
sample sizes.


Straight to checkout with commerce

Question: So I have created a product in Commerce: a £15 one-off donation.
I don't need a product page for this (I have a custom block with links for different donation amounts). When you click a link, I want to use the product ID, and push the product straight to checkout.
How would I go about this?
Answer: The way to programatically add a product to the cart and send a user to checkout would be to use the function:
function commerce_cart_product_add_by_id**($product_id,
                        $quantity = 1, $combine = TRUE, $uid = NULL)

and
function drupal_goto**($path = '', array $options = array(),
       $http_response_code = 302)

to redirect the user to checkout.
Here is information on the functions:

commerce_cart_product_add_by_id
drupal_goto
Comment: One way to do this is to create a custom module and create a new page via hook_menu. Have the links reference the product id (For instance: http://www.yoururl.com/yourproductaddingpage?id=22)and use arguments through hook_menu to relay the product id to the function itself. When the function receives the product the product is added to the cart and the drupal_goto function sends the user to checkout.
Comment: These functions work thanks. Further to this, because I'm handling one-off donations the cart is redundant (and causes confusion!), is there a way to force one product through checkout without using the cart?
Comment: I did this on my last site, I just used a hook to disable it for users that didn't have certain permissions. Disabling the cart than would allow you to do your redirect directly to checkout and the user would never see that there was a cart. You would also want to remove the drupal_set_message that tells the user that the product was added to cart.

The hook:
function hook_menu_alter(&$items){

  $items['cart']['access arguments'] = array('administer checkout');
  $items['cart/my']['access arguments'] = array('administer checkout');

    //unset($items);

}
Unsure where the message is
Comment: This is great through `template.php`, but how would I go about achieving this through a simple anchor link? Can I somehow call the functions on a condition?
Answer: This question popped updated because of being edited and I thought I would share how to do to this via module. I wrote a custom module to do just this because we have a lot of direct marketing and I often need the ability to throw someone directly into a checkout process via URL. This module takes a URL constructed of the SKU and the quantity and puts the customer into a checkour process. The URL is http://www.example.com/product/add/SKU/QUANTITY. As an example SKU 101 at a quantity of 2 would be:
http://www.example.com/product/add/101/2

Here is the module:
https://github.com/taz77/drupal-add-to-cart-url
Comment: This module also allows the passing of tracking query strings for Google Analytics and the Affiliate NG module. It preserves any query sting basically.

Straight to checkout with commerce


await fetch in flask not getting required return

Question: I'm trying to use await fetch in JS to use a function that I created in Flask.
My code for the JS is:
let dictionary = await fetch("/getName");
let user_name = await dictionary.name;

and my code in flask is:
@app.route("/getName")
def getName():
    output = {"name": ""}
    print("hello")
    if NAME_KEY in session:
        output["name"] = session[NAME_KEY]
        print(output)
    return jsonify(output)

why does this not work?
to be specific to causes the website to have a 400 bad request.
I removed the await feature from both of the variables, and instead it creates the error:

'Request' object has no attribute 'is_xhr'

I downgraded werkzeug to 0.16.1, and no nothing just shows up. There aren't any errors, but nothing comes and nothing gets printed by the system.
Here is my new code for JS. (Flask stays the same)
let dictionary = fetch("/getName");
console.log("start");
console.log(dictionary);
let user_name = dictionary.name;

Comment: If the answer below doesn't work then I would edit your question adding a bit more code/context that shows how your await code is being called.
Comment: @Luke thanks a lot, your suggestion down below works perfectly! if possible can you send me the webpage / videos that help me understand why stuff like this is required?
Comment: The short (too simplistic) answer is that JavaScript doesn't wait for one line of code to finish executing before executing the next. This is a problem because there are two common situations (DB queries and Web API calls) when it takes a while for a line of code to complete, but when you want it to complete before executing the next line of code.  To ensure that JS code does execute in an ordered sequence there are various techniques you can use. These include using async and await, or callbacks, or promises or the ".then" syntax which is popular when using fetch.
Comment: As an aside (and this is just my opinion) JavaScript is celebrated because of the fact that it doesn't wait for one line of code to execute before executing the next. Supposedly it makes things faster.  But I think that behavior is overrated because it introduces all these complications and associated techniques that aren't needed in most other languages.  For example, when making a DB call in the Flask/Python environment you won't see any of the above await/async techniques being used. The code just executes in an intuitive step by step fashion.  Javascript's convolutions are totally avoided.
Comment: @Luke hmm sounds interesting, however if it does truly make a big difference to performance then it should be "celebrated". Thanks for the help!
Answer: Most likely you haven't wrapped the await code inside a function that has been declared with async.  Or you have declared the async function incorrectly.  This example works:
Your Flask view/route:
@app.route('/getName')
def get_name():
    print("in getName view....")
    output = {"name": ""}
    return jsonify(output)

Your client side code calling the above /getName route:

    <title>REST</title>

    <button onclick="getName()">getName</button>

For a full working example (e.g. including a Flask app.py file and a Flask template) see:
https://github.com/lfernandez55/REST-auth/tree/stack-overflow-67606023
Comment: This Jsbin also demos how you can make a fetch GET call using the then syntax, or alternatively, using the await syntax you are working with: https://jsbin.com/ciziyar/edit?html,js,console,output .
Comment: Also,  in my experience the fetch .then syntax is more popular than using fetch with await.  If you want to see your fetch example using then rather than await,  look back one commit in the above branch.
Answer: Decoupling the javascript code from html (for searching the input user name in session dictionary), the above code can be broken into the following 3 files, making the code for the flask app a little more modular (with the following folder structure):

## app.py
from flask import Flask, jsonify, request, render_template
import os

# initialization
app = Flask(__name__)

# Route for handling session search
@app.route('/get_name', methods=["POST"])
def get_name():
    session_dict = {'coder', 'dev', 'test'} ## dummy dictionary as placeholder
    user_name = request.json["user_name"]  # Get the user input from the request
    print(f"checking if {user_name} is in session...")
    exists = f"{user_name} exists in session" if user_name in session_dict else f"{user_name} does not exist in session"
    output = {"name": exists}
    return jsonify(output)

# Routes for the home page
@app.route("/")
@app.route("/index.html")
def index():
    return render_template("index.html")  # Render the index.html template

if __name__ == '__main__':
    app.run(debug=True)

// app.js
const form = document.getElementById("rest-form");
const user_input = document.getElementById("user-name");
const response_elem = document.getElementById("name-exists");

form.addEventListener("submit", async (event) => {

    event.preventDefault();

    const user_input_value = user_input.value.trim();
    if (!user_input_value) {
        return;
    }

    try {
          const resp = await fetch("/get_name", {
            method: "POST",
            headers: {
              "Content-Type": "application/json",
            },
            body: JSON.stringify({ user_name: user_input_value })
          });
          const response_data = await resp.json();
          if (resp.ok) {
            const chat_response = response_data.name;
            response_elem.value = chat_response;
          } else {
            alert("Error when fetching response");
          }
    } catch (error) {
        console.log('Fetch error: ', error);
        alert(error)
    }

 });

<!DOCTYPE html>

        <title>Fetch Demo</title>

Fetch Demo

        <form id="rest-form" >

            User-Name:
            <input type="text" id="user-name" name="user-name">

           Response:
           <input type="text" id="name-exists" name="name-exists" readonly>

          <button type="submit" id="submit-button">Fetch</button>
        </form>

The next animation shows the demo, note that the dummy users coder, dev, test are already present in the demo session dictionary, hence searching them with await fetch succeeds (and the corresponding text is displayed in the response text box).
Also, notice that the POST method is used for the route get_name. While defining the Event listener for the submit button the event is wrapped with the async keyword in javascript code.


How to programmatically create new file/folder in Drupal 7 files directory?

Question: I tried to create new file/folder programmatically inside Drupal's sites/default/files. But I'm getting this error: "Can't Open file".
Here is what I'm doing: When my registered user visits a certain page, I will be generating a new file and I will provide them with the download link. When I tried to create a new file I am getting "can't open file" message. I'm in localhost in MS Windows.
Are there any other writable folder other than sites/default/files for doing this stuff?
Below is my code:
<code>$ourFileName = base_path()."/sites/default/files/members-pdf/testFile.txt";
$ourFileHandle = fopen($ourFileName, 'w') or die("can't open file");
fclose($ourFileHandle);
</code>
Above code is written inside a views's template file.
I want to use PHP file API and not Drupal 7 file API.
Comment: Why do you want to use the PHP file API and not the Drupal file API? The latter is just a set of handy wrappers around the former.
Comment: The part `base_path()."/sites/default/files/members-pdf/testFile.txt"` doesn't seem right, because base_path() will at least return a single slash. Try if `base_path()."sites/default/files/members-pdf/testFile.txt"` works better.
Comment: For what it's worth: `$ourFileName = $_SERVER['DOCUMENT_ROOT'] . '/sites/default/files/members-pdf/testFile.txt';` should work fine assuming your web server can write to that folder, and your Drupal installation is in the web root (not in a sub-folder)
Comment: @marcvangend thanks it worked for me. I removed the slash and the base_path(). Just "sites/default/files/members-pdf/testfile.txt" worked for me.
Comment: @Clive thanks clive. i am about to use TCPDF for creating a pdf file which uses only php file api. And also your solution worked for me. In my case my drupal installation is in subdirectory so i have to add subdirectory name after $_SERVER['DOCUMENT_ROOT'].
Answer: To create folders you can use file_prepare_directory and pass FILE_CREATE_DIRECTORY as second parameter like so:
<code>$new_folder = 'public://newfolder/';
file_prepare_directory($new_folder, FILE_CREATE_DIRECTORY);
</code>


Removing All Items From A ComboBox?

Question: How can I programmatically remove all items from a combobox in VBA?
Answer: Psuedo code ahead (updated with actual code):
Do While ComboBox1.ListCount > 0
    ComboBox1.RemoveItem (0)
Loop

Basically, while you have items, remove the first item from the combobox.  Once all the items have been removed (count = 0), your box is blank.
Method 2:  Even better
ComboBox1.Clear

Comment: @Sham Yemul: I can confirm ComboBox.Clear does indeed work, at least as of Excel 2010 on the ActiveX ComboBox control.
Comment: Access ComboBox and ListBox do not have `.Clear` methods too (current Access 365 version), but `lst.RowSource = vbNullString` removes all items. Otherwise the `Do ... Loop` seems the most beautiful workaround to me.
Comment: Can also confirm the Microsoft 365 Access Combo Box does not support `Clear`. Throws a compile error: `Method or data member not found`.
Comment: Sorry, VBA does not provide clear method for a combo box control.
Comment: What version of excel are you using, because. Clear works for me in 2016.
Answer: You need to remove each one individually unfortunately:
       For i = 1 To ListBox1.ListCount

           'Remove an item from the ListBox using ListBox1.RemoveItem
       Next i

Update - I don't know why my answer did not include the full solution:
For i = ListBox1.ListCount - 1 to 0 Step - 1
        ListBox1.RemoveItem i
Next i

Comment: Just pointing out (after struggling with it like an idiot) that you are removing the top item each time. If you try to make it `i` then you get a lovely out of bounds error about halfway through.
Comment: @PsychoData meaning that you can do `For i = ListBox1.ListCount to 1 Step -1` and it should work fine.
Comment: make that `For i = ListBox1.ListCount - 1 to 0  Step - 1 ListBox1.RemoveItem i Next`
Answer: The simplest way:
Combobox1.RowSource = ""  'Clear the list
Combobox1.Clear           'Clear the selected text

Answer: You can use the ControlFormat method:
ComboBox1.ControlFormat.RemoveAllItems

Answer: Best Way:
Combobox1.items.clear();

Answer: For Access VBA, if a ComboBox has been populated with a Row Source Type of Value List, I find the following works:
ComboBox.RowSource = ""

Answer: For Access VBA, which does not provide a .clear method on user form comboboxes, this solution works flawlessly for me:
   If cbxCombobox.ListCount > 0 Then
        For remloop = (cbxCombobox.ListCount - 1) To 0 Step -1
            cbxCombobox.RemoveItem (remloop)
        Next remloop
   End If

Comment: This method will only work when combobox source is set to valuelist
Answer: In Access 2013 I've just tested this:
While ComboBox1.ListCount > 0
    ComboBox1.RemoveItem 0
Wend

Interestingly, if you set the item list in Properties, this is not lost when you exit Form View and go back to Design View.
Answer:
me.Combobox1.Clear

This is the common method
Answer: I could not get clear to work.  (Mac Excel)
but this does.
ActiveSheet.DropDowns("CollectionComboBox").RemoveAllItems
Answer: If you want to simply remove the value in the combo box:
me.combobox = ""
If you want to remove the recordset of the combobox, the easiest way is:
me.combobox.recordset = ""
me.combobox.requery

Answer: Private Sub cmdClear_Click()
    ComboBox1.Value = Null
    ComboBox2.Value = Null
End Sub

Removing All Items From A ComboBox?


How to find out hazard ratio and confidence interval from LASSO cox regression ans plot momogram in R?

Question: I am working on a prognostic model based on time to event endpoint. The training data consists of 800 participants and test data around 400. The number of variables is 21. I was using glmnet package.
I can easily get the coefficients at minimum lambda. However, I read some articles where they have used a perturbation method to calculate the hazard ratio and 95% confidence intervals and then plot a nomogram. Can someone please help me with this. I would appreciate any help with the concept and how to apply that with R.
Answer: Your dataset is too small by a factor of >10 for split sample validation to be reliable.  It is also too small for feature selection to be reliable.  Consider data reduction (unsupervised learning), standard fitting, and resampling validation (100 repeats of 10-fold cross-validation or the bootstrap) as detailed in RMS.  Then you can use nomogram() in the R rms package.
Comment: Prof. Harrell, thank you very much for your time and valuable opinion on this. I will follow your advice. At the same time it would be great if you could also guide me on how to estimate HR with 95%CI from LASSO or adaptive LASSO in R...thanks again.
Comment: Without using the Bayesian lasso it is a difficult problem.   But the inventors of lasso have made a lot of progress in the past few years.   Look for papers by Tibshirani.


Why are cached images from Firebase Storage slower than others?

Question: I was experimenting with getting images from Firebase Storage as opposed to hosting them as an asset alongside my Angular project. I am hosting the project with Firebase as well.
I found that after an image is cached, it still takes upwards of 400 ms to load an image that was previously retrieved from Firebase Storage. Whereas a cached image that was hosted alongside the website takes around 10 ms.
I was under the impression that cached images were saved locally on the client, so why the huge difference in loading times?
Image is a 240 KB jpeg.
Network request for hosted asset (cached)

Network Request for Firebase Storage (cached)

I've read this question: Firebase Storage very slow compared to Firebase Hosting
But my question is in regards to image caching, so I didn't think the CDN would help anything.
I have solved this issue by making the image public following these instructions: https://cloud.google.com/storage/docs/access-control/making-data-public the cached image now loads in about 15 ms.
I'd still like to know why a cached image would take 400 ms to load, I think there is something I don't understand about caching.
Answer: It is expected that it is longer for Firebase Storage because the Firebase access is doing more work, and it does an authorization check to make sure that it has a token that proves the access was allowed by the Security Rules. Additionally, you can check on Caching data using the Cache-Control that.
Regarding to:

I have solved this issue by making the image public...

It's documented here, publicly cacheable objects can do better in terms of the performance.
Additionally, the extra time also depends on what region the user is in. One thing you can also do is to raise a ticket with the Firebase Support so they can further check if the particular request are coming into a region that is particularly slow.
Comment: Awesome, thanks. I'll look into the caching metadata as well. When it comes to private images, is it not possible to cache these publicly on a client that's received them previously? I don't think a client should need to go through security rules at every request, but the caching metadata article says it only applies to publicly accessible objects.
Comment: I suggest that you open another question for your concern.


Service worker Failed to load resource: net::ERR_UNSAFE_REDIRECT

Question: would you please tell me what i did wrong with my service worker installation, i have those errors appear in chrome console (see the image-1).
The script resource is behind a redirect, which is disallowed.                          service-worker.js:1

Failed to load resource: net::ERR_UNSAFE_REDIRECT          https://harampress.com/post/45/service-worker.js

i use Laravel framework in backend if that could help
just to know that my service-worker.js is in the root of the website, works fine on home page no errors on chrome console, but the moment i try to navigate to a post for example, those two errors appears (see image-2).
knowing that i use sw-precache and sw-toolbox, i generate the service worker useing sw-precache with gulp.
image-1
image-2
Answer: see the solutions here on github discussion
the message from the discussion in github that actually answered to my problem :

What @wanderview means is that when you do
navigator.serviceWorker.register('/sw.js')

The actual requesting URL is the sw.js located in the root
but when you do
navigator.serviceWorker.register('sw.js')

This is actually a Relative URL so what you actually request might be /post/blah/sw.js and your server might treat it as 404 and do some redirect.
Comment: Please include the answer here on SO


Is it possible to add an entity to the current route parameters?

Question: Using \Drupal::routeMatch()->getParameter(), it's possible to fetch objects that come from the parameters of the URL.
I'm wondering if it is possible to add values to the routeMatch. For example, if using the Groups module, nodes may be part of a group, however the group is not part of the path. Is there a way to load the group from the node, and add the group to the parameters that can be retrieved with \Drupal::routeMatch()->getParameter()?
Or, more broadly, I'd like to set the $group to be part of the "context" of a page request on a node page.
Comment: No, this is not necessary. The node instance is referenced in the route match and you can get the group info anytime you want with no performance hit. Whether you need a group cache context depends, because on a node page the main page content already depends on the more specific node and a more general cache context wouldn't make a difference. Unless you have page elements like blocks which only depend on the group.
Comment: It's the latter - I have blocks with are dependent upon the group.
Answer: To use it in a block you could create a custom route.node_group context which gets the context from the route match. Hierarchically you have to place it under the route context. When merging contexts, for example if the context bubbles up to the page level, route can replace route.node_group.
Comment: Thanks 4x4. I upvoted it.


Coefficient of $x^7y^6$ in $(xy+x+3y+3)^8$

Question:
Find the coefficient of $x^7y^6$ in $(xy+x+3y+3)^8$.

My solution:
Factor $(xy+x+3y+3)^8$ into $(x+3)^8(y+1)^8$. To get an $x^7y^6$ term, we need to find the coefficient of $x^7$ in the first factor and $y^6$ in the second factor. Using the binomial theorem, we get the coefficient of $x^7$ to be $17496$ and $y^6$ to be $28$. Multiplying the two gets us an answer of $489888.$
However, this is wrong. This is the answer key's approach:

$x^7y^6 = (xy)^6 \cdot x = (xy)^5 \cdot x^2 \cdot y$. Now, $(xy)^6\cdot x$ can be formed by choosing $6$ $xy$'s, $1$ $x$, and $1$ $3$, which can be done in $\binom{8}{6}\binom{3}{2}\binom{1}{1} = 56$ ways. $(xy)^5\cdot x^2\cdot y$ can be formed by choosing $5$ $xy$'s, $2$ $x$'s, and $1$ $3y$, which can be done in $\binom{8}{5}\binom{3}{2}\binom{1}{1} = 168$ ways. Thus the final coefficient is $3(56+168) = 672$ ways.

I completely understand their approach, but fail to understand why mine doesn't work. Don't we just calculate the number of ways to get $x^7$, and $y^6$, then multiply them?
Interestingly enough, I noticed that when you calculate the coefficient of $x^1$ (which is $x^{8-7}$) and $y^2$ (which is $y^{8-6}$), you get $3\cdot \binom{8}{1} \cdot \binom{8}{2} = 672$, which is the answer. I'm $99\%$ sure this isn't a coincidence, but why does this method work and not the other?
I know that I did not get any calculations wrong, because I double-checked everything with WolframAlpha; the error must be in my process.
Thanks in advance!
(Question from PuMaC 2017 Algebra B)
Answer: The factor for $x^7$ is $24$ and for $y^6$ is $28$, which their multiplication is the correct answer. Probably you're just applying the binomial theorem incorrectly.
As an add-on, your method is completely correct.
Comment: I instead got the $x$ term, not $x^7$
Comment: No, I used WolframAlpha just to check and they are correct
Comment: Well, I used Wolfram too, the x one is in fact incorrect, check again for (x+3)^8
Comment: I just noticed *facepalm*
Comment: to answer more precisely your aproach for the answer is 100% valid


How to connect Airflow to AWS S3 without secret keys?

Question: I want to connect Airflow to S3 and be able to take data from a bucket. However the only tutorial on how to do it uses AWS Secret Access Key. I don't want to use them. So I would like to ask If Airflow supports any other way to connect to S3 ?
Comment: The short answer: No.
Answer: If you run Airflow in EC2 machine, or in ECS containers, you can benefit from EC2 instance profile. In this way, you don't need to setup any credentials, and any boto3 request will use the authentication permissions provided by the Instance Profile. From Airflow side you can just use the default aws_default and only set up the region in the extra section.


Comparing date ranges on one graph

Question: I am having trouble figuring out how to display multiple date ranges, stacked on the same graph.

I'm using c3js which has proven great so far, and relatively easy to get it to do what I want.
The screenshot shows the 3 previous weeks, and the date ranges are displayed on the tooltip, with their totals.  However c3js by default will only display one x axis tick, and so overwrites each until the last remains - in this case 17 Oct to 23 Oct.
I'm not too keen on the idea of displaying several dates on the x axis anyway (there's no limit on the number of ranges that can be stacked), so I'm thinking about removing the x axis ticks all together and hacking the tooltip to display the actual dates you are hovering over instead of the range.  However, this has drawbacks as the system has the ability to take screenshots of the graph for sharing, and the tooltip isn't visible on the screenshots.
Is there a standard display for this kind of graph?  Or any ideas?  If it was my choice I would keep it linear - as time usually is - but I've been asked specifically to stack it.
Comment: Are you asking about how to get c3js to do this, or about how such data are best displayed generally?
Answer: I'm not familiar with c3js, but it seems a starting point is to add another variable to your data which is just the day of the week, and then use that as the X variable.
For the general question, I think your approach is fine and is common. An alternative view that can show different kinds of trends for such data is a cycle plot. See Introduction to Cycle Plots by Naomi Robbins.
Comment: Wow, thankyou!  I hadn't considered something like this.  Users can currently pick any date ranges, which could include 30 days, 4 days, 10 days etc. This has made me realise I should probably restrict it to logical patterns.

Comparing date ranges on one graph


Help with Btree homework

Question: I need to do a preorder traversal of a Btree, and among other things, print the following information for each page (which is the same thing as a node):

The B-Tree page number
The value of each B-Tree page pointer (e.g., address, byte offset, RRN).

My questions are:
1. How do you figure out the byte offset? What is it offset from?
2. Isn't the RRN the same as the page number?
Note: A Btree is NOT A BINARY TREE.  Btrees can have multiple keys in each node, and a node with n keys has n+1 child pointers.
Answer: The byte offset is probably the offset of the record from the beginning of the page.
I think the RRN is the relative record number. So if a record is the 5th record in the page, its RRN would be 5.
You need to know the page layout to know how to interpret the information in a page/node. Many solutions are possible.
What code do you have to write, and what code is given to you? I need to know more about exactly what the assignment is asking you to do before I can be of any more help.
Comment: I am not 100% sure what an RRN is, but I think it means "relative record number". If this is the case, then if every page have 5 keys, then an RRN will always be an integer from 1 to 5. Another way to explain it is, it uniquely identifies and record on a page. It wouldn't be a bad idea to double check this with someone.

If your given the code for the Btree's data structures, then those files should explain how to extract the information from a node.
Comment: The code for B-tree creation is given, sort of.  I need to make a program that traverses the B-tree, and outputs information about it.  The program can use the other files that were used in constructing the B-tree.  The main point of the program is so that the user can know the characteristics of the Btree.
Comment: If RRN refers to the keys in the page and not the page itself, then if every page has 5 keys, are the RRN numbers going to always be from 1 to 5, or does every single key in the tree have a unique RRN?

Help with Btree homework


CKEditor returns a value as array

Question: In my Form I have this entry that presents a CKEditor instance
$form['notes'] = array(
    '#input_format' => '1',
    '#rows' => '5',
    '#type' => 'text_format',
    '#base_type' => 'textarea',
    '#title' => t('Reminder Notes'),
  );

In the "values" array within my submit method I have this:
$form_state->getValue('notes'),

But when it saves the data it comes in with only the word "Array"
The install file schema is like this:
  'notes' => [
    'type' => 'text',
    'size' => 'medium',
    'not null' => FALSE,
    'description' => 'Notes to aid in managing the reminder',
  ],

How can I get this to save the data correctly rather than just giving the word "Array"?
Comment: The array will have `value` and `format` keys, you need to save both values to render the result later on
Comment: What is the implementation of the submit function?
Comment: You need to define '#default_value' or '#value' in the array.
Answer: You have to define #default_value or #value as below
$form['notes'] = [
  '#type' => 'text_format',
  '#title' => $this->t('Reminder Notes'),
  '#default_value' => '',
  '#format' => 'filtered_html',
  '#rows' => 5,
];

Then in your install file schema
function mymodule_schema() {
  $schema = array();

  // Define your custom table.
  $schema['mytable'] = array(
    'description' => 'Table for storing reminders',
    'fields' => array(
      'id' => array(
        'type' => 'serial',
        'unsigned' => TRUE,
        'not null' => TRUE,
        'description' => 'Reminder notes',
      ),
      'notes' => array(
        'type' => 'text',
        'not null' => FALSE,
        'size' => 'big',
        'description' => 'notes to aid in managing the reminder',
      ),
      // Add other fields as needed.
    ),
    'primary key' => array('id'),
  );

  return $schema;
}

CKEditor returns a value as array


Multiple recipient's Mail with attachment

Question: In sql server 2012 I have to send mail to multiple recipients with an attachment. The recipient has unique Id_no. and attachment file is saved according to that id_no. Hence matching to that id_no. the attachment has to be sent.
Can anyone help me with query?
Answer: An email has recipients and attachments that are not related
All recipients get all attachments
If you need to match one attachment to one recipient, send one email per recipient
Answer: In the case you know the id of the recipients and the information about the Id and the attachment path is stored in a table you can compose an attachment path based on that and add it to the email like:
DECLARE @AttchPath NVARCHAR(MAX)

SELECT @AttachPath = COALESCE(@AttachPath + ';','') +  <column_path>
FROM <attachments_table>
WHERE Id_no IN (SELECT Id_no FROM <recipients_table>)

Use @AttachPath to send the email.
Hope it helps!
Comment: Thanks!! Yes there is a table which has multiple recipient's email_id and recipient_id respectively.
Comment: You`re welcome, don't forget to mark it as an answer if it helped you!


How to keep a specific module on -dev version (with patches) on a Drupal8 site managed by composer?

Question: I'm trying to manage a Drupal 8 site entirely with composer. However, I need a -dev version of one module (search_api_solr) with a patch. How do I tell composer to just leave this module on the dev version for the time being and not touch it during update ?
Relevant part of my composer.json looks like this
"require": {
        "composer/installers": "^1.0.21",
        "drupal-composer/drupal-scaffold": "dev-master",
        "drupal/admin_toolbar": "^1.18",
        "drupal/bootstrap": "^3.2",
        "drupal/coffee": "^1.0@beta",
        "drupal/config_tools": "^1.0@alpha",
        "drupal/console": "~1.0",
        "drupal/core": "~8.2",
        "drupal/delete_all": "^1.0@alpha",
        "drupal/devel": "^1.0@RC",
        "drupal/facets": "^1.0@alpha",
        "drupal/imce": "^1.5",
        "drupal/masquerade": "^2.0@beta",
        "drupal/migrate_manifest": "^1.4",
        "drupal/migrate_plus": "^3.0@beta",
        "drupal/migrate_tools": "^3.0@beta",
        "drupal/migrate_upgrade": "^2.0",
        "drupal/pathauto": "^1.0@RC",
        "drupal/search_api": "1.0@beta",
        "drupal/search_api_sorts": "^1.0@alpha",
        "wikimedia/composer-merge-plugin": "~1.3",
        "drupal/search_api_solr": "8.x-1.x-dev"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,

I've specified the search_api_solr version according to this advice. However, when I run composer update I get:
Your requirements could not be resolved to an installable set of packages.
The requested package drupal/search_api_solr 8.x-1.x-dev exists as drupal/search_api_solr[...] but these are rejected by your constraint.
EDIT:
Meanwhile I have worked around this by removing the drupal/search_api_solr requirement completely from the composer.json, explicitly listing solarium/solarium instead and installing the patched version of search_api_solr-8.x-1.x-dev manually into the modules/unstable directory. Now composer update works as it shold without overwriting the patch.
Comment: I think the module version should be specified as `1.x-dev`, not `8.x-1.x-dev`. Are you using the updated composer endpoint for Drupal packages?
Comment: Also, you can specify a commit hash to 'lock' a dev version to specific commit instead of rolling dev updates.
Comment: As far as patches go you can use composer-patches add on to do that. Check out the Drupal Composer project or Acquia BLT to see how it is done.
Answer: Firstly, the 8.x- is not part of the composer version; the correct version is 1.x-dev or dev-1.x.
However, composer require drupal/search_api_solr:1.x-dev always updates to the latest revision on the dev branch. If you want to stay on a specific commit and prevent updates entirely, use something like composer require drupal/search_api_solr:1.x-dev#46e16862.
Comment: @christoph.burschka I want to download an installation profile from github, but when I do `composer require` it gives me error regarding a module that **it does not match the constraint**. In composer.json of that install profile, the module is in require section like this `"drupal/block_access": "dev-8.x-1.x#1cd59ba6856c897afae2551feaa8e6b7f312489c"`  This is the repo https://github.com/UH-StudentServices/student_guide


Dual graph of a tree

Question: It is stated here that:

For any connected embedded planar graph G define the dual graph G* by
  drawing a vertex in the middle of each face of G, and connecting the
  vertices from two adjacent faces by a curve e* through their shared
  edge e. Note that G**=G.

Given any planar drawing of a tree $G$ is its dual graph simply $K_1$? But then how does one justify $G^{**}=G$?
Comment: Maybe it only works, when $G$ has more than one face?
Answer: The dual graph need not be simple. Here's a badly drawn example of a black tree with $6$ vertices (and therefore $5$ edges and $1$ face) and its red dual: the dual has $1$ vertex, $5$ edges, all of which are loops, and $6$ faces.
Comment: I'm confused. What is the correct definition of a dual graph? How are its edges defined?
Comment: @Shahab: The definition that you gave is basically right; it just fails to take into account that an edge of $G$ may border only one face, in which case it gives rise to a loop in $G^*$ at the vertex corresponding to that face.

Dual graph of a tree


Angular 5 ng-model template parsing issue

Question: I am using angular 5 and trying to use [(ngModel)]="user.fname" like that. I have also imported FormsModule in app.module.ts. But still I am having a template parsing issue.
Answer: add formdodule to the app.module and imports.
import { FormsModule } from '@angular/forms';
imports: [  FormsModule]


"TypeError: string indices must be integers, not str" when trying to get item from python dictionary

Question: I have such a code in order to fetch some data from a url:
import urllib2
response = urllib2.urlopen('https://api.genderize.io/?name=joseph')
html = response.read()

>>> print html
{"name":"joseph","gender":"male","probability":"0.99","count":923}

Everything up to here is fine. But when I want to fetch "gender" data from this dictionary;
>>> print html["gender"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: string indices must be integers, not str

I get the above error.
How can I fix the problem?
Comment: `html` is a string, try `json.loads(html)['gender']`.
Comment: Done, I tried to be more detailed. Let me know if you still have questions! Good luck.
Comment: Thank you Maroun. Please publish your comment as an answer.
Answer: That's because html is of type string. To validate that, try:
>>> type(html)
<type 'str'>

Hence the error message saying that string indices must be integers, not str.
You should convert it to JSON using json module:
json.loads(html)['gender']


PowerShell: Copying Items from Network Share to Remote Server while using Credentials

Question: I have files stored on a network share \\domain.company.com\Server01\Folder1\Path that I am looking to perform a recursive copy to a remote server \\RemoteServer1\D$\Temp\Folder1.
Initially, I looked at using Copy-Item -Path "\\domain.company.com\Server01\Folder1\Path\*" -Destination "\\RemoteServer1\D$\Temp\Folder1" -Recurse -Force. However, since the script is running from a machine that doesn't have administrative access to the RemoteServer, I need to pass in Credentials for the Copy-Item.
As we know, this isn't possible due to the FileSystem which leads me to using New-PSDrive.
So here I am writing the following code block:
New-PSDrive -Name Source -PSProvider FileSystem "\\domain.company.com\Server01\Folder1\Path" -Credential $Creds
New-PSDrive -Name Target -PSProvider FileSystem "\\RemoteServer1\D$\Temp\Folder1" -Credential $Creds
Copy-Item -Path Source:\* -Destination Target: -Recurse -Force
Remove-PSDrive Source
Remove-PSDrive Target

Now my next problem arrises in that New-PSDrive errors when I pass in my credentials, informing me that "the network path was not found" even though I know fully that the passed in credentials have full rights to this network share and remote server.
My question here is what is the best way of copying items, folders, and items within those folders to a remote server while on my machine and when using passed in credentials for a different account.
Comment: Please provide the exact error you receive (with sensitive details changed/removed as needed). Does the `New-PSDrive` command fail for both `Source` and `Target`?
Comment: Try mapping to the share, not a subfolder?  Then if that works, use Get-ChildItem -Path "$psDriveName:" and see what it says, then "$psDriveName\subfolder".
Answer: In short, I discovered I was using the New-PSDrive command incorrectly in which it doesn't "map" the name, but rather creates a reference. You still need to specify the full path in the Copy-Item. See below for working solution:
New-PSDrive -Name Source -PSProvider FileSystem -Root "\\domain.company.com\Server01\Folder1\Path" | Out-Null
New-PSDrive -Name Target -PSProvider FileSystem -Root "\\RemoteServer1\D$\Temp\Folder1" -Credential $Creds | Out-Null
Copy-Item -Path "\\domain.company.com\Server01\Folder1\Path\*" -Destination "\\RemoteServer1\D$\Temp\Folder1" -Recurse -Force
Remove-PSDrive Source
Remove-PSDrive Target

Comment: Not so.  You should be able to access Source: and Target: drive just as you were doing it in your original post.  `Copy-Item -Path Source:\* -Destination Target:\ -Recurse -Force` should work
Comment: Are you adding the PSDrives from the same script/session as the one you are trying to access them from later?
Comment: Your original post said your error was with the mapping of the drive, not accessing the drive via Copy-Item.  As TheMadTechnician said, exact errors help.


Show that $\text{deg}(M_n(\mathbb{K})) = n$, where $\mathbb{K}$ is a field.

Question: Definition: Let $A$ be a ring and $Z=Z(A)$ its center. We say that $t \in A$ is algebraic over $Z$ if there exist $z_0,z_1, \ldots , z_n \in Z$ such that
$$z_0+z_1t+ \cdots + z_n t^n = 0 \quad \text{and} \quad z_n \neq 0.$$
Moreover, in this case we say that $t$ is algebraic of degree at most $n$. The degree of algebraicity is the smallest $n$ that satisfies this property; it will be denoted by $\text{deg}(t)$.
Show that $\text{deg}(M_n(\mathbb{K})) = n$, where $\mathbb{K}$ is a field.
Comments: I'm trying to use the Cayley-Hamilton's Theorem.
Comment: It seems your intro is missing a definition such as $\deg(A)=\sup\{\,\deg(t):t\in A\,\}$
Comment: @HagenvonEitzen No, scalar matrices are central.
Comment: You say you're trying to use the Cayley--Hamilton theorem. Where are you stuck after that?
Comment: As I guarantee that the characteristic polynomial coefficients are in $Z$?
Comment: What is $Z(M_n(\mathbb K))$ exactly?
Comment: Hint: Diagonal matrices are central
Comment: @DavidHill Oops, that' what I meant to write. I guess I got confuse cause I wanted to add a statement that we need only one direction here ...
Comment: Managed to solve the problem with the Newton's formula, then the coefficients saw linear combination of traces.
Comment: @QuangHoang $Z(G)$ is the center of a group $G$.
Comment: @Croos, I honestly have no idea how you got Newton's formula mixed into this!
Answer: A very simple computation shows that the center of $A=M_n(K)$ is the subring of scalar matrices, that is, those matrices which are diagonal and constant along the diagonal. It follows then that $Z=Z(A)$ is isomorphic to $K$ as a ring. Let $\phi:K\to Z$ be the isomorphism which maps each element of $K$ to the corresponding scalar matrix.
Let now $X$ be an element of $A$. As you observed, the Cayley—Hamilton theorem gives us a polynomial  $f$ with coefficients in $K$ such that $f(X)=0$ and the degree of $f$ is exactly $n$. From $f$ we can construct a new polynomial $\tilde f$, which is the result of applying $\phi$ to the coefficients of $f$. You can show very easily that $\tilde f(X)$ is zero.
From this, it follows that the degree of $A$ over $Z$ is at most $n$. let us show that it is equal to $n$. To do this, it is enough to show that there is a matrix $X\in A$ such that no polynomial of degree strictly smaller that $n$ vanishes on $X$. You should check that the permutation matrix $$\begin{pmatrix}0&1&0&0&0\\0&0&1&0&0\\0&0&0&1&0\\0&0&0&0&1\\1&0&0&0&0\end{pmatrix}$$ has degree $5$ in $M_5(K)$ and then generalize this to arbitrary $n$.


How to find the most commonly repeated set of double's in a text file? -Java

Question: I have a text file with several hundred doubles separated by line breaks:
Ex:
-0.020000000000010232
0.09500000000001307
-0.05500000000000682
0.1599999999999966
0.07000000000000739
-0.0799999999999983
-0.07000000000000739
0.060000000000002274
0.04000000000000625
0.04999999999999716
-0.10000000000000853 ...

For further context, I'm trying a super simple version of the application of machine learning to stocks, where derivative series of differences in price are dumped in a text file. The program, whilst constantly updating the "top three" patterns, will attempt to reference a pattern whenever it sees similar derivative values to predict future changes. Less relevantly, several other heuristics considering both volatility and volumes are also included in the refinement of these patterns.
The first part of this is finding/guessing basic patterns from the text file. I have two questions here:

How do I round off these values to the fourth decimal point?
(0.0001)
How do I find the three most commonly repeated sets of rounded
doubles and write them to a separate text file?
Comment: 1. Do you mean repeated most often, longest repeating group, or most frequent?
Comment: Repeated most often, longest repeating group.
Comment: I meant repeated most often (e.g. repeated in 5 places) OR longest repeating group (the longest sequence of repeats was 5)
Comment: @PeterLawrey I think OP means repeated most often
Comment: @DebosmitRay I think the OP wants a frequency count, only because that is usually what people ask for.
Comment: @PeterLawrey Yes yes. I had not noticed the point about 'frequency of a set'. So, my answer just created a frequency table of each double value. If OP does want something along the lines of a pattern freq, can you think of something better than an o(n^3) (might be wrong with the order) ?
Comment: @DebosmitRay you can do O(n) in each case, a full sort would be O(m log m) but if you only need top 3 you can simplify this.
Comment: @PeterLawrey Ah! Yes. I see what you are referring to. Thanks much!
Answer: For the first part, you can take a look at the DecimalFormat class. It should allow you to format numbers accordingly.
Secondly, you could use a Map to store the (rounded) number and its occurrence in the file.
Once finished, you would go through the map and see which 3 numbers have the highest occurrence (value for a given key).
Comment: How would I implement the Map such that it would find repeated sets of occurrences rather than repeated occurrences of a single double? Example: if 0.345 is repeated 8 times and 0.233 is repeated 7 times, they are not guaranteed to be adjacent to each other on the list.
Comment: @user3532216 so you want a frequency count, see my answer.
Answer:
For the fourth decimal point part.

You can use Decimal Format API.
DecimalFormatt df = new DecimalFormat( "#.0000" );
df.format(num); // 4 places of decimals

2a. Find the three most commonly repeated sets of rounded doubles.
Use a map data structure to map rounded decimal values to their frequencies. A HashMapAPI would be great.
2b. Parse HashMap to get the most frequent decimal values and write them to a file using PrintWriter Reference
Let me know if something is not clear and I will try to help you out
Comment: How does this retrieve frequencies of sets of double numbers? It appears that HashMap would order them and find their independent frequencies. Example: (0.566 appeared 9 times) vs (0.342, 0.222, 0.455, 0.333 appeared 5 times)
Comment: That part wasn't very clear from your clear. You can do some sort of an o(n^2) algorithm to see if doubles at index `i`, `i+1`,.. appear somewhere else in the string. You would use a Hashmap to store this data. Run-time wise, I can't think of anything better than o(n^2) right now.
Answer: You can do a frequency count in O(n) like this.
Map<Double, Long> freqMap = DoubleStream.of(doubles).boxed()
                    .collect(Collectors.groupingBy(d -> round4(d), Collectors.counting());
// you can sort by the count getting just the top 3.

While DecimalFormat will do the job, it is really slow.  You can just use some Maths.
private static final double WHOLE_NUMBER = 1L << 53;
/**
 * Performs a round which is accurate to within 1 ulp. i.e. for values very close to 0.5 it
 * might be rounded up or down. This is a pragmatic choice for performance reasons as it is
 * assumed you are not working on the edge of the precision of double.
 *
 * @param d value to round
 * @return rounded value
 */
public static double round4(double d) {
    final double factor = 1e4;
    return d > WHOLE_NUMBER / factor || d < -WHOLE_NUMBER / factor ? d :
            (long) (d < 0 ? d * factor - 0.5 : d * factor + 0.5) / factor;
}

For example,
double[] doubles = { 0.123400000000000, 0.123399999999997};
for (double d : doubles) {
    System.out.println(round4(d));
}

prints
0.1234
0.1234

As an exercise I though I would test this brute force for every possible values between 0.0001 and 9.99995 using multiple threads.
double[] ds = new double[17];
ds[0] = 1e-4;
for (int i = 1; i < ds.length; i++)
    ds[i] = 2 * ds[i - 1];

DoubleStream.of(ds).parallel()
        .forEach(x -> {
            int counter = 0;
            for (double d = x; d <= 2 * x && d < 9.99995; d += Math.ulp(d)) {
                if (Double.toString(Maths.round4(d)).length() > 6)
                    fail("d: " + d);
                if ((counter & 127) == 0 && counter % 10_000_000 == 0)
                    System.out.println(d);
                counter++;
            }
        });

Comment: Depending what OP is trying to do with the rounded number, this solution may not be correct.  It works for things like "comparing 2 numbers, based on their value rounded to 4 decimal point", but not for "writing number in another text file, rounded to 4 decimal point"
Comment: @AdrianShum can you give an example of what you mean?
Comment: What I mean is, in order to output a "rounded-to-4-decimal-place" number to a text file, he still need to perform formatting (by using `DecimalFormat` or by other way), else he will still see '0.123400000000000' or even things like '0.123399999999997' if he write the returned double value directly to file.
Comment: @AdrianShum Instead of saying, like X but not actually X. Can you give a specific example where if he rounds the result, he will see a problem?
Comment: @AdrianShum I have added your examples to my answer, happy to add more, because I am not clear on what you mean.
Comment: Thanks, the examples helped me understand the scope of the frequency count you were referring to earlier.


React Project - Request to ExpressJS Server returns 400 - Bad Request

Question: Parameters are set and once button is pressed, a request [Node request package] or fetch request is made to be received by expressjs request. While link address is created, response returned is 400 Bad Request.
I've tried both a fetch request and a node package 'request' and
Server JS
    const express = require ('express');
    const path = require('path') //core node module
    const app = express();
    const cors = require('cors');
    const router = express.Router();

    // app.use(cors())
    const publicdirpath = path.join(__dirname, '../public')
    console.log(path.join(__dirname, '../public'))

    app.use(cors());
    app.use(express.static(publicdirpath))

    app.post('/testcall', (req, res) => {
        if(!req.query.startdate && !req.body.enddate &&
    !req.body.projectnumber){
        return res.status(400).send({
            success: 'false',
            message: 'dates or project or both required'
        });
     }

     //call stored procedures
   });

    const port = process.env.PORT || 3000;
    app.listen(port, () => {
    console.log(`Server running on port ${port}`);
   });

JS Class that calls express JS

    handleDropdownClick = (event, selection) =>{
        const { name, innerText, value } = event.target;
        console.log( event.target  + " : " + innerText + " : " +
    this.props.formData);
        const request = require('request');
        switch(selection){
            case 1:
                //call api or stored procedure
                if(this.validation()){
                    //call api
                 request.get({url:"http://localhost:3000/testcall", qs:this.state.formData} , function (err, res, body) {
                        if(err){
                            console.error('error with request: error: ' + err + '. res: ' + res +  ' + body: ' + body);
                        }
                        console.log("Get response: " + res.statusCode + ". Body: " + body);

                    })

                    //Using Fetch
                    const jsonData = JSON.stringify(this.state.formData);
                    fetch('/testcall', {
                        method: "POST",
                        headers: {
                            'Accept': 'application/json',
                            'Content-Type': 'application/json',
                            'credentials': 'include'
                        },
                        body: jsonData
                    })
                    .then( response => {
                        return response.json();
                    })
                    .then( response => {
                        console.log(response);
                    })
                    .catch(function (e) {
                        console.log("fail: " + e);
                    })
                }
                break;
    //more code

Package.json
    {
      "name": "app-name",
      "version": "0.1.0",
      "private": true,
      "main": "index.js",
      "dependencies": {
      "body-parser": "^1.19.0",
      "cors": "^2.8.5",
      "express": "^4.17.1",
      "nodemon": "^1.19.1",
      "react": "^16.8.6",
      "react-dom": "^16.8.6",
      "react-scripts": "3.0.1",
      "request": "^2.88.0",
      "save": "^2.4.0",
      "semantic-ui-react": "^0.87.2",
      "table": "^5.4.1",
      "tedious": "^6.2.0",
      "webpack": "^4.29.6"
     },
    "scripts": {
      "start": "react-scripts start",
      "build": "react-scripts build",
      "test": "react-scripts test",
      "eject": "react-scripts eject",
      "dev": "run-p server start"
    },
    "eslintConfig": {
      "extends": "react-app"
    },`enter code here
    "browserslist": {
      "production": [
        ">0.2%",
        "not dead",
        "not op_mini all"
      ],
      "development": [
        "last 1 chrome version",
        "last 1 firefox version",
        "last 1 safari version"
      ]
    },
    "proxy":"http://localhost:3000"
  }

Index.js
    import React from 'react';
    import ReactDOM from 'react-dom';
    import './index.css';
    import App from './App';
    import * as serviceWorker from './serviceWorker';

    ReactDOM.render(<App />, document.getElementById('root'));

    serviceWorker.unregister();

File Path
public
src
 -api
 -components
 -storedprocedures
 -tabs
 -app.js
 -config.js
 -index.js
 -server.js
package-lock.json
package.json

Expected results would be for the expressjs to accept the call and complete logic.
Actual results are:
POST http://localhost:3000/calculatecommission 400 (Bad Request)
- for the npm package request
request.js:149 GET http://localhost:3000/calculatecommission?projectnumber=&startdate=2019-06-02&enddate=2019-06-28 400 (Bad Request)
- for fetch request
Answer: Got the same result with axios unfortunately. I'm wondering if the issue is that the express is server file is set up wrong. Maybe it's not getting the request at all.
Answer: You can use axios or use proxy at front-end
Answer: I would recommend using Axios as well, it's a whole lot easier to use then fetch. You can find out more here
https://alligator.io/react/axios-react/
Answer:
request.js:149 GET http://localhost:3000/calculatecommission?projectnumber=&startdate=2019-06-02&enddate=2019-06-28 400 (Bad Request) - for fetch request

Any time you get an error like this, don't forget, you are literally allowed to run this command yourself in the command-line interface.  That will give you the exact error that the install script is actually running into, i.e.:

GET https://www.google.com/

In my case, the error was: HTTPS not supported without installing LWP::Protocol::https.
So:

I resolved all of the issues from the GET command.

Since the network it was trying to access was private, I used sudo npm install --s.


Frequentist vs Bayesian Linear Gaussian Models

Question: Consider the following linear Gaussian system:

where $p(x)$ is our prior. The Bayesian inference problem can be expressed in closed-form as1:

Where can I find an equivalent "frequentist" estimation procedure? Can they also be expressed in closed form?

1 "Machine Learning - A Probabilistic Perspective", 2012, pp. 119. Kevin Murphy.
Answer: It is unclear what you mean by frequentist estimation, but I'll take a stab.
This problem is regression with a known covariance matrix where
$$
Y-b \sim N(Ax,\Sigma_y).
$$
The only unknown here is $x$, so the maximum likelihood estimator is
$$
\hat{x}_{MLE} = \text{argmax}_{x} p(y|x) = [A^T\Sigma_y^{-1}A]^{-1}A^T \Sigma_y^{-1}(y-b)
$$
which is the posterior mode in the limit as $\Sigma_x^{-1}\to 0$, i.e. the improper uniform distribution.
The sampling distribution for $\hat{x}_{MLE}$ is normal with mean
$$
[A^T\Sigma_y^{-1}A]^{-1}A^T \Sigma_y^{-1}A x
$$
and covariance matrix
$$
A[A^T\Sigma_y^{-1}A]^{-1}[A^T\Sigma_y^{-1}A]^{-1}A^T.
$$
The sampling distribution can be used to construct confidence, i.e. frequentist, intervals.
Comment: Thanks @jaradniemi. This is great; right what I was looking for. Any particular papers or textbooks you recommend that cover sampling distributions of these types of systems? (e.g. I am interested on similar derivations for a mixture of Beta-Binomial variables).


Macro, Capturar MOUSE e TECLADO

Question: Preciso fazer um macro, capturando mouse e teclado.
Como devo proceder em JAVA ?
preciso de alguma API específica ?
Comment: Precisa mesmo ser em Java? Java é uma linguagem de alto nível, e seu propósito está longe de tratar comunicação com hardware, com certeza precisará de alguma biblioteca específico para o SO em questão. Se você estiver usando Windows e quiser **controlar** mouse e teclado pode usar [**AutoIt**](https://www.autoitscript.com/site/autoit/), entretanto acredito que não conseguirá **capturar**.
Comment: O que queres dizer com macro? queres saber todos os movimentos do rato durante um período de tempo?
Answer: Se entendi bem, a ideia de fazer uma Macro seria para automatizar uma tarefa, como se fosse um usuário utilizando o computador.
A API do Java disponibiliza a classe Robot para isso. Ela contém métodos que disparam eventos simples como mouseMove, mousePress e keyPress.
Para automação de tarefas simples ela deve funcionar bem. Porém, a documentação alerta que em alguns sistemas podem ser necessárias permissões especiais para funcionar.
Não conheço muito bem essa API, então não posso adiantar quais são as limitações e problemas que podem ocorrer.
Se precisar de algo mais elaborado pode tentar uma API de terceiros, como a jNativeHook, que fornece uma API Java fazendo interface com métodos nativos da API do Sistema Operacional. Na verdade, a ideia é a mesma da classe Robot, mas eles prometem mais funcionalidades.
Comment: Muito obrigado.
Answer: Só para completar a resposta do @utluiz
Alguns exemplos práticos do uso dessas funções:
Rato descontrolado anda aleatoriamente pelo ecra:
public static  void movemouse(){
        Robot robot = null;
        try {
            robot = new Robot();
        } catch (AWTException ex) {
            Logger.getLogger(NewJFrame.class.getName()).log(Level.SEVERE, null, ex);
        }
        Random random = new Random();
        while (stop<=30) {
            robot.mouseMove(random.nextInt(500), random.nextInt(500));
            stop++;
            try {
                Thread.sleep(2000);
            } catch (InterruptedException ex) {
                Logger.getLogger(NewJFrame.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
    }

Programa que desliga o computador(possivelmente deve ter de alterar os valores devido a resolução)
public static  void desligaPc(){
    Robot robot = null;
    try {
        robot = new Robot();
    } catch (AWTException ex) {
        Logger.getLogger(NewJFrame.class.getName()).log(Level.SEVERE, null, ex);
    }

        robot.mouseMove(0, 800);
        robot.mousePress(InputEvent.BUTTON1_MASK);
        robot.mouseMove(300, 700);
        robot.mousePress(InputEvent.BUTTON1_MASK);
        robot.mouseRelease(InputEvent.BUTTON1_MASK);
    }
}

Comment: Obrigado pelos exemplos!

Macro, Capturar MOUSE e TECLADO


Correcting for multiple 2-way anova testing

Question: I want to study the effect of two factors on a paramater that I measure in several regions. I'm running a  2-way ANOVA test for each region (about a dozen) and I am wondering how I can deal to avoid the increase of the type I error. I'm thinking about performing an FDR correction on the p-values that I get with the ANOVA tests. I get 3 p-values for each test (one for each factor and the third for interaction between factors), so approximately 36 p-values altogether. I wonder how to appropriately correct for multiple tests and particularly whether I should correct the p-values all together or separately for each factor and for the interaction.
Thank you !
Comment: Can you please explain what you mean by "regions"?  What, exactly, stops you from being able to ignore the regions and do one AnOVa?
Answer: I don't know what exactly are the regions you are mentioning in your question but if your data are spatially correlated you should think of taking into account this correlation, by calculating a nonparametric statistical test.
In short, the algorithm could be described by the following steps:

For every region, conduct the 2way ANOVA and compare your signals by means of an F-value.
Select all regions whose F-value is larger than some threshold. (Usually this threshold corresponds to a p<.05. By changing it you affect the sensitivity/specificity of the test).
Cluster the selected samples in connected sets on the basis of spatial adjacency.
Calculate cluster-level statistics by taking the sum of the F-values within a cluster.

To construct your null distribution:

Create a random partition of your data by permuting randomly the labels of the factors (that depends strongly on the design of your study ie., if you have within or between subjects factors or both).
Calculate the test-statistic for this random partition by repeating steps 1-4.
Repeat steps  5-6 for a fair amount of times (usually 10K) to construct a null distribution.
Your p-value is the proportion of random partitions that resulted in a larger test statistic than the observed one.

You can check out these papers to get a detailed description of the implementation:
Holmes AP, Blair RC, Watson JDG, Ford I. Nonparametric analysis of statistic
images from functional mapping experiments. J Cerebr Blood Flow Metabol
1996;16:7–22.
Bullmore E, Suckling J, Overmeyer S, Rabe-Hesketh S, Taylor E, Brammer M.
Global, voxel, and cluster tests, by theory and permutation, for a difference
between two groups of structural MR images of the brain. IEEE Trans Med
Imag 1999;18:32–42.
Maris E, Oostenveld R, Nonparametric statistical testing of EEG- and MEG-data,  JNeuroscMethods, 2007; 164:177–190.
I can also point you out to a matlab-based free toolbox (I'm sure there are more but this is the one I know of) with bult-in functions to implement these kinds of tests: http://www.fieldtriptoolbox.org/.
Comment: The permutation-based approach is definitely the way to go. In case a more up-to-date reference was of interest, check out https://www.sciencedirect.com/science/article/pii/S0165027014002878


Vacuuming Postgresql has used up disk space

Question: I have just run Vacuum on a Postgres table to try to recover disk space, however the result is that all the disk space has been consumed.  Does Vacuum crete log files or transaction logs that can be deleted?
Answer: I'm assuming you performed a VACUUM FULL as the standard VACUUM just makes space in the data file that is associated with deleted records free so that Postgres can use that space for new records.  It doesn't release the space to the operating system.
VACUUM FULL does release space, but it does this by copying all the information that it wants to keep from the data file into a new data file, and then when complete it deletes the old data file.   As such VACUUM FULL requires extra space while it is running.
http://www.postgresql.org/docs/current/static/sql-vacuum.html
Comment: yes, VACUUM FULL.  Once it has finished should it free disk space or are there logs to manually delete?  It is actually a small database, however the disk is full.


Glue records for sub domains?

Question: Let's say I own example.com
I created an A record called dns.example.com then point it at a nameserver.
On the nameserver I gave it the name ns1.dns.example.com and also dns.example.com
Now I'm confused. Do I create glue records of ns1.dns.example.com or dns.example.com? Do I also still need the A records for my dns.example.com?
Answer: You need glue records if the NS records for a domain point to hostnames within that domain. For example, this would not need glue-records:
testing.example.com:
NS = ns1.infra.example.com.
NS = ns2.infra.example.com.

Where, this one would need glue-records:
testing.example.com:
NS = ns1.testing.example.com.
NS = ns2.testing.example.com.

In that case, the glue records go into the example.com domain.
example.com:
[...]
testing.example.com.     IN NS ns1.testing.example.com.
testing.example.com.     IN NS ns2.testing.example.com.
ns1.testing.example.com. IN  A 172.16.202.152
ns2.testing.example.com. IN  A 22.214.171.124

Comment: I think I understand it now. I'm using gandi for my dns managment. I went into glue record to add the ns1.testing.example.com  and after a few 12 hours it didn't work, I also didn't see it in my zone file. I assume it is because I don't have a NS record. Now I've gone into example.com and added the NS and A record. Also have the glue record. Hopefully I'll see it working in the next few hours!
Comment: the two **A** records shown are missing a **.**-dot after the domain; if its missing it will result in the domain being interpreted as `ns1.testing.example.com.example.com` @lbanz in the gandi glue management, you don't need the trailing dot.
Comment: @pete thank you, it is all working now. I couldn't get it to work at first because I didn't realise gandi is using relative path. So using ns1.testing.example.com. didn't work and I changed it to I use ns1.testing Now all seems to be working.
Comment: @lbanz didn't work even without the trailing dot? i have mine setup with the full ns domain and no issue, although my servers are only one sub deep. either way, glad it works

Glue records for sub domains?


Fitting a probability distribution to zero inflated data in R

Question: I am trying to learn how to fit a probability distribution to a vector of data, using the program R, but there are a lot of potential probability distributions to use!  So my question is, how do I find the best distribution for my data, and how do I prove that I have picked the right distribution?  Can I acquire AIC values for a whole set of different distributions?
The data are observational count data of bees visiting flowers.  Each species has a certain number of visits, hence the differing frequencies.  The goal is to find the best distribution to describe the bee visitation, show that I have selected the right one, and then use that distribution to sample from randomly for a set of simulations.
Here is what the data looks like, it is a vector of count observations.  It is zero inflated, with a long tailed distribution (maybe zero-inflated negative binomial?).
i.vec=c(0,63,1,4,1,44,2,2,1,0,1,0,0,0,0,1,0,0,3,0,0,2,0,0,0,0,0,2,0,0,0,0,
0,0,0,0,0,0,0,0,6,1,11,1,1,0,0,0,2)

And here are some basic parameters that I have calculated.  I am using standard deviation for sigma, and phi is the proportion of zeroes in the data.
m=mean(i.vec)
#[1] 3.040816
sig=sd(i.vec)
#[1] 10.86078
tab<-table(i.vec)
zero.prop<-as.numeric(tab[1])/sum(as.numeric(tab))
#[1] 0.6122449

As you can see, the standard deviation is much greater than the mean, and I have a very high proportion of zeroes.
Comment: It would be helpful to have some context: where do the data come from?  Why do you want to fit a distribution?  To me, it's the 11, 44, and 63, rather than the 0's, that are unusual.
Comment: I will add in some information to my question.
Answer: You can use Vuong test in pscl package to compare non-nested models. Here is an example
> m1 <- zeroinfl(i.vec ~ 1 | 1, dist = "negbin")
> summary(m1)

Call:
zeroinfl(formula = i.vec ~ 1 | 1, dist = "negbin")

Pearson residuals:
    Min      1Q  Median      3Q     Max
-0.3730 -0.3730 -0.3730 -0.2503  7.3544

Count model coefficients (negbin with log link):
            Estimate Std. Error z value Pr(>|z|)
(Intercept)   1.1122     0.3831   2.903  0.00369 **
Log(theta)   -1.9256     0.2839  -6.784 1.17e-11 ***

Zero-inflation model coefficients (binomial with logit link):
            Estimate Std. Error z value Pr(>|z|)
(Intercept)   -9.815     96.462  -0.102    0.919
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Theta = 0.1458
Number of iterations in BFGS optimization: 579
Log-likelihood: -80.51 on 3 Df

> m2 <- zeroinfl(i.vec ~ 1 | 1, dist = "poisson")
> summary(m2)

Call:
zeroinfl(formula = i.vec ~ 1 | 1, dist = "poisson")

Pearson residuals:
    Min      1Q  Median      3Q     Max
-0.7242 -0.7242 -0.7242 -0.4860 14.2795

Count model coefficients (poisson with log link):
            Estimate Std. Error z value Pr(>|z|)
(Intercept)  2.05911    0.08205    25.1   <2e-16 ***

Zero-inflation model coefficients (binomial with logit link):
            Estimate Std. Error z value Pr(>|z|)
(Intercept)   0.4561     0.2933   1.555     0.12
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Number of iterations in BFGS optimization: 11
Log-likelihood: -233.7 on 2 Df

> vuong(m1, m2)
Vuong Non-Nested Hypothesis Test-Statistic: 1.946095
(test-statistic is asymptotically distributed N(0,1) under the
 null that the models are indistinguishible)
in this case:
model1 > model2, with p-value 0.02582165

Vuong test also suggests that the zero-inflated negative binomial provides a better fit to your data compared to the ordinary negative binomial (not shown here, but you can fit both models and compare them).
Answer: I don't think you necessarily need to inflate zeros.
Your data seem quite consistent with a negative binomial:
> library(MASS)
> table(rnegbin(49,mu=3.1,theta=0.075))

 0  1  2  3  5 18 20 21 31 61
36  4  2  1  1  1  1  1  1  1
> table(i.vec)
i.vec
 0  1  2  3  4  6 11 44 63
30  8  5  1  1  1  1  1  1

So lets get some estimates:
> mean(i.vec)
[1] 3.040816
> theta.ml(i.vec,3.041)
[1] 0.145777
attr(,"SE")
[1] 0.04136887

So let's just take a look at mu = 3.041 and theta at say 0.14 (there's lots of uncertainty in theta here):
Here's three random samples from that distribution:
> table(rnegbin(49,mu=3.041,theta=0.14))

 0  1  2  3  4  5  6  7  8 18 49
30  5  1  3  2  2  1  2  1  1  1
> table(rnegbin(49,mu=3.041,theta=0.14))

 0  1  2  3  4  7  9 15 29 31 47 56
33  2  4  1  1  1  1  2  1  1  1  1
> table(rnegbin(49,mu=3.041,theta=0.14))

 0  1  2  3  4  7  9 12 16 48 66
36  4  1  1  1  1  1  1  1  1  1

They look similar enough to your data. Negative binomial seems at least plausible.
You may like to play with the functions in MASS
Comment: Thank you!  It won't let me vote up, but I appreciate your help.
Answer: Im not sure you can do much better than just plugin the empirical measure in that case, without further information on your data (especially since you have very few observations). And in that case the variance of your error should be of the order of the inverse of the number of your observations (via Efron-Stein).
Maybe you could use some convolution based estimator (like the density function in R, but with an integer supported kernel) to smooth things a little. Or see things as a mixture. But theres no reason to do so if you have no idea about where your data comes from.
Comment: Thank you!  It won't let me vote up, but I appreciate your help.


How to abort the ajax request and redirect in the ajax callback of form element

Question: I want to redirect users in the AJAX callback, but nothing I tried works.
How could I achieve that?
Answer: If you've got ctools installed (nearly all Drupal sites do), you can make use of ctools_ajax_command_redirect() in your AJAX callback, e.g.
ctools_include('ajax');
$commands[] = ctools_ajax_command_redirect('path/to/redirect');
return array('#type' => 'ajax', '#commands' => $commands);


How do I set up my first view with Draggable Views module?

Question: I'm learning how to use Draggable Views for Drupal 6 for the first time, following this tutorial: http://thedesignspace.net/MT2archives/000666.html . I'm encountering a problem though: in part 4, once I set up the draggable table ( http://thedesignspace.net/MT2archives/000708.html ) the entire preview disappears, showing no table. Switching the style to another type, such as table, brings everything back. Directly viewing the page from its path makes no difference. Any ideas of what the problem could be?
Answer: Not all of my modules were up to date when I encountered this problem. My mistake. Always update all modules first upon being stumped by incompatibility errors, even if the modules in question do not seem to have anything to do with the error.


HTTP: leading blank line in header - a valid http request header?

Question: Just wondering, is this a valid http request, please make a note of leading blank line(s) "\r\n" before the actually request headers.
\r\n\r\nGET / HTTP/1.0\r\n\r\n

Although it works with all the servers (apache, lighttpd, nginx), RFC doesn't mandate anything about leading blank lines and leave it to the implementation.
Thanks
Answer: So the standard says each new line should end with \r\n and each request should end with \r\n\r\n
What, at the bare minimum, is required for an HTTP request?
But most servers will handle leading newlines of most types
printf '\rGET / HTTP/1.1\r\nHost: www.example.com\r\nConnection: close\r\n\r\n' |
  nc www.example.com 80
printf '\r\nGET / HTTP/1.1\r\nHost: www.example.com\r\nConnection: close\r\n\r\n' |
  nc www.example.com 80
printf '\r\r\nGET / HTTP/1.1\r\nHost: www.example.com\r\nConnection: close\r\n\r\n' |
  nc www.example.com 80
printf '\n\rGET / HTTP/1.1\r\nHost: www.example.com\r\nConnection: close\r\n\r\n' |
  nc www.example.com 80

About being valid or not, lot of the stuff on net is 100% compliant with RFC standards. You can also see
https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
So a space would create a problem at start but many of the servers would remove leading blank new lines


Exercise with Circle Permutations in Combinatorics

Question: I have a problem in solving an exercise. It says:
$10$ cars are about to park in a circle plaza, which $3$ of them are cars of permanent residents. In how many ways can the cars be put so as to:
a) None of the permanent residents' cars are parked in consecutive places with an another permanent resident's car ?
b) At least $2$ permanent residents' cars are parked in consecutive places ?
I translated the exercise from Greek so I apologise in advance if I have not phrased something correctly.
My thought :
a) From the total of 10 cars, 3 of them belong to permanent residents and the other 7 to "temporary" residents so I will refer to them from now on with $P$ and $T$ respectively.
It is obvious that we have a case of circular arrangement here ( I think that's how is called in English). We need a part of the elements of the circle to be always arranged in a specific way so I considered it a good idea to create a group of objects : $G = (PTPTP)$ which consists of the $3$ $P$ cars and any $2$ of the $7$ $T$ cars and with that if my thinking is correct we have to calculate the number of circles of 6 objects $(T,T,T,T,T,G)$ and multiply the result with $3!*\frac{7!}{5!}$ which I think that they are the circles of the group. That gets me $5!*3!*6*7$ which is far away from the answer of the book which is $6!*\frac{7!}{4!}$. Obviously I'm making something wrong.
b) I have no clue on how to solve b) any hint or help would be appreciated.
Comment: As for part (b), notice that the if it is not true that no permanent residents are adjacent then it must be true that at least two permanent residents are adjacent.
Comment: Now, I believe that I know what you mean when you say "circle square", but keep in mind that this is a math forum. We will think of "square" as a rectangle with equal sides before we think of "square" as a plaza.
Comment: Didn't have that thought when I was translating, I'm going to edit it and put plaza to avoid confusion.
Comment: To be clear, in our circular plaza, do we care which direction north is?  I.e. if we label the cars $1,2,3,\dots$ and we start listing which car parked where from the northernmost space going clockwise is the parking sequence $(1,2,3,\dots,9,10)$ considered the same or different than the parking sequence $(2,3,4,\dots,10,1)$?
Comment: JMoravitz I think it's considered the same since we're talking for circle permutations and not simple ones
Comment: LuxGiammi correct answer for b is $9!-6!*\frac{7!}{4!}$ forgot to add it
Comment: Sorry, I did not take into account one detail, and my comment contained a wrong answer.
Comment: Assuming the answer to my above question for clarification is that they are considered the same, a common tactic is to let one person be considered "special" for the purposes of the problem and orient ourselves based on their position.  For the first problem, suppose that Mr. X is one of the permanent residents and suppose WLOG that he parked in the northernmost space.  You can then pick in what order the non-permanent residents appear and then have the two other permanent residents squeeze inbetween.  This gives a total of $7!\binom{6}{2}\cdot 2$ which should match the given answer.
Answer: For a, I don't understand your logic.  You are missing configurations like $PTTPTTPTTT$.  Let us attach a $T$ after each $P$.  We can do that in $7\cdot 6 \cdot 5$ ways.  We can then put our seven objects in order in $7!$ ways.  We then divide by $7$ because each order can be rotated to start with any object.  That gives $7!\cdot 6 \cdot 5$.
For b, we can just compute the total number of orders and subtract the answer to a.  There are $9!$ orders that start with a given car, so the answer is $9!-7!\cdot 6 \cdot 5$
Comment: You're right I was missing a lot of configurations with my thought. Your results are correct but sorry for my ignorance but I didn't understood your thought. I'm not a native English speaker and understanding maths in English is even more difficult than speaking it. Could you please be a little more analytic ?
Comment: What are you having trouble with?  Attaching a T to each P seems to me like what you were doing and makes sure you don't have two Ps together.
Comment: Now i got it that phrase is equivalent to $\frac{7!}(4!}$ . Very nice answer, thanks once again.


Fewer variables have higher R-squared value in logistic regression

Question: I am testing out 3 modeling approaches for malnutrition in children.  Theoretically, distal determinants (education,poverty) operate through proximal determinants (water, sanitation) in determining malnutrition rates.  The three logistic models, where stunting is a binary indicator for malnutrition, are:
<code>// Proximal determinants only: both binary indicators
stunting ~ water + sanitation

// Distal determinants only: both categorical indicators
stunting ~ i.education + i.poverty

// Both proximal and distal determinants
stunting ~ water + sanitation + i.education + i.poverty
</code>
I am surprised to find that the r-squared value of the second model is higher than the third model, as calculated by the correlation between the predicted and actual values (stata):
<code>predict predicted, xb
corr predicted stunting
local rsq = r(rho)
</code>
While I expected the strength of the relationship and statistical significance of the more proximal causes to decrease (as they were soaked up by the distal causes), I expected the combined model to have higher explanatory power (as measured by r-squared).  Does anyone have any explanation as to why the second model has the most explanatory power?  Let me know if I can provide additional information for answering this question.
Comment: Regarding R-squared and logistic regression, you may want to take a look at this post: http://stats.stackexchange.com/a/3562
Comment: Just a quick check: do any of the independent variables have missing values? If so, that alone can cause the r-squared statistics to be incomparable.
Answer: You should be careful just relying on the R^2 when interpreting fit in a non-linear regression. You may want to compare the Log-Likelihood.
However, a decrease in R^2 with an increase in variable generally means the variables are interacting in a way that is not proving additional explanation of the model. One of the causes may be, as you point out, that there are issues with intervening variables in the model. If this is the case you may need to find an instrumental variable, or use a structural model.


Where Is This Configuration Coming From?

Question: I am helping a team whose builds have started failing due to test failures.
The failures are being caused by missing connection string configuration. I checked the usual issues in respect of the config file to ensure that the connection string was specified with exactly the right name.
In the end I obtained the full path of the config file to check that the one on the build server contained the exact configuration that was expected.
AppDomain.CurrentDomain.SetupInformation.ConfigurationFile

The path did not point to the TestProject.exe.config file, but instead pointed to the vstest.executionengine.x86.exe.Config at the following location:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.executionengine.x86.exe.Config

This file contains no connection strings at all.
When I write out all of the available connection strings from configuration, I get the default connection string:

Name: LocalSqlServer Connection: data source=.\SQLEXPRESS;Integrated
  Security=SSPI;AttachDBFilename=|DataDirectory|aspnetdb.mdf;User
  Instance=true . Aborting test execution.

This is coming from the machine.config file (kudos petelids).
So the big question is:
Why is the vstest.executionengine.x86.exe.Config being used rather than the app.config (at runtime TestProject.exe.config)? (I can guess that this is because the process running is the test runner, but I think it is fair to say that you would expect the test runner to let the test project use its own config file, which is what normally happens).
Comment: Maybe it's coming from the machine.config file? - `%windir%\Microsoft.NET\Framework\[version]\config\machine.config` or `%windir%\Microsoft.NET\Framework64\[version]\config\machine.config` for 64-bit
Comment: Just a guess may be it is defined in the registry
Comment: @petelids - spot on. It is using that because there are no connection string in `vstest.executionengine.x86.exe.Config` - so why on earth is it using `vstest.executionengine.x86.exe.Config` rather than the app.config for the test project?
Comment: @SteveFenton - I'm not too sure to be honest _but_ [this SO question may help](http://stackoverflow.com/questions/4811778/mstest-and-app-config-issue)? Note the EDIT/RESOLUTION in the question.
Answer:
I think it is fair to say that you would expect the test runner to let the test project use its own config file, which is what normally happens

That's perfectly true assumption when working with NUnit, xUnit, etc. but not with Microsoft Test Runners (MSTest and VSTest). They simply ignore target assembly config file and use their own config.
There are two solution to this problem:

Change MSTest to NUnit or xUnit
Use custom config file (not the default one)
Comment: I was under the impression you had to explicitly pass a flag to make the tests run under the same process as MSTest, otherwise it runs them in isolation by default: `/noisolation: Run tests within the MSTest.exe process. This choice improves test run speed but increases risk to the MSTest.exe process.` http://msdn.microsoft.com/en-us/library/ms182489.aspx
Comment: Yes but it will start new process that will use its own config file.
Answer: MSTest runs tests in isolated mode by default.
The solution to this issue was to add a new test project and move the tests into it.
The new project behaved like all of the other projects, running the tests in an isolated process and using the app.config file from the test project.
I'm putting this down to a quirk.


C++ - Code crashes when trying to sort 2d vector

Question: I'm writing a code in C++ and it's always giving the same error: Segmentation Fault, but I don't know why this is happening. I've created a small program that gives the error.
#include <vector>
#include <iostream>
#include <algorithm>

using namespace std;

int N = 17;
vector<vector<int>> v;
void func(vector<int>& x){
    for(int i = 0; i < N; i++){
        v.push_back(vector<int>(x));
    }
}

int main(){

    int n = 5;
    vector<int> v2(n, 0);
    func(v2);
    cout << v.size() << endl; // prints 17

    sort(v.begin(), v.end(), [&n](const auto &a, const auto &b){
        for(int i = 0; i < n; i++)
            if(a[i] != b[i]) return a[i] < b[i];
        return true;
    });

    return 0;
}

The function func is supposed to do a recursive computation. I pass v2 by reference because I push and pop elements from it. When I got to a certain point in recursion, I add it to the global 2d vector v.
My code's working to every test case I tried except this one.
The fun thing is that the programs works with no error when N < 17.
The error is happening at the lambda sort. There is an 'a' that breaks the code. Even if erase the for loop, it still crashes.
I know that using global variable is bad, but in competitive programming this is a common practice.
I'm compiling with the following:

g++ test.cpp && ./a.out

My g++ version:

(Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

EDIT:
Thanks to @DrewDormann, I managed to fix it.
Just changing the return of the lambda function to false instead of true.
Comment: Have you tried running this with a debugger or ASan? (for ASan, using compiler flags `-Og -g -fsanitize=address` may show more insightful error messages)
Comment: `std::sort` with a comparison predicate requires that `comp(a,b)` and `comp(b,a)` must never both return `true`.  That is, it can not report that `a` must be before `b` and `b` must be before `a`.  The `return true;` in your lambda violates this requirement.
Comment: `std::sort` sort comparator should compare two arbitratry container items which are given as `a` and `b`, yours is completely wrong
Comment: Thank you for helping. I've edited the question.
Comment: Consider using `std::lexicographical_compare` to implement your comparison function.
Answer: std::sort requires irreflexivity. From [alg.sorting] in the standard:

The term strict refers to the requirement of an irreflexive relation (!comp(x, x) for all x)

However, the lambda returns true for comp(x, x).

The fun thing is that the programs works with no error when N < 17.

It's undefined behavior. It's unlucky if it works by chance.