Why does deferred.resolve not behave as specified in the q manual?

Question: This may be a stupid question, but I am unable to wrap my head around this.
Consider the following piece of code:
function throwError() {
    throw Error("can't touch this.");
}

var def = q.defer();

def.promise.then(
  function() {
    console.log("no error");
  },
  function() {
    console.log("error");
  }
);

q.fcall(throwError).then(def.resolve, def.resolve).done();

In my opinion this should print error. Instead, it prints no error.
The q manual states:

The reject function is a shorthand for resolving with a rejected
  promise.

q.fcall(throwError) should produce a rejected promise, so the deferred should be rejected, right?
Please note that this is a purely hypothetical question, I know that this would not be used in the real world.
Answer: The issue here seems to be around the language used, so I'm going to try to break it down a bit more to make it clearer. When you call
var rejected = q.fcall(throwError);

then rejected is indeed a rejected promise. What you are then doing is
var def = q.defer();
def.promise.then(
  function(arg1) {
    console.log("no error");
  },
  function(arg2) {
    console.log("error");
  }
);

rejected.then(
  function(){
    def.resolve();
  },
  function(err){
    def.resolve(err); // err === new Error();
  }
);

When a promise is rejected, it will run the rejection callback, triggering def.resolve(err);. Q has no knowledge that def is anything other that some random arugment, so you might as well be calling def.resolve('random');. You are resolving def with some argument. In this case, it happens to be an instance of Error. That means that your no error callback will be called, and arg1 === new Error().
So now, given the documentation quote you mentioned

The reject function is a shorthand for resolving with a rejected promise.

what are you are doing is essentially transforming the rejected promise rejected into the promise def.promise which will always be fulfilled, but where the fulfillment handler will receive an error as the first argument.
To achieve what you are expecting, you need to properly make def.promise be rejected. You can accomplish this two ways.

Reject it using def.reject(err);.
As your quote says, resolve it with a rejected promise, def.resolve(rejected);.
Comment: Actually I wanted to use `def.resolve` twice. As I stated above, resolving with a rejected promise should trigger the error handler, not the fulfillment handler. So, IMO, the second function should be called, not the first one.
Comment: @raffomania Sorry, not following. `q.fcall(throwError)` does produce a rejected promise as you say, but since you have passed `def.resolve` as the error handler, it will resolve `def` without an error, just like if you called `def.resolve()` somewhere else.
Comment: Perhaps the confusion is around `resolving with a rejected promise.` What you are doing now is resolving with an `Error` object, not resolving with a rejected promise.
Comment: Please have a look here: [Using deferreds in q](https://github.com/kriskowal/q#using-deferreds). There you can see exactly my example. `q.fcall` should produce a rejected promise, not an error.
Comment: @raffomania Correct, the return value of `fcall` returns a rejected promise. Then you are calling `.then`, which extracts the error associated with the rejection, and you pass that error as the first argument. I'll expand my answer.
Comment: Now I understand. It seems so clear to me, I can't believe I didn't see this before. Thank you very much for this thorough explanation!


z score vs z score of z scores

Question: Suppose you have n time series, and you want to study the difference between the value of any pair of these time series. There seems to be two ways to go about this:

One way is to simply calculate the difference between the value of each pair of the times series, and look at the z score of the that difference.
The other way is, for each time series, calculate the z score that compares its current value with its historical value. Then for each pair of the time series, calculate the difference between their z scores and look at the z score of that difference.

Will the two method produce significantly different results?
Comment: Have you tried to check it yourself?
Comment: I am doing it empirically right now. Trying to do it theoretically seems a bit too complicated.
Answer: In the end, you are trying to compare two distributions.  The answers to the following question will help you solve the problem quickly and in a theoretical manner: Test to measure statistical significance of z-scores?
Another option would be to perform a one-way ANOVA of all $n$ data sets, assuming equal variance.  If the ANOVA identifies a significant difference in one or more groups, a Tukey multiple comparison procedure or pairwise t test can be conducted to identify the time series of interest.

z score vs z score of z scores


What is the profession / preferred way to handle input validation

Question: I am new to C# and SQL. But over the last few years while learning both in college a question really begins to burn inside me. Here it is:
It seems to me that there are really two very generic ways to handle input validation (i.e. checking for required fields, and data in the correct ranges ect).
The first, and the way shown traditionally is: Once you develop your UI, and have connected it to a database back end in some manner. On the user interface, you check for correct input, such as blank text boxes, number ranges, or to ensure a radio or check box is selected ect.
The second, and the way shown in database development is: To set check constraints on fields such as no nulls allowed, unique values, and even ranges and required fields.
My dilema is this. Given that in modern languages like C# you can do general execption handling, and also given that major league fault tolerance is built into most databases like SQL Server with regard to handling data changes in respect to committing all or none. Details like this, and to this level, would be hard to program in anything but the simplest of programs.
So my question is, why not build all the requirements directly into the table at the database back end. Take advantage of the aformentioned fault tolerance, and just forget about programming if statements to ensure correct data is input, and instead just use a generic catch all execption handler if the data is not committed.
Perhaps that is how it is done, if so I would really like to know for sure. If not, why? My preference is to avoid writing code whenever possible. Less code, less debugging, and less problems when it comes to updating. So I would tend to go with that approach of letting the DB back end do the work. Is this the generally correct thing to do.
I know that general execption handling is considered "expensive" in terms of resources. But surley once you get past 5 or 10 if statements to handle different fields and their constraints, it must be more efficient code wise to just do a general execption handler. It certantly seems easier to understand overall. (At least the way I do it).
Thanks for your help with this.
Comment: This cracked me up: "My preference is to avoid writing code whenever possible. Less code, less debugging, and less problems when it comes to updating."
Comment: interesting, i've often wondered about this myself
Comment: possible duplicate of [Where to put data management rules for complex data validation in ASP.NET MVC?](http://stackoverflow.com/questions/2881821/where-to-put-data-management-rules-for-complex-data-validation-in-asp-net-mvc)
Comment: @jack-t-colton - I do software development, the primary rule in our college is to always blame it on the DB guys! :->
Answer: OK, here is why you need it in both places.
First the integrity of the data should be paramount and data can be changed directly in database tables (deliberately through a script to say update a million prices or by accident or even by disgruntled or criminal employees trying to disrupt the database or steal from the company).  Therefore is it reckless to avoid using constraints directly in the database and it leads to bad data.
Now at the user interface level, you want to prevent the user from wasting his time submitting bad data and you want to prevent the servers and networks from wasting their time trying to process it, so you write checks at that level. Plus you don't want the data in an inconsistent state if you need to insert to several tables and aren't using a transction (which you should be using but I would suspect it happens less often than it should.) Plus the users hate it when you try the insert and it fails and tells you that X is wrong and then they fix X and now Y is wrong but it was wrong before, the process just didn't get as far as Y before.
Comment: +1 - the data itself is absolutely paramount, so put constraints and other rules as close to the data as possible (in addition to usability at the app tier). If you miss something in the app, it's better that the app fails than if the database gets corrupted.
Comment: For SQL server look at Transactions in Books Online for the SQL code to do transactions. Also look at try catch.
Comment: +1 - I totally agree with the DB integrety. As pointed out, DB's have extream fault tolerance built in. So for example, a partial recordset cannot be input, its either an all or nothing affair. I also like your point regarding the network traffic. This would seem like a good reason to do checking on the UI, without them you have to submit the data and wait for the DB to tell you it is not correct. I must learn about using transactions. Thank you for highlighting this, can you recommend a good resource. Finally your point on X and Y both wrong. Good design makes sense. Thank you for your answer
Comment: After some time and taught I have accepted this as the final answer because you hit on two important points that I do consider very important. First the fact that data integrety is paramount. Second only to this, is the user experience (good design). Typically DB people say let the UI handle the error checking, less network traffic, more secure, all that stuff. While programmers say let the DB handle it, more fault tolerance, transaction mgmt ect. You have hit on two very different, but much more important and practical considerations. I still prefer to let the DB do the work though! :->
Answer: You do both.
Create constraints at the DB - level, and check for those constraints on the client level as well.
The validation on the DB makes sure that no invalid data gets in your DB, no matter how the data is inputted.
The validation at the client side improves the user-experience.
Comment: I agree with your arguement totally. With one execption. When handling a general execption you can handle specific cases first, and less and less specific cases, right back to a generic case. You can do this in one block of code and the user wont know any different. i.e. a number out of range. This will generate an error when the record is comitted to the DB which your program can then take in and handle as a specific execption. As opposed to actually writing code behind the UI or in some logic elsewhere and duplicating exactly what the DB will do for you for free. Thanks for your answer.
Comment: Sometimes you need a data access layer between the client and DB where you can implement validation of additional business rules, or complex constraints that the DB engine is unable to represent elegantly.
Comment: So are you saying then that the examples taugh in college are just that, examples for tutorial. In the real world we will let the DB handle errors when possible or elegant (such as the simple examples I gave), and only program when really needed for very special cases.
Answer: You generally can't build all the logic for checking into the database. Also not validating user input sufficiently is a good way to open yourself up to attack.
One way to write lesss guard code in every method is 'Code Contracts' a product of microsoft research.
All input should be validated both client and server side. Always.
Also with a giant catch it would be hard to tell which field was in error. So you would end up writing a lot of which field exploded code at the other end.
Comment: +1 I must make effort to learn about "code contracts". I have never heard of them before. Can you recommend a good resource for beginners. Another great point is the security issues you mentioned. We have not covered that yet. Are code contracts designed to deal with security also. If so then I am still unconvinced however of the need write code for handling every possible type of user input as opposed to using an execption handler. Thanks for your answer.
Answer: While I generally advocate putting as much in the database as possible (which means that you can have a high degree of confidence about the "raw" data as possible), that isn't always possible, even with the powerful constraints and triggers available in SQL.
In addition, there are high-level "integrity" things which may change over time, and it is not realistic to always have temporally-dynamic conditions in constraints.  i.e. all HR records since 2007 must have a non-NULL birthdate, but prior ones are allowed to remain NULL, but any row cannot ever be set back to NULL.
My point is you can almost never put it all in the database.
Put the things in that you can, and put others at higher levels in the system.  The database is a very important part of any system, but it isn't the only part.  As long as its design helps it protect its perimeter and be able to provide reliable service and guarantee what it says it will guarantee so that other parts of the system can rely on their assumptions, then that's about the most you can ask for.
Answer: In addition to all answers made here, like that UI control improves drammaticaly UX for the user, and can completely change "image" of your app, that validation on DB is made for correct insert the data to DB, but on client it have to be done for correct insert of the client data.
Consider an example of standalone enterprise app. A client work at home, he filled 20 invoices late night on his notebook in Mongolia. The day after he came back, and sync it with his office SAP server. If the error will be figure out only during sync of the data, you can imagine what  awful is this situation.
Just an example. There could a plenty of others, I'm sure.
Good luck.
Comment: Good example. But surley you would program you client side database on your notebook with the same constraints as those that exist on the server. Doing so will ensure your data can be committed the next day without failure while still not requiring all the extra code behind the UI. Thank you for your answer.
Comment: @Francis: by saying "compile" in my post I mean the hypothetical client fills invoice's form of some app that perform only DB input data validation, I'm not talking about programming a software :)
Answer: Its 2 years later and I have a decent amount of experience now. I am not going to accept my answer as the right one as many here have done a great job and I am very happy with their answers. But I want to add another important consideration that looking back over my experience has not been highlighted here. I also use stack overflow for reference as I progress and I always find myself looking back over my questions and answers which is another reason I wanted to add this. Like a note to my future self.
While working at that company, I was asked to build an app that would do job abc. With this I also had to build part of the database. As I was finishing with the company I learned that they were writing another app which would use my database. Effectively my point is, that as many have pointed out, data is paramount, and you don't know how it is going to be accessed when you're gone.
I have also learned that there are 3 places that data needs to be verified:

on the actual database as explained
on the server side code behind which is not the same as the DB or client side validation
on the client side

There is another worry. With the advent of new tech like tablets and smart phones. This is yet another place where validation has to be implemented. The same rules for a 4th time (unless its a web app).
I later learned that prior to MVC we had CGI forms which had something to do with handling data over the network (I humbly admit ignorance on hardware side) but from what was explained to me it seems there may even be a 5th place to do validation (although I am open to being totally wrong about that).
I think the next guru in computer science will make a name for himself if he can find a way to abstract all that verification and validation to one place so that such rules don't have to be altered in a bunch of places.
worst case:

DB
Server side code
Client side code for web apps

What about if:

There may be a native client app (i.e. windows, linux or mac (at least 6 now))
There may be various phone apps (android, iPhone, and win phone to name 3, at least 9 now))
There may be some CGI or whatever

This totals 10+ places without much exaggeration and there are other operating systems.
Even for a simple age range this is getting to be messy, but what if they bring out some new email format, or other complicated validation, or you have to change a bunch of validation rules. Now you have to modify them across at least 3 or 4 places which in itself is bad.
The major problem with that is that you are modifying a lot of code and infrastructure that has been invested in, tested, and usually proven to work and delivered to the market...
As the number of client sides grow, modifying well tested code, can't be a good thing. I think this is going to be a major headache for the future. I wonder if there will be a design pattern or best practice to resolve it. If anyone knows of one, please tell me.


Intuition behind why order of a subgroup divides order of a group.

Question: I'm trying to get some intuition on why order of a subgroup divides order of a group. I know Lagrange's Theorem and its proof (every element of a group is an element of some coest, cosets are disjoint and each one has the same number of elements as subgroup), but the proof is more like magic than any intuition.
So I've been trying to prove this fact (divisibility of order) without Lagrange theorem. Let's say we take a group $H$ and try to extend it, by taking $g \notin H$ and considering group $G$ generated by set $H \cup {g}$. Such a group would have to contain $g^{-1}$ and every word over alphabet $\{g, g^{-1}, H\}$. Is it possible to prove intuitively that such construct is of size $|H|\cdot n$ for some $n \in \mathbb{N}$? This would essentially prove that any group $G$ containing $H$ has order divisible by $|H$|, because we would be able to take $g \in G$ such that $g \notin H$, extend $H$ by it just like above and continue this process until $H$ is extended into $G$.
Comment: The intuition for Lagrange's Theorem is this: We have a bag called G, and inside are a bunch of bags called H, all of the same size. So the number of elements in one small bag divides the total number of elements. All the small bags have the same size because "multiply by g" is a bijection, with "multiply by g^{-1}" as an inverse
Comment: No, it is much more than that. Read it a bit longer. The comments at the duplicate are very positive! By the way, [this article](https://www.jstor.org/stable/2690624?seq=1) explains the history of Lagarnge's theorem. This is also very interesting.
Comment: Your main question is harder. I'll think about and come back if I get something
Comment: This questions has been asked many times here. See for example [this duplicate](https://math.stackexchange.com/questions/336812/understanding-lagranges-theorem-group-theory) and its links. It has amazing visualisations and very good explanations.
Comment: And every single one of them is just repetition of the proof
Comment: @user128409235: It might be disappointing for you, but I think it is the only proof known. It is also very simple and known since around 150 years ago. I would not expect that a different proof would exist. In any case, if you want to prove the result, what can you do? You somehow need to show that $G$ can be split into parts of equal size $|H|$ --- in the proof we construct such a partition explicitly.
Answer: Here's a (partial) response to the question in the title.
It's easy to develop that intuition for finite cyclic groups. Division with remainder implies that the order of every element is divides the order of the generator, which is the order of the group. Then it's only a short step to see that any subgroup of a cyclic group is cyclic.
I don't know whether the argument in the body of the question is easy to complete.
Answer: Let $H$ be a subgroup of the finite group $G$. For each $g\in G$ the set $gH$ has exactly $|H|$ elements. It is easily seen that two such sets   with a common element are in fact equal. Therefore $G$ is partitioned into finitely many subsets, each of size $|H|$.
Comment: Yes, that is the standard proof. But for some reason the OP finds it "more like magic" and wants a different proof.
Answer: As far as I know, the only known proof of Lagrange's theorem is the standard one.
That is, cosets of $H$ partition $G$ and each coset has the same size; thus $|G|= |H|[G:H]$.
I believe the following is how Jordan proved Lagrange's theorem, which to some people might be easier to understand. But the proof really is the same old argument.

Proof: List the elements of $H$: $h_1,h_2,h_3, \ldots$.
Let $p$ be some element not in this list. Then $ph_1, ph_2, ph_3, \ldots$ are all distinct and none of them lie in $H$, as you can easily prove.
Next let $q$ be some element not among
  $$h_1,h_2,h_3,\ldots$$
$$ph_1,ph_2,ph_3,\ldots$$
You can show that $qh_1, qh_2, qh_3, \ldots$ are all distinct and none of them are among the elements above.
Continuing in this manner, we eventually exhaust all elements of $G$ (since $G$ is finite):
$$h_1,h_2,h_3,\ldots$$
$$ph_1,ph_2,ph_3,\ldots$$
$$qh_1,qh_2,qh_3,\ldots$$
$$rh_1,rh_2,rh_3,\ldots$$
$$\vdots$$
So we conclude that the order of $H$ divides the order of $G$.

Note that in the proof we count the elements of $G$ by first listing the elements of $H$, then the elements of $pH$, $qH$, $rH$, $\ldots$
So it is really pretty much the same as the usual proof.


Negation, and supremum proof

Question: Let $A \subset \mathbb{R}$ be non-empty and bounded from above. Let $\epsilon>0$ show that
$\exists a \in A:A>sup(A)- \epsilon$
My hint is to form a negation of the statement so it should be: ?
$\neg \exists a \in A:A \leq sup(A)- \epsilon$
My attempt:
Let $a \in A$. Since $A$ is bounded from above $\exists$ a number $b \in \mathbb{R}$ s.t. $a \leq b$. But I am not really sure on how I should use the property of supremum in this case.. any hints is appreciated
Answer: You probably meant $\exists a\in A[a>\sup A-\epsilon]$.
The negation is not $\neg\exists a\in A[a\leq\sup A-\epsilon]$ but $\neg\exists a\in A[a>\sup A-\epsilon]$.
Further this is equivalent with $\forall a\in A[a\leq\sup A-\epsilon]$.
That actually tells us that $\sup A-\epsilon$ is an upper bound of $A$.
However we have $\sup A-\epsilon<\sup A$ so this contradicts that $\sup A$ is the least upper bound of $A$.
Our conclusion is that the negation of the statement is not true, hence the statement is true.
Comment: so it contradicts the first property of least upper bound?
Comment: what do you mean with "first property of least upper bound"? Characteristic for the least upper bound of $A$ is that all elements smaller that it are no upper bounds of $A$. This purely on base of definition of least upper bound.
Comment: thanks I will keep this in mind

Negation, and supremum proof


Exam FM problem: Financial calculator necessary for finding $i$ from $a_{\overline{n}\rceil i}$? Edited

Question: I am currently studying for the Exam FM for actuaries, and the calculator that I have is a TI 30X IIS, which was very helpful for me during the Exam P.
I cam as far as studying bonds, and the following is what I have been working on.

A 1000 par value bond pays annual coupons of 80. The bond is redeemable at par in 30 years, but is callable any time from the end of the 10th year at 1050.  Based on her desired yield rate, an investor calculates the following potential purchase prices, $P$:
1), assuming the bond is called at the end of the 10th year, $P=957$
2), assuming the bond is held until maturity, $P=897$
The investor buys the bond at the highest price that guarantees she will receive at least her desired yield rate regardless of when the bond is called.  The investor holds the bond for 20 years, after which time the bond is called.  Calculate the annual yield rate the investor earns.

For this problem this is what I have so far.
1), $957 = 80a_{\overline{10}\rceil j}+1000v^{10}_j$
2), $897 = 80a_{\overline{30}\rceil j}+1000v^{30}_j$
It does not look like the problem is designed so that the present value factors $a_{\overline n \rceil}$ cancel out nicely, and yet this question came from the exam so I am wondering if a financial calculator is needed to solve this problem. I would love it if someone could tell me the following. (it would be most helpful if you have actually taken the Exam FM)
a), Is it possible to solve this problem with a basic scientific calculator?  If so, how do we get rid of the annuity-present-value factor?
b), Is a financial calculator necessary for the Exam FM?  I know that BA II+ is allowed for the exam and it seems like it is a financial calculator rather than a scientific one.
P.S.
I found that the yield rate can be calculated as $j=9\%$ from the first two information, but I am still having trouble finding the annual yield rate the investor earns.
The answer is supposedly $9.24\%$.
I think that the present value of the bond at $t=20$ is
$$P=80a_{\overline{20}\rceil .09}+1050(1.09)^{-20} \approx 917.64$$
so since the investor is purchasing the bond at the minimum yield which happens at the end of $t=30$, she buys it at $897$ so
$$897(1+i)^{20}=917.64(1.09)^{20}$$
which gives me $i \approx 9.12\%$
Comment: I own the BA II+ Professional.  I'm sure the regular version is plenty good enough.  As far as any other exam you take, the advantage of the multiview will be clear after you use it for a few minutes.  You can see around 4 lines of stuff at once.  I can't find my TI 30X IIS right now so I can't say much more than that as to why it's better.  Look for a comparison online.  Think of it this way.  It's $20, it may never help you pass an exam (over having the TI 30X IIS), but it'd suck if it could have and you didn't have it.  An exam is worth thousands, I'm talking $20 one time.
Comment: Buy a financial calculator.  You're at a disadvantage if you don't have one.  Just because there exists a person who did in fact take the test without one doesn't mean it's the good thing to do.  Second, for tests other than FM, buy a TI 30XS Multiview.  Again, without it, you're at a disadvantage.
Comment: I've never bought a financial calculator before, so the one that I know that is suggested BA II+... do you think that it's a good one?

For the other exams, I passed the Exam P with TI 30X IIS (I think it's different from multiview).  Are you suggesting that it would be advantageous for the exams that I would take later?
Comment: Thank you very much, that is actually very encouraging.  I think I will buy one.
Answer: I used a plain Texas Instruments non-financial calculator for exam FM. The benefit of multiple choice problems is that there are a finite number of possible solutions. Once you've set up the correct equation, and you've discovered it's too hard to solve without numerical methods, you can just plug the different answers in and see which one works.
Since the expressions are usually either increasing or decreasing, you can also use binary search. You plug in a number that you think might be high, then one that you think is low, then plug in their average, and if that's too high then you plug in the average of the middle and the lower one, etc.
Comment: Ah, what was I thinking.  I totally forgot that it was multiple choice.
Comment: @hyg17 see another possible solution in my edit.


How can can I get theme callback to work for an error page?

Question: I've created a custom error handler for 404 errors (page not found).
I've set up Home » Administration » Configuration » System  » Error Pages for 404 errors to point to http://example.com/myerrorpage, and then I have the following in hook_menu
/**
 * Implements hook_menu().
 */
function MYMODULE_menu() {
  $items = array();
  $items['myerrorpage'] = array(
    'title' => 'myerrorpage',
    'access callback' => TRUE,
    'page callback' => 'myerror_page',
    'theme callback' => 'myerror_theme',
    'type' => MENU_CALLBACK,
  );
  return $items;
}

When I enter a non-existing path (e.g. http://example.com/bogus) the page callback is invoked as expected.
However, the theme callback is not.
To get the  theme callback to fire, I need to enter the following path: http://example.com/myerrorpage.
I do not understand why the theme callback does not fire at the same time as the page callback.
Is there any way to get the theme callback (or hook_custom_theme()) to fire when a non-existing path is given?
Comment: Looks like a bug in core for me. I would expect it to work just like aliases do.
Comment: At first glance this actually does look like a bug (or at least an undocumented limitation) - `menu_get_custom_theme()` is called earlier in the bootstrap then `drupal_deliver_html_page()`, so overriding the menu item to the 404 page in that second function can have no effect on the current theme callback
Comment: As a nasty hack you could provide your own delivery callback function and `unset($_GLOBALS['theme']); drupal_theme_initialize();` between the calls to `menu_set_active_item()` and `menu_execute_active_handler()`. Not pretty, but it should work
Answer: In menu_get_custom_theme(), you can see that theme callback on menu router would not be called if the menu router doesn't exist. But you can achieve that with hook_custom_theme().
I'll go like this:
/**
 * Implements hook_custom_theme().
 */
function MYMODULE_custom_theme() {
  $router_item = menu_get_item();
  if (empty($router_item)) {
    return 'THEME_NAME';
  }
}


Can't Retrieve Text Data Posted By Form Tag in Express JS

Question: I am developing a simple pipeline that when a user highlights a text block in the chrome page and click the extension, it shows the highlighted text inside the  defined in popup.js of chrome extension and send the text highlighted to the server running in Express.js.
The current status is popup.html running the popup.js and popup.js writes down highlighted text into document. So far, it works well.
However, the running terminal of node app.js keep showing undefined whenever I hit the chrome extension favicon. I expects the highlighted text shown up in the terminal log.
What would be the possible reason for this wrongful action?
popup.html

        <meta charset="UTF-8">

popup.js
chrome.tabs.executeScript( {
    code: "window.getSelection().toString();"
}, function(selection) {

    document.write(selection[0]);

    var text =
        '<form action="http://localhost:8080/example" method="post" id="hlgt_form">' +
        '<input type="hidden" id = "hlgt" name = "hlgt" value= ""> ' +
        '</form>';

    document.write(text);

    document.getElementById('hlgt').value = selection[0];
    // it stores highlights into value of <input>

    document.getElementById('hlgt_form').submit();
});

app.js (run by node app.js)
const express = require('express');
const app = express();

app.use(express.json());

const port = 8080;

app.listen(port, () => {
    console.log(`Server running on port: ${port}`);
});

app.post('/example', (req, res) => {
    console.log(`${req.body.name.hlgt}`);
});

Answer: I think the problem is in your express app. You don't seem to be using the body-parser middleware to actually obtain the HTTP POST data in your request, and even if you did, you would reference that info as req.body.hlgt, without .name.
Try something like this:
const express = require('express');
const bodyParser = require('body-parser');
const app = express();

app.use(express.json());
app.use(bodyParser.urlencoded({ extended: true }));

const port = 8080;

app.listen(port, () => {
    console.log(`Server running on port: ${port}`);
});

app.post('/example', (req, res) => {
    console.log(`${req.body.hlgt}`);
});

Answer: Write only these two lines
app.use(bodyParser.urlencoded({ extended: false }));

app.use(bodyParser.json({limit:"5MB"}));


Cant Build a Docker, because NPM fails?

Question: im trying to Build a APP with Docker, but it fails for NPM problems, and im not sure whats the problem?
it even says NPM is outdated but i updated it already...
the CLI also cant or wont make a logfile for this
root@ubuntu:/var/www/html# docker build -t booru ./booru
[+] Building 16.5s (9/19)                                                                                docker:default
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 685B                                                                                0.0s
 => [internal] load metadata for docker.io/library/node:18-alpine                                                                                0.3s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 45B                                                                                0.0s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 2.67kB                                                                                0.0s
 => [stage-1 1/6] FROM docker.io/library/node:18-alpine@sha256:ca9f6cb0466f9638e59e0c249d335a07c867cd50c429b5c7830dda1bed584649                                                                                0.0s
 => CACHED [stage-1 2/6] WORKDIR /app                                                                                0.0s
 => CACHED [builder  2/10] WORKDIR /build                                                                                0.0s
 => CACHED [builder  3/10] ADD ./package.json ./package-lock.json ./                                                                                0.0s
 => ERROR [builder  4/10] RUN npm ci                                                                                16.1s
------
 > [builder  4/10] RUN npm ci:
15.89 npm notice
15.89 npm notice New minor version of npm available! 10.2.4 -> 10.4.0
15.89 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.4.0>
15.89 npm notice Run `npm install -g email@example.com` to update!
15.89 npm notice
15.90 npm ERR! code 1
15.90 npm ERR! path /build/node_modules/sharp
15.90 npm ERR! command failed
15.90 npm ERR! command sh -c (node install/libvips && node install/dll-copy && prebuild-install) || (node install/can-compile && node-gyp rebuild && node install/dll-copy)
15.90 npm ERR! sharp: Please see https://sharp.pixelplumbing.com/install for required dependencies
15.90 npm ERR! sharp: Installation error: Invalid Version: 1.2.4_git20230717
15.90
15.90 npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-02-27T10_19_21_922Z-debug-0.log
------
WARNING: current commit information was not captured by the build: failed to read current commit information with git rev-parse --is-inside-work-tree
Dockerfile:6
--------------------
   4 |
   5 |     ADD ./package.json ./package-lock.json ./
   6 | >>> RUN npm ci
   7 |
   8 |     ARG NEXT_PUBLIC_NO_INDEX
--------------------
ERROR: failed to solve: process "/bin/sh -c npm ci" did not complete successfully: exit code: 1
root@ubuntu:/var/www/html# dir /root/.npm/_logs/
root@ubuntu:/var/www/html#

Update an APP with Docker and NPM
Comment: The NPM notice to update your version is not an error but a notice. Have you tried to inspect the og file mentionned ? `/root/.npm/_logs/2024-02-27T10_19_21_922Z-debug-0.log`
Comment: yes, but for some  reason there is no such log file, the folder is empty :(
Comment: The logs are located in the container as you run the npm command within the build. See: [Docker Build Npm Error Logs](https://stackoverflow.com/questions/24192210/docker-build-npm-error-logs) to retrieve your npm logs and edit your question to add them if possible
Answer: It looks like dependencies are messed up Please see https://sharp.pixelplumbing.com/install for required dependencies .  But also, you have updated npm but not to the latest version. Try these, making sure all dependencies are installed and updated and update is if it hasn't fixed it
Comment: Thanks i did

`xxxx@ubuntu:/var/www/html$ sudo npm install -g firstname.lastname@example.com

changed 14 packages in 2s

24 packages are looking for funding
  run npm fund for details`

but it still says

`15.60 npm notice Run npm install -g email@example.com to update!`
Comment: It's because you updated your npm on host device, the notice is for the npm version inside your container so you'll have to update your npm version in your container before running the `RUN npm ci`

Cant Build a Docker, because NPM fails?


Analytical solution for optimization problem

Question: Is there an analytical solution for the following optimization problem:
$b, y \in \mathbb{R}^n$ (vectors), $b^* = \arg min_{b \in \mathbb{R}^n} f(b)$, where $f(b) = ||b-y||_{2}^{2} + \varphi ||b||_{1}$ where $\varphi > 0$
Its so close to $L_1$ regression, but I cant solve this even in that way.
Comment: $\phi$ is a constant which greater than 0
Comment: Ah--so $\phi$ is not a function.  This problem is called the "Lasso."  See https://stats.stackexchange.com/search?q=lasso+closed+form for the general problem.  Since yours is so specific, it is plausible it would have a specific answer (+1).
Comment: Welcome to CV.  It would all depend on what "$\varphi$" means: that detail is essential for answering your question.
Answer: Using vector math re-writing your problem, also replacing b with X:
$$f(X)=(X-Y)^T(X-Y)+\varphi |X|^T 1$$
First Order Conditions:
$$f(X)'=0$$
$$2(X-Y)+\varphi \space\mathrm{sign}(X)=0$$
Solution:
$$X=Y-\mathrm{sign}(X)\varphi/2$$

when $|Y_i|\ge\varphi/2$:
$$X=Y-\mathrm{sign}(Y)\varphi /2$$
when $|Y_i|<\varphi/2$:
$$X=0$$
Comment: @whuber, I spelled it out, 1st order conditions
Comment: +1.  But what does "F.O.C." mean?


OS upgrade before or after P2V migration

Question: We are in the phase of initiating P2V migration of old hardware servers with possible OS upgrade in order to free some space in DC and reduce energy consumption.
Some of the servers will be merely P2Ved, some requires OS upgrade.
I've been thinking on how better to achieve the goal for server which require OS upgrade, but it seems to be not so trivial and applicable to all servers that we have.
Most of the servers are Windows Server 2003 machines and it's planned to upgrade them to Windows Server 2012. Since there is no direct upgrade from 2003 to 2012, it seems more logical to create a new VM with Windows Server 2012 and then transfer application and data to it.
What else are the reasons NOT to perform migration with subsequent OS upgrade so I can have more arguments to convince customers to the approach with creating a new VM and transfering application data?
Comment: Don't upgrade from 2003 to 2012, do a fresh build in a VM and migrate data and applications, it would be a complete waste of time to go from 2003>2008 R2>2012
Answer: Assuming you have the licenses and the room on your VM server, your best option is going to be to stand up your new 2012 servers and do a standard migration. You will likely end up with headaches by trying to do a double upgrade from 2003 to get to 2012, stopping at 2008 R2 in the middle.
If you have servers that you absolutely need to upgrade rather than rebuild, you could always do the P2V on the old OS, then upgrade inside the VM, knowing that if it fails miserably, you still have the physical box to fall back on.
Comment: To add...it is also very likely that the servers running 2003 are also running older version of the apps on them, requiring them to be upgraded as well in the process.  Doing a true migration to both an up to date OS __AND__ updated apps will allow for a clean new base to work from.


What options exist for self service password reset for Exchange Online users

Question: We recently migrated Exchange on-premise to Exchange Online. We now have the issue where remote users cannot reset their password for our on-prem AD(previously they could use the exchange web app).
The only option I'm seeing is to pay for Azure AD premium which allows password write-back to our local AD.
What if we deployed a domain controller in Azure or Azure domain services. Would we be able to get self-service password resets doing this or would we still need Azure AD premium?
Any other options out there that aren't $6 a user?
Thanks!
Comment: Dang, nobody out there has ideas?
Answer: Using Azure domain services or adding a domain controller in Azure will not help. I think you can still use OWA for exchange online to reset the password (but I am not an exchange guy, so cannot confirm). If exchange does not have that option, then you will have to use premium licenses.
Comment: Yeah, premium license or 3rd party seem to be the only options. Thanks!


Thymeleaf passando 2 parâmetros para URL

Question: Boa noite pessoal,
Estou com o seguinte problema:
Estou tentando passar 2 parâmetros na URL utilizando o Thymeleaf, porém não estou conseguinto recuperar o valor da variável "nome" para passar no segundo parâmetro.
Obs.: Euconsigo passar o número da página.
Por gentileza, poderiam me ajudar?
Controller:
@GetMapping("/listar")
public String listar(@RequestParam(name = "nome", defaultValue = "") String nome,
        @RequestParam(name = "page", defaultValue = "0") int page, ModelMap model) {
    if (!nome.isEmpty()) {
        model.addAttribute("produtos", produtoService.findByNome(PageRequest.of(page, 5), nome));
        model.addAttribute("nome", nome);
    } else {
        model.addAttribute("produtos", produtoService.listarTodos(PageRequest.of(page, 5)));
    }
    model.addAttribute("currentPage", page);
    return "produto/lista";
}

HTML que quero passar os parâmetros:

Answer: consegui resolver o problema.
A montagem da URL estava errada. O correto seria:
th:href="@{/produtos/listar?page=${i}&nome=${nome}}"


ROC curve for outcome variable and individual variable

Question: Suppose we have a binary outcome variable $y$ and we plot an ROC curve with a predictor variable $x$. What is the interpretation of this? An ROC curve plots the FPR (false positive rate) and TPR (true positive rate) across all thresholds. Here, we don't have any probabilities/thresholds.
Comment: The ROC is FPR/TPR. I don't understand the question.
Answer: I think it's unorthodox, yet I've seen it done, this approach is common in credit scoring. It's calculated this way:

Sort your $X$ variable in ascending order
At each point $x_i \in X$ (or every $n$ points if the set is too big) calculate:
$b_i = (\#$ of $0$'s in $Y \, | \, X < x_i)/(\#$ of $0$'s in $Y) =$ propotion of negative observations when $X < x_i$ and all negative observations.
$g_i = (\#$ of $1$'s in $Y \, | \, X < x_i)/(\#$ of $1$'s in $Y) =$ propotion of positive observations when $X < x_i$ and all positive observations.
Plot each pair $(b_i, g_i)$ and you receive a curve similar to ROC.

Interpretation is quite analoguous to the ROC curve - area under the curve you just plotted will be a measure of discriminatory power of given variable. Line y = x will indicate that variable X won't be a good predictor. If the curve is convex, the variable is promising, you should check how it performs in a model.


Understanding THE_REQUEST variable in RewriteCond

Question: RewriteCond %{THE_REQUEST} ^(.+)\.php([#?][^\ ]*)?\ HTTP/
I understand everything in the line above except for the following segment:
([#?][^\ ]*)?\ HTTP/
I've done some research and found out that the square brackets are used to match any one of the characters within them. However, I have also learned that ? is used to make the preceding token optional and that ^ means "match start."
As such, why do the square brackets in the segment above contain both ? and ^? I thought that the square brackets were simply used as a "character class."
Also, what is the purpose of HTTP/ in the segment specifically? All of my searches have come to no avail.
Answer: First understand what is THE_REQUEST.
THE_REQUEST variable represents original request received by Apache from your browser and it doesn't get overwritten after execution of some rewrite rules. Example value of this variable is:
GET /index.php?id=123 HTTP/1.1

Now the part you want more clarification on:
([#?][^\ ]*)?\ HTTP/

Here is what is happening here:

It matches ? or # literally since inside [...] almost all special characters are matched literally
Followed by a negated character class that matches 0 or more characters of anything except a space
? after ([#?][^\ ]*) makes it an optional match
\ HTTP matches a space followed by HTTP

Now let me tell you that matching # is not needed here since a web server never receives a URL after #. That is all handled by client browsers.
It is better to use this RewriteCond:
RewriteCond %{THE_REQUEST} ^(.+)\.php(\?\S*)?\ HTTP/ [NC]


How to scroll to the first invalid input in angular forms?

Question: I am working in an big angular Project. how can i scroll to the first invalid input in angular forms without using jquery ?
Comment: You need to add a working snippet or a link to codepen/jsfiddle with your issue - https://stackoverflow.com/help/minimal-reproducible-example
Answer: You can do this way:-
<code>myApp.directive('accessibleForm', function () {
return {
    restrict: 'A',
    link: function (scope, elem) {

        // set up event handler on the form element
        elem.on('submit', function () {

            // find the first invalid element
            var firstInvalid = elem[0].querySelector('.ng-invalid');

            // if we find one, set focus
            if (firstInvalid) {
                firstInvalid.focus();
            }
        });
      }
   };
});
</code>
Comment: in which file i should write this code?
Comment: You can use it in directive like above.


Recall that $ p \rightarrow \sim q$ is equivalent to $p \land \sim q$, how can this be used as an explanation for how to use proof by contradiction.

Question: Recall that $p \rightarrow \sim q$ is equivalent to $p \land \sim q$, how can this equivalence be used as an explanation for how to use proof by contradiction.
I'm having a hard time answering this question because I do not understand the role of $p \land \sim q$. So i know $p \land \sim q$ means $p$ and not $q$ but how does this play a role in proof by contradiction.
I get $p \rightarrow \sim q$ because that is basically the procedure of how a person uses proof by contradiction.
Any insights?
Comment: Recall that $p \to \lnot q$ is **NOT** equivalent to $p \land \lnot q$: try with a truth-table.
Comment: Recall $\neg(p\to q) ~\iff~ p\wedge \neg q$ which is more useful in proof by contradiction.
Answer: Just a small comment, you wrote that $p\rightarrow \neg q$ is equivalent to $p \land \neg q$. Didn't you want to say that $p\rightarrow \neg q$ is equivalent to $\neg p \lor \neg q$?
Now suppose you are trying to prove that $p\rightarrow q$ by contradiction. Then you assume that $\neg (p\rightarrow q)$, which is $p\land \neg q$. Now once you have proven that $p\land \neg q$ is false, you have $\neg \neg(p\rightarrow q)$, which is $p\rightarrow q$.
Answer: The "general form" of a Proof by contradiction is:

we have to prove a proposition $P$ from a set $S$ of axioms or earlier theorems of the theory we are working in. We consider $\lnot P$ in addition to the set $S$ of premises; if this leads to a contradiction $\mathbb F$, then we can conclude that the set of premises $S$ implies the negation of $\lnot P$, i.e. $P$.

In symbols:

if $S \cup \{ \lnot P \} \vdash \mathbb F$, then $S \vdash P$.

Consider the following example of a proof by contradiction of a conditional statement:

Suppose $a \in \mathbb Z$. We have to prove: if $a^2$ is even, then $a$ is even [here the claim we have to prove has the form: $P \to Q$].

Proof. For the sake of contradiction suppose $a^2$ is even and $a$ is not even. [i.e. we are assuming $P \land \lnot Q$, which is the negation of $P \to Q$].
Then $a$ is odd and thus there is an integer $c$ such that $a = 2c +1$.
Then $a^2 = (2c +1)^2 = 4c^2 +4c +1 = 2(2c^2 +2c)+1$, which means that $a^2$ is odd.
Thus $a^2$ is even and $a^2$ is not even, a contradiction [i.e. we have derived $P \land \lnot P$].

Thus, having arrived at a contradiction, our original supposition that $a^2$ is even and $a$ is odd could not be true, and we have proved the theorem.


VSTS REST API Error when using Basic Authorization (username & password)

Question: Using postman tool I generated a bearer token with Headers, Basic authorization with username and password and Body.
I am getting an error when I am trying same through web service request in VSTS.
What is the syntax of providing username password with basic authorization?
Comment: You state you are getting "an error". **What** is the error?
Comment: Take a look at the specification: https://tools.ietf.org/html/rfc7617
Answer: If you have PAT(Personal Access Token), you can use that for Basic authentication with no username and PAT in password field.
Answer: You can try below sample to use the username and password with Basic authorization to call the REST API:
C#:
var httpClient = new HttpClient();
var byteArray = Encoding.ASCII.GetBytes(string.Format("{0}:{1}", "Domain\\username", "password"));
httpClient.DefaultRequestHeaders.Authorization
                    = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
var response = await httpClient.GetStringAsync("https://account.visualstudio.com/_apis/git/repositories");
Console.WriteLine(response);

PowerShell:
Param(
   [string]$collectionurl = "https://account.visualstudio.com",
   [string]$project = "ProjectName",
   [string]$user = "username",
   [string]$token = "Password/PAT"
)

# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))

$baseUrl = "$collectionurl/$project/_apis/wit/reporting/workitemrevisions?includeLatestOnly=true&api-version=5.0-preview.2"
$response = (Invoke-RestMethod -Uri $baseUrl -Method Get -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}).values

Please see Choosing the right authentication mechanism for more information about the Authentication.
Comment: He wants to know using Postman tool, not with C#. The tags does not have C#.


Windows Service Reading from MSMQ through WCF...Throttling

Question: I have a windows service that reads my message queue through WCF. I want the service to process not more than 50 messages from queue. I set the throttling configuration to 50 , but it does not seem to do anything. It just process one message after the other.
Am i missing something?
My web.config :
<serviceThrottling
        maxConcurrentCalls="1"
            maxConcurrentSessions="1"
            maxConcurrentInstances="50" />
My Service File:
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, InstanceContextMode = InstanceContextMode.PerCall)]
Please let me know if there is anything to be changed.
Answer: I don't think you should set maxConcurrentCalls and maxConcurrentSessions, can you try to remove them?


Load an XmlNodeList into an XmlDocument without looping?

Question: I originally asked this question on RefactorMyCode, but got no responses there...
Basically I'm just try to load an XmlNodeList into an XmlDocument and I was wondering if there's a more efficient method than looping.
Private Function GetPreviousMonthsXml(ByVal months As Integer, ByVal startDate As Date, ByVal xDoc As XmlDocument, ByVal path As String, ByVal nodeName As String) As XmlDocument
    '' build xpath string with list of months to return
    Dim xp As New StringBuilder("//")
    xp.Append(nodeName)
    xp.Append("[")
    For i As Integer = 0 To (months - 1)
      '' get year and month portion of date for datestring
      xp.Append("starts-with(@Id, '")
      xp.Append(startDate.AddMonths(-i).ToString("yyyy-MM"))
      If i < (months - 1) Then
        xp.Append("') or ")
      Else
        xp.Append("')]")
      End If
    Next

    '' *** This is the block that needs to be refactored ***
    '' import nodelist into an xmldocument
    Dim xnl As XmlNodeList = xDoc.SelectNodes(xp.ToString())
    Dim returnXDoc As New XmlDocument(xDoc.NameTable)
    returnXDoc = xDoc.Clone()
    Dim nodeParents As XmlNodeList = returnXDoc.SelectNodes(path)
    For Each nodeParent As XmlNode In nodeParents
      For Each nodeToDelete As XmlNode In nodeParent.SelectNodes(nodeName)
        nodeParent.RemoveChild(nodeToDelete)
      Next
    Next

    For Each node As XmlNode In xnl
      Dim newNode As XmlNode = returnXDoc.ImportNode(node, True)
      returnXDoc.DocumentElement.SelectSingleNode("//" & node.ParentNode.Name & "[@Id='" & newNode.Attributes("Id").Value.Split("-")(0) & "']").AppendChild(newNode)
    Next

    '' *** end ***
    Return returnXDoc
End Function

Answer: Dim returnXDoc As New XmlDocument(xDoc.NameTable)
returnXDoc = xDoc.Clone()

The first line here is redundant - you are creating an instance of an XmlDocument, then reassigning the variable:
Dim returnXDoc As XmlDocument = xDoc.Clone()

This does the same.
Seeing as you appear to be inserting each XmlNode from your node list into a different place in the new XmlDocument then I can't see how you could possibly do this any other way.
There may be faster XPath expressions you could write, for example pre-pending an XPath expression with "//" is almost always the slowest way to do something, especially if your XML is well structured. You haven't shown your XML so I couldn't really comment on this further however.


Why regularization does not always reduce weights? (Michael Nielson's book)

Question: I have been dealing with a problem in his online text book:

L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization. Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that: supposing $\lambda$ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where $n$ is the total number of weights in the network.

My first idea to approach this problem was to compare weights before-and-after stochastic gradients. So here is the result:
Before:

Each weight entry in $w^l$ is Gaussian distributed with mean 0 and var 1
$w^l$ means the sum of all the weights in $l^{th}$ layer from $(l-1)^{th}$ layer
$v^l$ means the variance of all the weights in $l^{th}$ layer from $(l-1)^{th}$ layer
$w^2$: 75.94,   $w^3$: 4.26,   $|w^2|_1$: 33.03,  $|w^3|_1$: 12.51, $v^2$: 1.01,    $v^3$: 1.15

After:

$w^2$: 130.23,  $w^3$: -72.15, $|w^2|_1$: 12.75,  $|w^3|_1$: 26.65, $v^2$: 0.08,    $v^3$: 3.62

Weights in 2nd layer have been decreased, although the amounts of the decreases were not enough to hit $\frac{1}{\sqrt(n)}$. However, weights in 3rd layer increased when they were supposed to be lower the original. I'm curious if regularization does reduce weights. And is there any specific reason why weights in different layer are affected in the opposite way? What am I missing here? And is there any other way to approach the problem? Any insight please?
My codes are below (they ran on Colab):
<code>#Download the dataset from Nielsen's github
!wget -L https://github.com/mnielsen/neural-networks-and-deep-learning/raw/master/data/mnist.pkl.gz

#Download mnist_loader.py from Dobrzanski's github: his version's for python 3
!wget -L https://raw.githubusercontent.com/MichalDanielDobrzanski/DeepLearningPython/master/mnist_loader.py

#Download network2.py from Dobrzanski's github: his version's for python 3
!wget -L https://raw.githubusercontent.com/MichalDanielDobrzanski/DeepLearningPython/master/network2.py
import mnist_loader
import network2
import numpy as np

training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
net = network2.Network([784, 30, 10],cost=network2.CrossEntropyCost)
net.large_weight_initializer()
b0 = net.weights[0]
b1 = net.weights[1]
print(f' W^2 : {b0.sum():.2f}\t\t W^3 : {b1.sum():.2f}')
print(f'|W^2|: {np.linalg.norm(b0, ord=1):.2f}\t\t|W^3|: {np.linalg.norm(b1, ord=1):.2f}')
print(f' V^2 : {np.var(b0):.2f}\t\t V^3 : {np.var(b1):.2f}')
net.SGD(training_data, 30, 10, 0.1, lmbda = 5.0, evaluation_data=validation_data, monitor_evaluation_accuracy=True)
c0 = net.weights[0]
c1 = net.weights[1]
print(f' W^2 : {c0.sum():.2f}\t\t W^3 : {c1.sum():.2f}')
print(f'|W^2|: {np.linalg.norm(c0, ord=1):.2f}\t\t|W^3|: {np.linalg.norm(c1, ord=1):.2f}')
print(f' V^2 : {np.var(c0):.2f}\t\t V^3 : {np.var(c1):.2f}')
</code>
Comment: Very closely related: https://stats.stackexchange.com/questions/206178.  Also see https://stats.stackexchange.com/a/452468/919 for an example where regularization first decreases the magnitude of a coefficient in a model and then increases it.
Comment: Thank you for your comment, Whuber. I will check them up once I go back home tonight.
Answer: When there are local optima, it's quite possible to end up in larger weights due to the randomness of the procedure and the heuristics used (e.g. SGD). But, considering the global optima in both cases, adding regularization decreases the weights' norms in which they are regularized for.
Let the total loss be decomposed as the following, where $L_e$ is the reconstruction error, like cross-entropy, or MSE/MAE; and $L_r$ be the regularization error, e.g. $L_2$ regularization (squared sum of the weights).
$$L=L_e + \lambda L_r$$
When there is no regularization, weights are chosen such that $L_e$ is minimized. Let the $L_r$ term be $A$ with these weights. So, the regularized loss would be $L_{e,min} + \lambda A$. Let the solution weights be $w^*$.
If we had used the regularized loss from the beginning, $L_e$ would be larger (or at least the same) since no other $w$ can achieve less magnitude. And, $L_r$ should be lower as well because a higher magnitude means a sub-optimal solution since $w^*$ has a lower absolute magnitude.
So, if you're able to reach global optima, it's certain that the regularization will decrease the collective magnitude of the weights. However, individual layers can be affected differently as what matters is the total.
Regarding the sum of weights w/o taking the absolute value, you can't guarantee anything because the sum could be close to zero but the weights can be enormously large. So, I would always check the weights' magnitudes in the form they're regularized, e.g. absolute value if $L_1$ regularization is applied, or squared sum if $L_2$ regularization is applied etc.
Regarding the $1/\sqrt n$ tail off cut-off, I haven't seen the statement before or checked the corresponding section in the book you mentioned. This answer merely tries to address your title question in a more general manner, and warns about the validity of some of the metrics you checked.
Comment: Thank you for your insight, Gunes. I have one thing unclear. The goal of the regularization of the network is to avoid overfitting. If $L_e$ increases and $\lambda L_r$ is a positive number, then $L$ would increase compared to the original $L$ without regularization. But we generally want to lower $L$ in order to achieve better accuracy of the model. So here, are we kind of sacrificing accuracy in order to avoid overfitting?
Comment: We aim better performance in test data, not the training. So, regularization is used for increasing the generalization performance, not the training.


CSS não aplica 100%

Question: Boa noite, estou utilizando CSS para estilização de um menu na vertical, porém quando tento aplicar 100% de altura ele não corresponde
código html

        Link 1

        Link 2

        Link 3

css
li:first-child {
    height: 150px;
    background:black;
    text-align: center
}

li a:hover {
    background:black;
}

a {
    color: white;
}

.menu-principal {
    height: 100%;
    width: 240px;
    background: red;
}

o menu em vermelho deveria pegar 100% da altura
obs: estou utilizando bootstrap tb
Answer: Você precisa também dizer, pra body e html que eles precisam de 100% de height
html, body {
    height: 100%;
}

Comment: Não funciona, brother

CSS não aplica 100%


Paperclip: Specify jpeg / jpg quality

Question: Is there an easy way to specify the jpeg quality of an image to be saved in the paperclip plug-in for rails 3?
Answer: Yes, it is pretty easy. Paperclip has an option called :convert_options which can be used for that purpose. It takes parameters that ImageMagick / Ghostscript can accept.
Here is an example:
class User < ActiveRecord::Base
  has_attached_file :avatar, :styles => { :thumb => ["100x100>", :jpg] },
                             :convert_options => { :thumb => "-quality 92" }

Answer: Building on @DanneManne's answer, if you want to convert the original image rather than add another style, simply define a style called :original and set convert_options for it like so:
has_attached_file :image,
  styles: { original:  {} },
  convert_options: { original:  "-quality 10" }

Paperclip: Specify jpeg / jpg quality


How can i tune autovacuum configs for pg toast tables?

Question: I have a pg_toast table which has a lot of dead tuples accumulating. The autovacuum job on it runs for a very long time.
I wanted to tune the autovacuum config, to make it more aggressive.
When i'm trying to set the config I get the following error:
ERROR:  permission denied: "pg_toast_16432" is a system catalog

Query used:
ALTER TABLE pg_toast.pg_toast_16432 SET (autovacuum_vacuum_scale_factor = 0);

Postgres version: 11
Can someone please suggest how to tune autoconfig on toast tables?
Answer: You cannot directly modify the autovacuum configuration for individual system catalog tables, however, you can consider using other performance optimization techniques like indexing and query optimization to reduce the need for aggressive autovacuuming.
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answer: You can set a storage parameter on the table itself, as described in the documentation. If the table is called tab, run
ALTER TABLE tab SET (toast.autovacuum_vacuum_cost_delay = 0);

Note that setting autovacuum_vacuum_scale_factor to 0 won't do what you want. To speed up autovacuum, increase autovacuum_vacuum_cost_limit or reduce autovacuum_vacuum_cost_delay.


Why $\mathbb{Z}[\sqrt{-5}]\cong\mathbb{Z}[X]/(X^2+5)$ and $\mathbb{C}\cong \mathbb{R}[X]/(X^2+1)$?

Question: This is a very elementary question.
In Matsumura's book "Commutative ring theory", I've found the following isomorphism:
$$\mathbb{Z}[\sqrt{-5}]\cong\mathbb{Z}[X]/(X^2+5).$$
As a quotient ring is involved, my idea is to use the first isomoprhism theorem, i.e. I want to define a homomorphism $\mathbb{Z}[X]\to\mathbb{Z}[\sqrt{-5}]$ whose kernel is exactly $(X^2+5)$, but here I get stuck.
Indeed,
$$\sum_{i=0}^n a_nX^n\mapsto a_0+a_1\sqrt{-5}$$
is a group homomorphism w.r.t. $+$, but this is not a ring homomorphism as
$$\Big(\sum_{i=0}^na_iX^i\Big)\Big(\sum_{i=0}^mb_iX^i\Big)=\sum_{i=0}^{n+m}(\sum_{k=0}^i a_kb_{i-k})X^i\mapsto a_0b_0+(a_1b_0+a_0b_1)\sqrt{-5}$$
which is different from
$$(a_0+a_1\sqrt{-5})(b_0+b_1\sqrt{-5})=(a_0b_0-5a_1b_1)+(a_1b_0+a_0b_1)\sqrt{-5}.$$
What's the right idea to get this isomorphism using the first isomorphism theorem?
PS: Similarly, I've seen that $\mathbb{C}\cong \mathbb{R}[X]/(X^2+1)$ and the various proofs define explicitly an isomorphism, but how can I obtain this result from the first isomorphism theorem?
Comment: Take the evaluation map $f: \Bbb Z[X] \longrightarrow \Bbb Z[\sqrt {-5}]$ defined by $x \rightarrow \sqrt {-5}$ and which is identity on $\Bbb Z.$ Then the $f$ is an onto ring homomorphism with kernel $(x^2+5).$ Then use first isomorphism theorem to conclude.
Answer: Try
$$\phi:\Bbb Z[x]\to\Bbb C\;,\;\;\phi p(x):=p(\sqrt{-5})$$
Observe that the image is $\;\Bbb Z[\sqrt{-5}]\;$ , and the kernel is...
For the second one, try
$$\psi:\Bbb R[x]\to\Bbb C\;,\;\;\psi p(x):=p(i)$$
Comment: @LBJFS It is much simpler: observe that $\;q\in\ker\phi\iff q(\sqrt{-5})=0\iff (x+\sqrt{-5})\,\mid q(x)\;$ . But now take into account that $\;q\;$ is a real polynomial and thus its complex roots appear as conjugate pairs...
Comment: I have some trouble when I try to compute the kernel... :/ assume I don't know that the kernel is $(X^2+5)$, is there a simple idea to get it?
Comment: I try to be more explicit: $0=\sum_{k=0}^n a_n(\sqrt{-5})^n=\sum_{k\,\text{even}}a_n(-5)^{\frac{n}{2}}+\sum_{k\,\text{odd}}a_n(-5)^{\frac{n-1}{2}}\sqrt{5}$ and I'm not able to go on.


How to use input argument as variable in Ansible Playbook

Question: I have tried looking at 'command' and 'shell' ansible modules, but haven't been able to use it successfully.
The script I am using is un-customizable and must use variable 'RESPONSE_FILE=' as a standard input.
My code is:
<code>- host: all
  gather_facts: no
  tasks:
  - name: Run Config Tool
    environment:
     response_file: /opt/config/cfg.properties
    shell: /apps/opt/configtool {{ response_file }}
</code>
Comment: If your script requires `RESPONSE_FILE` (upper case) why are you setting `response_file` (lower case)? Is your script looking for an environment variable, or a command line argument, or...?
Comment: I tried both upper/lower_cases, hasn't worked. Yes its looking for a VAR=
Answer: <code>- host: all
  gather_facts: no
  tasks:
  - name: Run Config Tool
    shell: /apps/opt/configtool response_file="/opt/config/cfg.properties"
</code>


sql query: multiple rows append to show in one row

Question: Following data
query:
Select EmployeeName from employee

Output
EmployeeName
------------
John
Micheal
Annie
Marry
Scott

I want the above out put into following:
EmployeeName
------------------------------
John Micheal Annie Marry Scott

can any one guide to how to query (in sql server 2008) to show multiple rows data into one row.
Comment: is it that hard to include your table schema?, and sql for which database specifically?
Comment: If you post code or XML, **please** highlight those lines in the text editor and click on the "code samples" button ( { } ) on the editor toolbar to nicely format and syntax highlight it!
Answer: In SQL Server 2008, you can use something like:
SELECT
   STUFF((SELECT ' ' + EmployeeName
          FROM dbo.Employee
          FOR XML PATH('')), 1, 1, '')

Answer: Well, if you are using mysql, you can use GROUP_CONCAT: http://dev.mysql.com/doc/refman/5.0/en/group-by-functions.html#function_group-concat, like this:
select GROUP_CONCAT(EmployeeName SEPARATOR ' ') FROM employee

For other databases, you could try googling the name of your database and GROUP_CONCAT, you'll probably find something.


Features differ between classes

Question: Good evening everyone.
Regarding the topic related to Sparse Clustering (for example K-Means).
For example, in "Witten DM, Tibshirani R. A framework for feature selection in clustering" the authors say
"In Figure 2, we apply this method to a simple example with 6 equally-sized classes, where n = 120, p = 2000, and 200 features differ between classes"
What does it mean that 200 features differ between classes? How does this translate to R / Python / Matlab?
REF:
Witten DM, Tibshirani R. A framework for feature selection in clustering. J Am Stat Assoc. 2010 Jun 1; 105 (490): 713-726. doi: 10.1198 / jasa.2010.tm09415. PMID: 20811510; PMCID: PMC2930825.
LINK TO REF:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2930825/
Comment: It means that the distributions of 200 of these features are different depending on the class they belong to.
Comment: Mh, one thing I didn't understand. According to your reasoning, for example:
F1 belongs to A with N (x, 1)
Does F2 belong to B with N (x1,1) with x different from x1?
But there are 6 classes.
Or
F1 belongs to A, B, C, D, E, F with N (x, 1)
Does F2 belong to A, B, C, D, E, F, with N (x1,1) with x different from x1?
..
F200 belongs to A, B, C, D, E, F with N (x200,1)

F201 belongs to A, B, C, D, E, F with N (x201,1)
..
F2000 belongs to A, B, C, D, E, F with N (x2000,1) with x201 = x202 = .... = x2000?

Thanks.
Comment: Sorry but I can't write it clearly and sorted.
Comment: Did you mean to write those in response to my answer?
Comment: I don't know how it works with comment replies, sorry.

However, I was referring to both.
Comment: I've edited to clarify.
Comment: Ok, thank you. In the case of the paper example, the first 200 features differ, for example, in the average.
But does each feature belong to a single class or does a feature belong to all classes but do the distributions between features change?

F1 -> A,B,C,D,E,F with N(x1,1)
F2 -> A,B,C,D,E,F with N(x2,1)
or F1 -> A, F2 -> B?

(My earlier comment may help you understand what I mean, but it's not neatly written)
Comment: Sorry, @dave.

In my example "n=120, p=2000, q=200", it means 120 obs per class (120*6), 2000 feature per class (2000*6)?
Comment: What are n, p, and q?
Comment: n=number of obs
p= number of feature
q = number of feature differ beetween classes.
Answer: It means that these features (jointly, not necessarily marginally) have some distribution for class A that differs from the distribution for class B.
It could be that the marginal means all differ between the classes.
It could be that the marginal variances differ between the classes.
It could be that one marginal mean is different and the difference is in variance for the rest of the features.
It could be that the marginal distributions are identical but the correlation between the features is different in class A than in class B (e.g., independent in class A but correlated in class B).
Let's give some examples, all of which I simulated in R.
First, we set up the simulation.
library(ggplot2)
library(MASS)
set.seed(2022)
N <- 1000

NOW let's simulate and plot three independent variables, two of which differ in means between the classes.
# Three variables, two of which differ in means between the classes
#
cA  <- rep("A", N)
cB  <- rep("B", N)
F1A <- rnorm(N, 0, 1) # N(0, 1) on F0 class A
F1B <- rnorm(N, 4, 1) # N(4, 1) on F0 class B
F2A <- rnorm(N, 0, 1) # N(0, 1) on F1 class A
F2B <- rnorm(N, 4, 1) # N(4, 1) on F1 class B
F3A <- rnorm(N, 0, 1) # N(0, 1) on F2 class A
F3B <- rnorm(N, 0, 1) # N(0, 1) on F2 class B
d <- data.frame(
  group = rep(c(cA, cB), 3),
  value = c(F1A, F1B, F2A, F2B, F3A, F3B),
  feature = rep(c("Feature_1", "Feature_2", "Feature_3"), rep(2*N, 3))
)
ggplot(d, aes(x = value, fill = group)) +
  geom_density(alpha = 0.5) +
  facet_grid(~feature) +
  theme_bw()

NEXT, let's simulate and plot three independent variables, one of which differs in means between classes and one of which differs in variances.

FINALLY, let's simulate and plot three variables, all with equal margins between the classes but with a differnt correlation structure on features 1 and 2 between the classes.
# Three variables, all with equal margins between the
# classes but with a differnt correlation structure
# on features 1 and 2 between the classes
#
cA  <- rep("A", N)
cB  <- rep("B", N)
F_A <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.9, 0.9, 1), 2, 2))
F_B <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, -0.9, -0.9, 1), 2, 2))
#
# We have positive correlation in A and negative correlation in B,
# but the margins all are N(0, 1)
#
F1A <- F_A[, 1]
F1B <- F_B[, 1]
F2A <- F_A[, 2]
F2B <- F_B[, 2]
F3A <- rnorm(N, 0, 1) # N(0, 1) on F2 class A
F3B <- rnorm(N, 0, 1) # N(0, 1) on F2 class B
d <- data.frame(
  group = rep(c(cA, cB), 3),
  value = c(F1A, F1B, F2A, F2B, F3A, F3B),
  feature = rep(c("Feature_1", "Feature_2", "Feature_3"), rep(2*N, 3))
)
ggplot(d, aes(x = value, fill = group)) +
  geom_density(alpha = 0.5) +
  facet_grid(~feature) +
  theme_bw()

THE marginal distributions of all three features are the same! However, plotting the values of feature $1$ and feature $2$ in a scatterplot reveals there to be a difference.

dxy <- data.frame(
  group = rep(c(cA, cB), 2),
  Feature_1 = c(F1A, F1B),
  Feature_2 = c(F2A, F2B)
)
ggplot(dxy, aes(x = Feature_1, y = Feature_2, col = group)) +
  geom_point() +
  theme_bw()

CLARIFICATION
In my answer, $X$ has three components (features), $x_1$, $x_2$, and $x_3$, as in writing a regression like $y=x_1+x_2+x_3$. In the first two cases, $x_1$ has a different distribution for class $A$ than for class $B$, and ditto for $x_2$. In the final case, the marginal distributions are the same in classes $A$ and $B$, but the joint distribution of $(x_1,x_2)$ has positive correlation for class $A$, while the joint distribution of $(x_1,x_2)$ has negative correlation for class $B$.
Comment: Ok thank you very much for your kind and thorough reply.

Much of the data described in the papers talks about features that differ in averages, so your first example you have summaries described.

So, I understand, in the example case of my question, a pseudocode could be:

N <- 120
# Six classes are created as in your example (cA, .., cF)
# 2000 features are created, of which the first 200 differ (for example for the average) for each class (1,200) such as F1A, F1B, F2A, f2B
#The rest (201,2000) are created as F3A, F3B

then I can apply the K-Means and Sparse K-Means algorithm.

Features differ between classes


Solve the diophanic equation $y^2 = x^4 -4x^3 + 6x^2 -2x +5$, where $x,y \in \mathbb{Z}$

Question: Solve the diophanic equation $y^2 = x^4 -4x^3 + 6x^2 -2x +5$.
Methods I know:
1) look modulo p for some prime p, when using this method I almost always conclude there are no solutions, so i don't think it is handy to use this in this particular case.
2) Combinations of factorization and estimation: factorize your specific function and Search for an upper bound.
I used 2), and i get in a little bit of trouble.
I tried factorize the right hand side, but did not come any further then:
$y^2 - 10 = (x-5)(x^3 + x^2 - x - 3)$.
I don't seem to get an upper bound here..... :(.
Any hints on going to the right direction or on using a new method trying to solve this probelem?
Kees
Comment: WA gives solutions, but no explanation ([link](http://www.wolframalpha.com/input/?i=solve+y^2+%3D+x^4+-4x^3+%2B+6x^2+-2x+%2B5+for+integer+x%2C+y)). I wonder if it just does a search for small numbers.
Answer: We may suppose that $y\geq 0$. Now
$$x^4-4x^3+6x^2-2x+5=(x-1)^4+2x+4$$
Hence we have
$$(y-(x-1)^2)(y+(x-1)^2)=2(x+2)$$
This imply that $(x-1)^2\leq (x-1)^2+y\leq 2|x|+4$
and it is easy to finish.
Comment: If $(x,y)$ is a solution, then $(x,-y)$ is also a solution, so we can suppose that $y\geq 0$ to bound $|x|$.
Comment: We have $x^2-2|x|+1\leq x^2-2x+1\leq 2|x|+4$, so $x^2-4|x|-3\leq 0$, etc
Comment: wow, nice factorization you got there. Why can we assume that $y \geq 0$ ?
Comment: hmmmmm, true that. Are we trying to find a bound of $|x|$ in the last step, because i don't see how |x| cant go to infinity if we have:

$|x| \leq - \frac{(x-1)^2 - 4}{2} $


SQL Aggregate records by CASE WHEN

Question: I am trying to aggregate the records' sum depending on the value that match the CASE WHEN
I am able to aggregate the sum but I don't know how I can group the records depending on the description value.
SQL Query:
SELECT
T1.Company, T1.DueDate,
SUM(T1.Amount) AS TotalAmount,

DATEDIFF( day, CONVERT(DATETIME, CAST(T1.DueDate AS VARCHAR(8)), 112), DATEADD( day, - 1, CONVERT(DATETIME, CAST(T3.FromDate AS VARCHAR(8)), 112) ) ) AS Age,
CASE
      WHEN
         DATEDIFF( day, CONVERT(DATETIME, CAST(T1.DueDate AS VARCHAR(8)), 112), DATEADD( day, - 1, CONVERT(DATETIME, CAST(T3.FromDate AS VARCHAR(8)), 112) ) ) <= 0
      THEN
         'Current'
      WHEN
         DATEDIFF( day, CONVERT(DATETIME, CAST(T1.DueDate AS VARCHAR(8)), 112), DATEADD( day, - 1, CONVERT(DATETIME, CAST(T3.FromDate AS VARCHAR(8)), 112) ) ) > 0
         AND DATEDIFF( day, CONVERT(DATETIME, CAST(T1.DueDate AS VARCHAR(8)), 112), DATEADD( day, - 1, CONVERT(DATETIME, CAST(T3.FromDate AS VARCHAR(8)), 112) ) ) <= 30
      THEN
        'Late'
END AS Description

FROM
    Table1 T1
    LEFT JOIN
      Table2 T2
      ON T1.Company = T2.Company
      AND T1.YR = T2.YR

    INNER JOIN
      Table3 T3
      ON T1.Company = T3.Company
      AND T3.YR = YEAR(CURRENT_DATE)

GROUP BY T1.Company,T1.DueDate, T3.FromDate

My query returns the following.
 Company    | DueDate        | TotalAmount | Age  | Description
------------+----------------+-------------+------+------------
 123        | 20200423       |    150      |   7  |   Late
 123        | 20200604       |    18000    |  -35 |   Current
 123        | 20200515       |    500      |  -15 |   Current

However, what I really want to accomplish is to aggregate records if they have same Description value.
So, the correct result should be :
  Company   | TotalAmount  |Description |
------------+--------------+------------+
 123        |     150      |  Late
 123        |     18500    |  Current

The 2nd and 3rd row falls under "Current". Therefore, it should combine the 2 rows and add 18000 + 500 = 18500. DueDate and Age column is just for reference. The most important fields are TotalAmount and Description
I tried grouping by the alias Description which is used in CASE-WHEN but I didn't work.
I'd appreciate any help.
Thanks!
Comment: This query must have run in MySQL as it should raise several errors! Please tag your DBMS.
Comment: @jordan, please tag your RDBMS and note its version (MySQL 8+ supports CTEs). And re-check your `GROUP BY` clause. Your [prior question](https://stackoverflow.com/q/61660164/1422451) indicates `mysql`.
Comment: @Parfait: it looks like SQL Server.. except for the use of `current_date`.
Comment: What's the point of the `left join` on `table2`, while it seems this table is not used anywhere in the query?
Comment: @GMB, I thought so too but SQL Server would raise an error of the *description*  and maybe *Age* (if not uniquely captured by *DueDate* and *FromDate*) for not being in `GROUP BY` clause. OP may have remove too much code in posting.
Comment: @GMB There are actually fields I'm using from table2. I just tried to make the query as simple as possible so I removed a few fields. I'll update the question.
Comment: @Parfait It doesn't seem to support CTE. I tried the suggestion from CodeDifferent but editor returned error. I'm doing this on DataLake query editor. I still haven't figured out the version.
Answer: You copy-and-paste the CASE statement into the GROUP BY clause:
GROUP BY T1.Company, CASE WHEN ...

which is a lot of code to repeat.
Or you can use a common table expression (CTE):
;WITH
    cte AS
    (
        SELECT
        T1.Company, T1.DueDate, T1.Amount

        DATEDIFF( day, CONVERT(DATETIME, CAST(T1.DueDate AS VARCHAR(8)), 112), DATEADD( day, - 1, CONVERT(DATETIME, CAST(T3.FromDate AS VARCHAR(8)), 112) ) ) AS Age,
        CASE
            WHEN
                DATEDIFF( day, CONVERT(DATETIME, CAST(T1.DueDate AS VARCHAR(8)), 112), DATEADD( day, - 1, CONVERT(DATETIME, CAST(T3.FromDate AS VARCHAR(8)), 112) ) ) <= 0
            THEN
                'Current'
            WHEN
                DATEDIFF( day, CONVERT(DATETIME, CAST(T1.DueDate AS VARCHAR(8)), 112), DATEADD( day, - 1, CONVERT(DATETIME, CAST(T3.FromDate AS VARCHAR(8)), 112) ) ) > 0
                AND DATEDIFF( day, CONVERT(DATETIME, CAST(T1.DueDate AS VARCHAR(8)), 112), DATEADD( day, - 1, CONVERT(DATETIME, CAST(T3.FromDate AS VARCHAR(8)), 112) ) ) <= 30
            THEN
                'Late'
        END AS Description

        FROM
            Table1 T1
            LEFT JOIN
            Table2 T2
            ON T1.Company = T2.Company
            AND T1.YR = T2.YR

            INNER JOIN
            Table3 T3
            ON T1.Company = T3.Company
            AND T3.YR = YEAR(CURRENT_DATE)
    )

SELECT      Company, Description, SUM(Amount) AS TotalAmount
FROM        cte
GROUP BY    Company, Description

Comment: thanks. I tried copying and pasting `case` statement to the `group by` but it only changed the order of records but didn't combine 2nd and 3rd row. I'm doing this for datalake and I think `cte` isn't supported.
Comment: Did you take `Due Date` and `Age` out of the `GROUP BY` clause?
Comment: It worked this time! It wants both `CASE` statements to be present in the `GROUP-BY` Otherwise, it won't work. Thank you very much! I can sleep now.
Comment: I will update the question details with answer tomorrow. I'm choosing your answer the best answer. I appreciate the help.

SQL Aggregate records by CASE WHEN


Getting associated trade agreement (PriceDiscAdmTrans) from a record in SalesLine table in dynamics ax

Question: How do I get the associated record in trade agreement journal (PriceDiscAdmTrans) from the record in SalesLine Table?
Answer: There is no associated PriceDiscAdmTrans record to a SalesLine.
There may be an associated PriceDiscTable record to a SalesLine, but the rules to find it is defined by workings of SalesPurchLine::setPriceAgreement and the PriceDisc class.
Answer: When a PriceDiscAdmTrans is posted in PriceDiscAdmCheckPost it will create a PriceDiscTable which a salesLine may use to determine the discount via the PriceDisc classes


Execute inline sql to update a table using nHibernate

Question: Sorry if this has been asked before, I did a search but couldn't find anything.
Is it possible to execute inline sql in nHibernate? I have something like this which I would like to run against the dB:
_session.CreateSQLQuery(
              @"update things
                set defaultThing = 0 where parentId = :parentId AND thingId <> :thingId")
                .SetInt32("parentId ", parent.Id)
                .SetInt32("thingId", thing.Id)
                ;

I suppose I could loop through a bunch of 'things' and set the defaultThing setting to false then call _session.Update(thing), but if I can do it how I outlined above, that would be great.
Comment: Don't know about using SQL for this, but using HQL it's possible.
Answer: Yes, just use ExecuteUpdate() on that query. It's the equivalent of IDbCommand.ExecuteNonQuery().
As kay mentioned, you can use HQL too. Check 12.3. DML-style operations
Comment: Good to know, but I would still use HQL in order to keep it DB neutral.
Comment: Of course I have read it. You wrote "you can use HQL". I would have recommended it, but nevermind :)
Comment: @kay.herzam yeah, I'd recommend it too :-)
Comment: hql and db neutral - that's awesome. diego and kay, thanks for the enlightenment, that's just what i needed!


Online resizing of kvm guest root filesystem?

Question: I have a Linux guest that uses an LVM volume directly as root file system (that is, there is no partition table). libvirt config looks thus:
<os>
  <type arch='x86_64' machine='rhel6.4.0'>hvm</type>
  <kernel>/boot/vmlinuz-X.Y.Z.el6.x86_64</kernel>
  <initrd>/boot/initramfs-X.Y.Z.el6.x86_64.img</initrd>
  <cmdline>console=ttyS0 root=/dev/vda</cmdline>
  <boot dev='hd'/>
</os>

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native'/>
  <source dev='/dev/vg/guest'/>
  <target dev='vda' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>

From inside the guest:
$ mount
/dev/vda on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Is it possible to resize the guest's root partition without rebooting the guest? Just doing lvextend on the host and resize2fs from the guest does not seem to be enough.
Answer: Yes, you need to use virsh blockresize to notify the guest of the change.
The syntax should be something like:
virsh blockresize INSTANCE_NAME --path /dev/vg/guest --size NEWSIZEG


Unable to Modify Files on Windows Share from OS X on Domain User Account

Question: A few of my end users use Macs. One of them is currently unable to save files to a our Windows file server running Windows Server 2003. The file permissions for his user account are setup properly, evidenced by the fact that when I login to a Windows box using his user account I can save files to the affected folder.
The Mac he is using is joined to our domain and he logs in using his domain account. The only thing I can think of is that somehow the account is accessing our file server using his local account credentials even though he is signed into the domain.
Does anybody have any idea what might be causing this problem and what I should do to resolve it?
Thanks in advance.
Answer: The way its setup at my work, even though I'm logged into the machine as my domain user, that information doesn't get passed to the Windows shares.  When you connect to the server, that server shows up under the "SHARED" option in Finder.
If you click on this shared server (which is NOT the mounted drive), you should get a little bar at the top of the mount listing which indicates which user you're connected as.  You may have to disconnect first, then use DOMAIN\username, plus your password, to gain access to the shares.  This is because, as you so gathered in your question, he's using local credentials as opposed to his domain account.
This is how it works on my OSX/SharePoint shares, so hopefully it will work for you!
Andrew
Comment: Good comment and I'll give you the check since you were the only one to reply. Turns out I hadn't investigated enough though and it was only Word 2011 that is causing this problem. I have found others out there with the same issue but no solution yet...thanks for your help!


Save me from changing the look of CKEditor using Javascript

Question: I see that code is being rendered to produce this CKEditor (via the CKEditor module) contains an inline style that creates a height of 200 pixels. I'd like to change this value to 100px to save a bit of screen space.

I would like to change this or override this is a maintainable way. In the hopes it was in a tpl.php I tried find . -name '*.php' -exec grep -lin 'presentation' {} \; with little luck.
Can anyone point me to where can override this value? Or better put, how can I do better than this:
jQuery(function() {
        console.log('Resizing ckeditor')
        jQuery('#cke_1_contents').attr('style', 'height: 100px; max-height: 100px;');
  });

Comment: I'm not sure what you mean by "value". You want to change the style of the ``  element?
Comment: You should explain what you want to achieve and perhaps specify whether you are using the Wysiwyg or CKEditor module (in case it matters).
Comment: Are you trying to resize the height of the ckeditor?
Comment: Yes, I'm trying to reduce the height of this element to 100px instead of 200px.
Comment: @othermachines, I'm using the CKEditor module
Answer: You can use a CSS override in your stylesheet to achieve this:
.cke_contents {
  max-height: 100px;
  height: 100px !important;
}

And, of course, if you only want to change one instance of the editor, just use an ID selector instead.
Comment: Would this take precedence over an inline style? I was under the assumption that inlines had the highest weight / priority? So far any changes that I made to the style sheets have not appeared to take any kind of effect -- even after drush cc all and clearing browser cache.
Comment: That's what !important is for.


C7 to Bb in "I'll See You in my Dreams"

Question: "I'll See You in my Dreams" verse to chorus seems awkward.
It's shown as C7 To Bb but seems to need to go through F which is not shown on lead sheets. I'm a relative novice and grappling with this.
Thanks
Comment: It's not clear what you're asking. Are you just wondering if the chord changes you have are correct? (Also, adding punctuation to your post will make it much easier to understand.)
Comment: Welcome! Right now this question isn't clear enough to answer. Can you use the "edit" button to show or link to the lead sheet you're talking about?
Answer: While the most common thing for a C7 chord to do is resolve as a dominant chord to a tonic F or Fm chord there are dozens of ways that dominant chords are used in music. Some are covered in functional diatonic harmony and some are not. Even though there are established functions for different types of chords there are no rules in music for what chord one should use or what chord should follow another chord.
In this particular case the C7 goes to Bb. It is not uncommon to go from a V chord to a IV chord. It happens in a large percentage of blues songs, not to mention the I IV V IV progression (think "La Bamba").
Granted the pull is strong to the I (F) chord in this case because the V chord ends a section but this was intentional on the composers part. He created a surprise, a deceptive or delayed resolution to the I since it does end up there 2 bars later.
Answer: Following up on @JohnBelzaguy's good answer: the pattern V7, IV7, I (or I7) is very common in the later bars of 12-bar and other blues. It is not a classical sound, but it does sound good. :)
In fact, a parallel descent, easier to notate in a particular key, say C, doing G7 Gb7(=f#7) F7 and then C7 is a good and standard blues-y sound.
Yes, it is possible to rationalize some of this in terms of secondary dominants, tritone substitutions, and such, but it is perhaps simpler for operational understanding to simply know the sound of these patterns, rather than looking for 18th-century-practice explanations. :)
(Believe me, I tried... :)
Answer: There are so many different versions chord-wise, of this song!
However, in key F (it's been performed in just about every conceivable key!) the verse ends on V (C) or V7 (C7), as is often the case when there's another verse following. The 1st beat of the verse is actually a IV chord (on the word see).
At the end of the verse, comes the chorus. As in maybe the majority (?) of songs, the chorus (middle 8) will start on IV. It does here, on B♭.
So, yes, one might expect an F7 or even Cm7 between the verse end and the chorus, but it doesn't happen. It is shown in one version, and there's no reason not to use it, though. It's just what the writers wanted - leading the listener, perhaps, to expect another verse, or just a whim.
I suspect this is a case where some theory is known which makes what they wrote wrong! There is no wrong in music!
Comment: An F7 would imply that the chorus starts in Bb.  It doesn't.  It starts on the subdominant of F major.
Comment: @LaurencePayne - I thought that's what I put. On, not in.
Answer: The first half of the verse establishes F major as tonic, ventures to D7, then 'cycle of 5ths' it's way back to F. All very standard.   Second half strays slightly further, the same 'cycle of 5ths' structure only gets as far as C7, setting us up nicely for a chorus back home in F major.  Still standard stuff.
And the chorus IS in F major.  There's no question of the B♭ and B♭m chords being tonicized, they're plainly IV and iv in the key of F.  It just doesn't start on the tonic chord.
Yes, you can do that.   'Sweet Georgia Brown' doesn't hit the tonic chord until bar 13 of the 16-bar chorus.

C7 to Bb in "I'll See You in my Dreams"


Combinatorics Urn Problem

Question: I have 208 unique marbles:

From this 208, I select 15 marbles without replacement and place them into an urn.
I do this again with a new urn, 3 more times, but using the original
208 marble set, so I apply replacement after each selection of 15.
I now have 4 urns of 15 marbles each, with no duplicate marbles within
a particular urn, but possibly duplicates across urns.

What is the probability that there are exactly 14 marbles that appear in more
than 1 urn $?$.
Comment: What have you tried?
Comment: An easier question is the expected number of marbles appearing in more than one urn, which seems to be about $5.883$, so the probability that there are exactly $14$ marbles that appear in more than $1$ urn is probably small, perhaps of the order of $0.02$ though I may be wrong
Comment: This doesn't address the original question, but can you clarify how you arrived at your answer?
Comment: Consider the probability that a particular ball appears in 0 or 1 urns.
Comment: The original question seems very difficult (although I may be missing some elegant approach). Where did you find this question?
Comment: @angryavain I realize it is difficult but this scenario happened in real life (though not with marbles) and was shocked and would like to know the probability...
Comment: Typically we care not about the probability of a particular outcome, but of a collection of outcomes that are "more extreme" than the one you see. For instance, the probability of flipping $n/2$ heads in $n$ tosses is approximately $\sqrt{\frac{\pi}{2n}}$ which might seem like a small number, but is actually the most likely outcome; looking at that number alone is not really going to give you a sense of how surprising it is. Similarly, in poker the probability that you get any particular hand is equal to the probability of getting a royal flush in hearts.
Comment: If you get $75$ coin flips in $100$ tosses, you usually look at the probability of "getting $\ge 75$ coin flips" (rather than "getting $=75$ coin flips") to get a sense of how surprising it is. In your situation you might reconsider asking for the probability of $\ge 14$ marbles appeared in more than one urn if you want a sense of how surprising getting $14$ marbles was. (Not sure if this is any easier to compute though.)
Comment: I appreciate the engagement but no one has attempted to solve the original question, which is clear
Comment: Each of those $14$ reappearing marbles may have multiple combinations - some of them may appear in $2$ urns while some in $3$, some in $4$ or a scenarios where all $14$ appear in two urns (and only $1$ in those two urns from the rest of the marbles). Very tedious.
Comment: Exploring the simpler problem of 2 urns, I got $$\frac{\binom{208}{14}(208-14)(208-15)}{\left[\binom{208}{15}\right]^2}.$$  From this, I agree with @MathLover that the math gets ugly fast.  I was tempted to apply https://brilliant.org/wiki/linearity-of-expectation/.  However, I'm not qualified to have an opinion on the validity of applying the principle here.
Answer: import numpy as np
from tqdm import tqdm
count = np.zeros(208)
rng = np.arange(208)
exactly14 = 0
morethan14 = 0
history = []
for i in range(4):
    choice = np.random.choice(rng, 15)
    history.append(choice)
    count[choice] += 1
for i in tqdm(range(100000)):
    choice = np.random.choice(rng, 15)
    count[choice] += 1
    history.append(choice)
    count[history[0]] -= 1
    history.pop(0)
    ccount = count[count > 1].sum()
    if ccount == 14:
        exactly14 += 1
    if ccount > 14:
        morethan14 += 1

print(exactly14/100000)

0.11
IMHO this problem suffers from combinatorial explosion thus it's easier to go with monte-carlo
Answer: Following the advice from Quester I simulated the problem with Monte Carlo
import numpy as np
from tqdm import tqdm
import matplotlib.pyplot as plt
count = np.zeros(208)
rng = np.arange(208).astype(int)
urns=np.zeros([4,15]).astype(int)
exactly14 = 0
morethan14 = 0
lessthan14 = 0
history = []

for i in tqdm(range(1000000)):
    for j in range(4):
        urns[j, :] = np.random.choice(rng, 15, replace=False)

    n_unique=len(np.unique(np.ravel(urns)))
    n_multi=len(np.ravel(urns))-n_unique
    history.append(n_multi)
    if n_multi == 14:
        exactly14 += 1
    if n_multi > 14:
        morethan14 += 1
    else:
        lessthan14+=1

print(exactly14/1000000)

0.000489

Combinatorics Urn Problem


Load Balancer sftp persistence when a server goes offline

Question: Let's say we have the following scenario:
We have two identical *nix servers using a shared filesystem. We connect through SFTP (not FTPS) to one of them to upload a file to the shared filesystem, the server goes offline and we get redirected to the second system which is still available.
My question is, would there be any connection persistence or the user will have to relogin? I guess a relogin would be needed because the ssh sessions are not shared between the two systems...
Thanks in advance :)
Comment: Based on my readings on the Internet, the user have to relogin, because the session are not shared, as what you already mentioned.
Answer: It depends on what you mean by "get redirected". If it's something like a DNS failover, then the client will know that the connection has been dropped so persistence is not possible.
It is possible to build an advanced load-balancing service which will terminate the SFTP connection and share state with another load-balancer so that any single component failure will be transparent to the client. However this could be very expensive if you went with products like the F5 BigIP.
In summary, it is possible but it would be expensive and add significant complexity.
Comment: Thanks for the reply. Can you provide me a little bit more information about your second statement? (building a load-balancing service etc)