Comparing Forecast Errors

Question: I want to compare my VAR-forecast to an ARMA-benchmark. The VAR forecast has a smaller MPSE (mean squared prediction error) and I want to check for significance. For this purpose I applied the Diebold-Mariano test with Stata but it gives always complete insignificance. Many Stata-users weren't sure about the automatically selected maxlag-option, me neither.
Is there any other test to compare MPSEs of unnested models?
Comment: What do you think about my answer? If it is helpful and clear, you may accept it by clicking on the tick mark to the left. Otherwise, you may ask for further clarification. This is [how Cross Validated works](https://stats.stackexchange.com/tour).
Answer: While you are asking for another test, the Diebold-Mariano test is really just what you need. You could even implement it yourself - it is so simple.

Regress the difference between the forecast losses (the loss series from forecast 1 minus the loss series from forecast 2) on a constant (i.e. an intercept).
Test whether the constant is equal to zero in population. In other words, test the significance of the intercept.
Use autocorrelation-robust standard errors in the test if you have a forecast horizon $h$ greater than 1; set the lag of the autocorrelation equal to $h-1$ when calculating the autocorrelation-robust standard error. For $h=1$, use vanilla standard errors.

If you reject the null of the constant being equal to zero, you conclude there is evidence against equal predictive ability of forecast 1 and forecast 2. If you cannot reject, then there is not enough evidence against that.

Comparing Forecast Errors


How to send a pipe with psexec?

Question: I'm trying to execute a pipe on a remote server by using psexec.
The command that I'm trying to execute is
psexec \\servername DSQUERY USER -name *userpart* | DSGET USER -samid -display

Currently, the | pipe symbol get executed locally which is not what is wanted (since dsget does not exist on my machine).
I tried to use quote around the whole command
psexec \\servername "DSQUERY USER -name *userpart* | DSGET USER -samid -display"

but that resulted in psexec trying to run that whole quoted sentence as one executable, which obviously does not exist.
How can I pass a pipe symbol to the remote server so that it can execute it?
Answer: Two possible suggestions come to mind: try to escape the pipe character with a caret (ie. ^|) if that doesn't work, you can create a batch file on the remote machine (including the piped command) and execute the batch file using psexec.
Comment: Using a pipe did not work with the error that DSGET is not an argument of DSQUERY. Of course creating a batch file will work, but I feel unsatisfied. Though I give you +1 for telling me how to escape a character in the command prompt.

How to send a pipe with psexec?


Ramsey Number Upper Bound

Question: Prove that if A= R(k,m-1) and B=R(k-1,m) are both even then R(k,m) $\le$ A+B-1
Would be great if someone help in solving this one....
Answer: A counterexample would have to be a graph of order $A+B-1$ in which each vertex is incident with $A-1$ red edges and $B-1$ blue edges. In other words, the red graph would have $A+B-1$ vertices, each of degree $A-1.$ Since $A+B-1$ and $A-1$ are both odd numbers, this is impossible by the "handshake lemma".

Ramsey Number Upper Bound


whats the triad formed root +maj 3rd +maj 2nd called?

Question: Extremely dissonant but is there an official name for it.
formula:
root +maj 3rd +maj 2nd
Comment: Could it be that you are just mistaken about an inversion? Can we get a picture of the example.
Answer: The most straightforward interpretation is as a simple triad, without any implied notes. In this case, the formula for the chord would need to be re-interpreted as:
root - major third - diminished third
which is enharmonically the same as the formula in your question. With root C, you'd have
C - E - Gb
I know that in German and Dutch, there is a specific name for this chord: hartvermindert (German), and hardverminderd (Dutch).
As far as I know there is no specific term for it in English.
The most common chord symbol for this chord is C(b5), because it simply is a major triad with a diminished fifth.
Note that more often than not, this chord voicing (interpreted enharmonically) is used as the upper structure of a different chord. E.g., that chord based on the note C could be used as an upper structure of an Am6 chord, of a D9 chord, or of an Ab7(#5) chord:

Am6: (A -) C - E - F#
     (R)   b3  5   6

D9: (D - A -) C - E - F#
    (R - 5)   b7  9   3

Ab7(#5): (Ab -) C - E - Gb
         (R)    3   #5  b7
Answer: It's a chord and chord symbol  that doesn't come up much in many styles but you could look at it as and add#11 chord with an implied 5th. So in C major it will contain the notes C, E, and F# with the G implied  and it would be called Cadd#11.
A much better way to look at this chord is to look at it in set notation. In this instead of using the typical naming scheme we just need it based the distance from root in semitones. So for this we can look at it as the set containig the notes 0, 4, and 6. Not really the prettiest thing especially when explaining to other musicians, but it best reflects the name without getting crazy.
Comment: so the chords that utilize that major 2nd implies chords larger than triads
Answer: Let's take a concrete example. In C major, your notes are C, E and F#.
You could consider this a part of an F#m7b5 chord. It's missing the A, which defines the minor quality of the chord, but I've seen it notated that way (in the progression Am, F#m7b5, Fmaj7, G).
This depends on the surrounding progression. If the three notes are in isolation, I'd use the add#11 from Dom's answer.
For interest's sake, I located an instance of a very similar chord in a song. It's notated as A(#4), which I suspect is rather unconventional:

It's built on a b7 root, so it's kind of a b7sus4 sort of thing. The notated piano part does not have a third, but the guitar chord diagram does:
 %3/.4/.0/0.0/0.0/0.3/[G(#4)]

The notes are G, C#, D, G, B, G, if I read it correctly.
Comment: You can't imply a 3rd in a chord. The only note you can imply is a perfect  5th from the root and calling it C diminished makes no sense as you have a major 3rd not a minor 3rd.
Comment: I guess I'm trying to say that those three notes can imply a m7b5. As in, that's how I've seen them notated in real chord charts. Whether that is correct depends on the function of the chord in the surrounding progression. Agreed that it's not a diminished chord.
Comment: I half agree with you. While you could use those notes to be apart of a m7b5 those notes alone don't make that chord and if I just wanted you to play those notes and you played a m7b5, it would not be what I wanted. Adding any tone besides the 5th alters the color of the chord so when naming a chord you want to look for names that won't introduce notes that will alter the overall sound.


What do I name this chord?

Question: When I'm warming up to play piano, I often play a little right-hand improv  with the left hand sounding out the following chord sequence, over and over, then ending in a final Am:
Am F G C F X E E7
Chord X is composed of the notes D E A and B.
So, the key is Am (I think). In that context, what would I name chord X?
Comment: The name of the chord depends on the function. You want to accept Tim's answer, but does your chord function like a dominant? How are you hearing it in relation to the rest of the series?
Comment: Is D in the bass?
Comment: @Richard On the left hand, I play the notes in that order, yes. On the right hand, I usually begin my phrases on the B.
Answer: D6sus2 will work. Or E7sus4. The latter is better in key Am. There is always an equivalent 'sus2' for 'sus4' on a different root.
Comment: I personally vote for the D6sus2, especially with D as the bass. +1!
Comment: @Richard - that would work, as the D6 is pretty close to Bm7, thus making the ubiquitous ii>V>I, with I=Am.
Answer: I agree with Tim's D6sus2 suggestion, but I thought I'd add to the entire progression:

Am F G C F X E E7

We begin in A minor; that's clear enough.
But after this, there's a brief move towards the relative major; the F G C is really just a IV-V-I in C major.
Then we end up moving back to A minor (if we ever left). The F chord is VI, and the E and E7 are just V moving to V7.

As such, it's best to understand the X as something that connects VI to V. Ideally, this would be a predominant of some kind, and one of the most common predominants is IV. Sure enough, the reading of D6sus2 is in fact a IV chord.
I like this better than reading it as an inversion of E7, and for two (and a half) reasons. First, it's relatively uncommon for a seventh chord to lose its chordal seventh, and then adding it back again is even more strange. Furthermore, with D in the bass of this X chord, our ears will tend to view that as the root.
The half a reason is a bit speculative: by reading it as a type of D chord, you have a IV-V at the end, which mirrors the IV-V-I you had in C towards the start of the progression. (You may think that's finding patterns where there really aren't any, and I wouldn't really disagree.)
Comment: How many songs have sequences similar - with, say, Dm instead of the first F chord? And maybe B half-dim in place of X? I Will Survive, several Garry Moore...
Comment: With a Bhd7, it's a straight circle-of-fifths progression from G to E; very common!
Comment: From Am, surely? Am Dm G C F Bo E Am. I'd call that circle of 4ths ascending, can't understand 5ths !!
Comment: Same difference; ascending fourths or descending fifths!

What do I name this chord?


Web.config transformation and search and replace

Question: I need to switch out an IP address in multiple WCF services in web.config. With web.config transformation, is there any way, besides specifying each an every address by xpath, to create a search and replace statement. E.g. switch out IP address 184.108.40.206 with 22.214.171.124 for all instances of 18.104.22.168
Comment: This must somehow be possible... anyone?
Answer: Say your Web.config is something like this (a simplified scenario, but // in XPath works everywhere):
<configuration>
    <endpoint address="184.108.40.206" />
    <endpoint address="126.96.36.199" />
    <endpoint address="18.104.22.168" />
    <endpoint address="126.96.36.199" />
</configuration>

then you will need something like this:
<?xml version="1.0"?>

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">

    <replaceAll>
        <endpontAddresses xdt:Locator="XPath(//endpoint[@address='184.108.40.206'])" xdt:Transform="SetAttributes(address)" address="220.127.116.11" />
    </replaceAll>

</configuration>

NOTE: this XPath will look for every  element in the whole Web.config and check whether given  element has address attribute with value equal to "18.104.22.168".
If you need something more general then try this:
<?xml version="1.0"?>

    <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">

        <replaceAll>
            <endpontAddresses xdt:Locator="XPath(//*[@address='188.8.131.52'])" xdt:Transform="SetAttributes(address)" address="184.108.40.206" />
        </replaceAll>

    </configuration>

This will look at every XML element (due to the asterisk: *) and check whether it has address attribute with value equal to "220.127.116.11".
So this will work for a file like this:
<configuration>
    <endpoint name="serviceA" address="220.127.116.11" />
    <endpoint name="serviceB" address="188.8.131.52" />
    <endpoint name="serviceC" address="188.8.131.52" />
    <endpoint2 address="184.108.40.206" />
    <endpoint3 address="184.108.40.206" />
    <endpoint4 address="188.8.131.52" />

    <innerSection>
    <endpoint address="220.127.116.11" />
    <anotherEndpoint address="18.104.22.168" />
    <sampleXmlElement address="220.127.116.11" />
    </innerSection>
</configuration>

Now if you want to limit substitutions to a certain section i.e. <system.serviceModel> then you can use an XPath like this:
    <endpontAddresses xdt:Locator="XPath(/configuration/system.serviceModel//*[@address='220.127.116.11'])" xdt:Transform="SetAttributes(address)" address="188.8.131.52" />

This will update addresses only in <system.serviceModel> section
<configuration>
    <endpoint name="serviceA" address="220.127.116.11" />
    <endpoint name="serviceB" address="18.104.22.168" />
    <endpoint name="serviceC" address="22.214.171.124" />
    <endpoint2 address="126.96.36.199" />
    <endpoint3 address="184.108.40.206" />
    <endpoint4 address="22.214.171.124" />

    <innerSection>
    <endpoint address="220.127.116.11" />
    <anotherEndpoint address="220.127.116.11" />
    <sampleXmlElement address="184.108.40.206" />
    </innerSection>

    <system.serviceModel>
    <endpoint name="serviceB" address="220.127.116.11" />
    <endpoint name="serviceC" address="220.127.116.11" />
    <endpoint2 address="188.8.131.52" />
    <innerSection>
      <endpoint address="126.96.36.199" />
      <anotherEndpoint address="184.108.40.206" />
      <sampleXmlElement address="22.214.171.124" />
      </innerSection>
    </system.serviceModel>

</configuration>

Give those a try and choose the one that suits your needs most.
NOTE: This has a limitation that you need to specify what is the name of the attribute that contains the IP (22.214.171.124) but I think its better to be explicit rather than have magic happen here. If you have many attibute names just repeat the
Comment: Optionally you can use another MSBuild task to do search and replace in a file. A task that does this is File task from MSBuildExtensionPack (available at codeplex.com)
Comment: how do you replace values inbetween "<><>" instead of attributes like this?


Need to disable the dropdown once the option is selected

Question: Once the option in the dropdown is selected, that cell needs to be disable and no further changes it should permit.
Basically need to remove the edit rights for that particular cell once the option is selected or anything is entered in that dropdown cell(while dragging and filling the dropdown option).
SS Link: https://docs.google.com/spreadsheets/d/127P05KL1tzGru8PMXFLM57y20m1qOVnRS8IAfptsIh4/edit?usp=sharing
Since I am new to google script, I am sure it is possible but don't know the way out.
Answer: I believe your goal is as follows.

You want to protect the cell when the dropdown list of column "A" is changed.

In this case, how about the following sample script?
Sample script:
Please copy and paste the following script to the script editor of Spreadsheet and save the script. And, in order to run the script when the dropdown list of column "A" is changed, please install OnEdit trigger to the function installedOnEdit.
When you want to use this script, please change the dropdown list of the column "A". By this, the script is automatically run by the installable OnEdit trigger.
function installedOnEdit(e) {
  const { range } = e;
  const sheet = range.getSheet();
  const dataValidation = range.getDataValidation();
  if (sheet.getSheetName() != "Sheet1" || range.columnStart != 1 || !dataValidation || dataValidation.getCriteriaType() != SpreadsheetApp.DataValidationCriteria.VALUE_IN_LIST) return;
  var protect = range.protect();
  protect.addEditor(Session.getEffectiveUser());
  protect.removeEditors(protect.getEditors());
  if (protect.canDomainEdit()) {
    protect.setDomainEdit(false);
  }
}

When this script is run, the cell is protected. In this case, only the owner can edit the cell.

Note:

This script was tested using your provided Spreadsheet. When you change Spreadsheet, this script might not be able to be used. Please be careful about this.

References:

Installable Triggers
protect()


Getting the incorrect residue for this function?

Question: I am trying to compute
$$Res_{z=i}\frac{z^{2}}{(z+i)^{2}(z-i)^{2}}$$
I made
$$\phi(z)=\frac{z^{2}}{(z+i)^{2}}$$
Then
$$Res_{z=i}\frac{z^{2}}{(z+i)^{2}(z-i)^{2}}=\frac{\phi^{(2-1)}(i)}{(2-1)!}={\phi^{(1)}(i)}$$
Since $z_{0} = i$ is a pole of order 2.
Next I tried computing the derivative of $\phi(z)$ using the quotient rule.
$$\phi^{1}(z)=\frac{2z(z+i)^{2}-2(z+i)(z^2)}{(z+i)^{2}}$$
Now when I plug in $z=i$ here is what I get
$$\phi^{1}(i)=\frac{2i(2i)^{2}-2(2i)(i^2)}{(2i)^{2}}=\frac{-8i+4i}{-4}=i$$
The problem is that this is the incorrect residue. The answer is supposed to be $-\frac{i}{4}$. I'm not sure where I messed up.
Answer: The derivative was taken incorrectly.  Note that
$$\frac{d}{dz}\left(\frac{z^2}{(z+i)^2}\right)=\frac{2z(z+i)^2-2z^2(z+i)}{(z+i)^\color{blue}4}$$
Comment: Namely, OP, you forgot to square the denominator.


How do I update a module schema?

Question: I have already created a module (test_auth), which I installed on a Drupal site.
I added test_auth_schema() & test_auth_update_8101() to test_auth.install. When I run drupal cache:rebuild, it doesn't create my table in the database.
I put echo 1; exit; in both functions, but they don't seem invoked.
What am I missing?
Comment: Run the update.php script.
Comment: Hi @Kstack via URL? project url/update.php ?
Comment: yes. https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Extension%21module.api.php/function/hook_update_N/8.2.x
Comment: Use drush updb or run update.php
Answer: Simply clearing the cache doesn't make Drupal do anything on updated modules. hook_update_N() implementations are invoked from update.php, similarly to what happens with previous Drupal versions. This means, for example, that once Drupal runs mymodule_update_8001(), it will not run the same update hook again; for new updates, mymodule_update_8002() is required.
Differently from hook_update_N(), hook_schema() is not invoked when a module is updated, but when a module is installed. So, it is expected that test_auth_schema() is not invoked, when the module is not installed.
The alternative to run update.php is using Drush (i.e. drush updatedb).

How do I update a module schema?


Is it true that finite-dimensional vector spaces can have infinite sequences as elements?

Question: I recently encountered the following definition of a finite-dimensional vector space in Axler's Linear Algebra Done Right: A vector space is called finite-dimensional if some list of vectors in it spans the space.
This I was then thinking about $V = \{(x, 0, 0, \cdots): x \in \mathbb{R} \}$ i.e., the set of all infinite sequences whose first value is some real number and for which all other values are 0.
The book defines ${\mathbb{R}}^{\infty} = \{(x_1, x_2, \cdots): x_j \in \mathbb{R} \text{ for } j = 1, 2, \cdots \}.$ He defines addition and scalar multiplication with ${\mathbb{R}}^{\infty}$ as you'd expect:
$(x_1, x_2, \cdots) + (y_1, y_2, \cdots) = (x_1+y_1, x_2+y_2, \cdots)$ and $\lambda(x_1, x_2, \cdots) = (\lambda x_1, \lambda x_2, \cdots)$.
Clearly $V$ is a subspace of $\mathbb{R}^{\infty}$. Now since any $v = (x, 0, 0 \cdots) \in V$ can be written $v = x(1, 0, 0, \cdots)$ where $x \in \mathbb{R}$ and $(1, 0, 0, \cdots) \in V$, $(1, 0, 0, \cdots)$ alone spans $V$. Since this list of just one vector in $V$ spans $V$, I am inclined to think that $V$ is finite-dimensional.
This does not mesh with any pre-existing intuition I had about what finite-dimensional vector spaces were, nor can I find any examples like this of a finite-dimensional vector space online, which leads me to think there is an error in my reasoning. Is this correct? If not, can someone point me to the error in my reasoning?
Comment: $\{(x,y,z) : x+y+z=0\}$ is two dimensional, but it consists of 3-tuples of numbers. Does that bother you?
Comment: Yes ,$V$ is finite-dimensional. Why does this bother you ?
Comment: Just to make sure: A finite-dimensional space is spanned by a *finite* list of vectors. But of course a finite-dimensional space can have infinitely many elements.
Comment: Am I right that this bothers you because it's hard to imagine this subspace geometrically as living inside of an infinite dimensional space? This failure of the "arrows with magnitude and direction" picture is exactly why we find the abstract notion of vector space useful. It keeps the features of "arrows with magnitude and direction" that are useful without being limited by our geometric imagination in terms of what spaces we're able to consider.
Comment: Here's another perspective: a line is one-dimensional. It is still one-dimensional if it's in a plane. Or in 3D space. And this extends to any dimensional space.
Comment: The elements of a finite-dimensional vector space can be infinite objects, of arbitrarily large size. Consider the $0$-dimensional real vector space whose only member is $(0,0,0, …)$.
Comment: @Charles Hudgins yes -- I was having trouble geometrically conceptualizing this subspace (and perhaps being unwilling to let go of the desire for an arrows with magnitude and direction image)
Answer: Vector spaces can be thought of abstractly as containing objects that can be added together and also can be multiplied by scalars and still stay within $V$ -- it doesn't matter what the vectors actually are, at least from a theoretical viewpoint. In applications, we care very much about what they are because we are not trying to prove theorems but usually get a concrete answer.
There are many examples of vectors that look nothing like lists of numbers at all. The one that helped seal this abstract viewpoint for me was function spaces (spaces where each element is a function!):

Here's how this works using a simple example of the vector space of polynomials with degree $\leq 2$ over $\mathbb{R}$ with real coefficients: $V:=\mathcal{P}_2(\mathbb{R})[\mathbb{R}]$ (i.e, each element $v \in V$ is a function $f:\mathbb{R} \to \mathbb{R}$ of the form $ax^2 + bx + c \textrm{ with } a,b,c\in\mathbb{R}$).
How big is each element?
Each element is a continuous function $f$ over the real numbers, so technically it represents an uncountably infinite number of pairs
$f = \{(x,f(x))\quad x \in \mathbb{R} \}$
What is the dimension of $V$?
$\dim V = 3$ because the vectors $\{1\}, \{x\}, \{x^2\},$ span $V$
So, as you can see, the nature of the vectors in a vector space has nothing to do with the dimension of the vector space. The dimension is the smallest number of vectors $v \in V$ needed to span $V$.
Note also that $V$ is a finite dimensional subspace of the uncountably infinite dimensional space of continuous functions over the reals: $C(\mathbb{R})$.
So you can have a finite-dimensional vector space as a subspace of an infinite dimensional vector space which itself may contain objects that represent infinite lists. The two concepts are separate.

You can see this with simpler objects too:
Let $\mathbf{b} = [1,1,1,1,1,1]$ be a basis for $V(\mathbb{R})$, then $\dim V = 1$ even though $\bf b \in \mathbb{R^6}$
What is this subspace of $\mathbb{R^6}$? Its the line $L = a\mathbf{b}\; a \in \mathbb{R}$ which passes through the origin and is parallel to $\bf b$.
Comment: The example with function spaces really does help; I hadn't really the fact that when the elements are continuous functions, each is technically representing an uncountably infinite number of points.
Answer: I think people above have given great answers. I want to add one comment that might help you think about it.
You can think of the dimension as the "complexity" of the space. In your example, the space is very "simple" in the sense that to represent any point in the space, you only need one coordinate (in particular, you only need the value of the first entry, since every other entry is 0). In this sense, the "complexity" or "dimension" is 1, since you only need one value to represent each point. I'm sort of abusing the word complexity here (we don't usually use this word to talk about dimension), but hopefully it gives some intuition as to what the dimension is representing).
Answer: Perhaps you're confused about the number of entries of an array/sequence, versus the dimension of a vector space. For example, consider the following two arrays
\begin{align}
v = (1, 1, 1),\\
w = (1, -1, 1).
\end{align}
Together, they span a 2 dimensional vector space $V_{vw}$, where every element is of the form $(a, b, a)$. Since $v$ and $w$ both have 3 entries, we can also think of them as vectors in the 3 dimensional vector space $\mathbb{R}^3$. Attached plots of $V_{vw}$ in $\mathbb{R}^3$ below to help you understand it.

Your example is just another variant of this: a 1 dimensional subspace in an infinite dimensional vector space.
Answer: As others have already mentioned, yes this vector space is finite dimensional, specifically 1-dimensional, if we're talking about vector spaces over $\mathbb{R}$. You already showed it pretty well:
Each $v\in V$ has a unique representation as $x\cdot (1,0,0,...)$ for one and only one $x\in\mathbb{R}$. So $\{(1,0,0,...)\}$ is a base of $V$.
I think your confusing cardinality and the number of dimensions a little bit. There are lots of finite-dimensional Vector Spaces with infinitely many elements. In fact, most vector spaces you would "commonly" think about, like $\mathbb{R}^2$ or $\mathbb{R}^3$ also have infinitely many elements.


Unexpected multiplication results with integer values read by scanf

Question: I am a beginner at C and I am trying to write a vector multiplication code. I read in an array and the scale. Then I multiply this scale with each element in the array.
for (i = 0 ; i < 5 ; i++)
{
    scanf("%d", &numbers[i]);
}

puts("Please enter the scale:");
scanf("%d", s);

puts("The scaled vector is:");

for (j = 0 ; j < 5 ; j++)
{
    int r = numbers[j] * s  ;
    printf("%d\n", r);
}

However, when I run this code I receive unexpected values with the following inputs:
1
2
3
4
5

Scale:
2

Output:
6130616
12261232
18391848
24522464
30653080

When I replace the s in here numbers[j] * s by 2, for example, it will return the expected output.
Answer: Send a pointer to scanf, so that you can get the value:
puts ("Please enter the scale:");
scanf ("%d" , &s);

Comment: Probably it's a typo since he did it right a few lines above. Nice spot by the way, I didn't notice it ;)
Comment: +1, and I'll bet his compiler agreed (*uninitialized local variable 's' used* or the like).


view of unformatted list invalidating cache issues

Question: I have a view that shows the title of a single content type as an unformatted list. I have an argument for a nid-contextual filter that I pass to the view in
function modulename_views_pre_view($view, $display_id, array &$args) {
 ....
 $args[0] = $filter_value;
 }

The value for the contextual filter is a value the user can select and is set in \Drupal::service('user.private_tempstore'). Every time the user selects a different value from a select form element, the view should show different results. However, it caches the output and only if I clear all caches it shows the correct output. But if I format the view output as a table, then everything works totally fine and the new results are shown immediately.
The cache tags for this view are given as follows:
print_r($view->getCacheTags());
Array ( [0] => config:views.view.users_in_class [1] => node_list )

Do I have to specifically invalidate something here or am I missing something else ? I thought that the cache tag node_list would already be enough. And why are results being cached when having the format as an unformatted list?
Answer: The cache tag node_list only works for database changes in the node table, but this is not happening when the user selects a filter value.
This is from the point of the View a cache context. So if you don't find a way to configure the View to handle this correctly, you could assign a custom cache context for the filter value.
The whole picture, which the View can't know because you provide only the filter value to it, is a mix of tag and context. You would need a cache tag for the private tempstore and a context for the current user to get the caching right:
function modulename_views_pre_view($view, $display_id, array &$args) {
  ...
  $uid = \Drupal::currentUser()->id();
  $view->element['#cache']['tags'][] = 'users_in_class.' . $uid;
  $view->element['#cache']['contexts'][] = 'user';
}

and then invalidate the tag when changing the private tempstore of a user $uid:
Cache::invalidateTags(['users_in_class.' . $uid']);

Comment: Thanks, that makes sense. I got it working as you described it. Just so I fully understand this: when you add the new cache tag `'users_in_class' . $uid` how does this specifically point to the `$tempstore` object in the `$_SESSION`. Does it invalidate the cache as soon as any of the values in the user's `$_SESSION` array changes ?
Comment: no, this is a custom cache tag and you have to invalidate the tag at the same time when you modify the value in the private tempstore, see the last code example in the answer


JS native format date method

Question: I've got trivial question for which I can't find simple answer - how can I format Date object in order to get it in format I need? For example, "20.01.2014". I know about moment.js library but I must do it using only native JS. Please, give me answer. Thanks!
Comment: Maybe [this](http://stackoverflow.com/questions/3552461/how-to-format-a-javascript-date) will help?
Answer: You use the getHours, getMinutes, and getSeconds (or getUTCHours, getUTCMinutes, and getUTCSeconds) methods on your Date instance, which give you numbers, and then do the formatting with string manipulation.

JS native format date method


How to extract free induction decay parameters from a plot

Question: I'm new to Machine Learning. I have a question.
Given a Free Induction Decay graph, similar to the one shown below, how can I use Machine Learning to extract features (i.e., $f$, $f_0$, $V_{pp}$) from the graph? Should I use supervised or unsupervised learning? Is this a Classification or Regression problem? Can I use the Sci-Kit Learn Python Library or should I use MATLAB?
If anyone can help, that would be great.
Comment: nonlinear least squares
Comment: This is an interesting type of problem because there are numerous plots in the literature that do not have enough explicit information, but may sometimes be extractable. That is important for reproducible science.
Comment: There are not any units on your plot to calibrate against. It is not clear to me that there exists a unique parameter that would best-agree with the plot because of that.
Comment: @Galen Well, this is really going backwards. The plot is generated from the features above. But can you extract the features from the plot? That's the question.
Comment: I suspect you cannot in this instance because you do not have any reference of how much time we are looking at on the x-axis, nor the size of the signal on the y-axis. The shape of the curve alone is probably not enough.
Comment: @Galen As for the units, the y-axis has no formal units (i.e., arbitrary units). It's just a measure of the contrast given by the signal or signal intensity $S(t)$. On the other hand, the x-axis is typically measured in microseconds $\mu s$.
Comment: Is the task to input an image to a learning algorithm and extract the numerical data from the image itself? Like, in simpler terms, given a picture of a parabola, retrieve the coefficients? It seems like you'd have to do this in 2 stages: (1) collect the numerical/tabular data from the image (2) interpret the data as some parameterized function. We have a question about (1) here: https://stats.stackexchange.com/questions/14437/software-needed-to-scrape-data-from-graph as for (2), it seems to be ordinary NLS.
Comment: @Sycorax. Please may you clarify what you refer to when you mention NLS? I am unable to parse the acronym currently.
Comment: @Refath Ok, in which case you could use units such as pixel length in the first step that Sycorax describes. In order for the estimated parameters to be physically meaningful, I think you would need to determine the ratios of signal/pixel and/or time/pixel.
Comment: @Galen OK, I'll look into it.
Comment: @Galen Thanks everyone. I looked into it and got some interesting results using SciPy's `curve_fit()`. Now I have a different question: **what type of Machine Learning model should I use to _classify_ different types of graphs** (i.e., sinouisdal, exponential, etc.)?
Comment: @Refath Congratulations on working out a solution. Please feel free to post what worked as an answer below.
Comment: @Refath One question per post please.
Comment: Hi @Galen, thanks for the suggestion. I opened a new question. I'm also adding my answer to this question.
Answer: Here are the 3 steps I used to extract parameters from an FID. Note that this is only what I could come up with in a short amount of time, so others may have better solutions.
Notes:

Every once in a while, SciPy seems to fall in a "false minimum" problem, where it believes it has found the curve of best fit, but really hasn't. I'm not sure why this happens or how to fix it.
Yes, I am extracting only a finite amount of points from the FID, in this case from the domain (-9,10). Of course, this can be expanded.

Step 1: Create a Random Damped Oscillator
A damped oscillator is analogous to an FID. The inputs = [], outputs = [] arrays serve to store discrete values from the graph. These arrays will later be converted into a pandas pd.DataFrame, which will be the input for the regression network. Use numpy.rand() to generate random values for A, w, T2. Then use a for loop to add values to the inputs = [], outputs = [] arrays.
inputs = [], outputs = []

y = A*np.cos(w*dom)*(2.718**(-dom/T2))

def expDec(t, A, w, T2):
    return A*np.cos(w*t)*(2.718**(-t/T2))

for i in range(-9, 10):
    inputs.append(i)
    outputs.append(expDec(i, A, w, T2))

Step 2: Create DataFrame
Create a pandas pd.DataFrame. This seemed the best way to "hand-over" the graph's values.
points = {'Input': inputs, 'Output': outputs}
x = pd.DataFrame(points, columns = ['Input', 'Output'])

Step 3: Best-Fit Curve
There's a few parts to this. First, explicitly inform the program what type of graph you want to be extracting the parameters from. In our case, it's an FID, analogous to a Damped Oscillator. Then, retreive the values from the graph and store them in the ins, outs variables. Pass these variables onto SciPy's curve_fit function.  This function will return what SciPy believes are the best parameters that fit the FID. Now pass on these parameters to realFunc(), which will append all the output values in the domain (-9,10) to the fit=[] array.
def realFunc(t, A, w, T2):
    return A*np.cos(w*t)*(2.718**(-t/T2))

ins = x['Input'].values
outs = x['Output'].values
fit = []

constants = curve_fit(realFunc, ins, outs, maxfev=1000)

for i in range(-9,10):
    fit.append(realFunc(i, A_fit, w_fit, T2_fit))

plt.plot(x['Input'], x['Output'])
plt.plot(x['Input'], fit,"ro-")

And that's it! All of these steps can be followed at the Interactive Jupyter Notebook found here.


Autenticação Django com Base de Dados Legado

Question: Estou desenvolvendo um projeto com um banco de dados legado (Firebird), e consegui fazer a configuração do settings.py e está funcionando com o banco de dados.
Na hora de criar um login usando forms, não estou conseguindo criar uma autenticação com o banco, pois não estou usando a tabela padrão de usuário do django, e estou tentando validar com uma tabela já existente do sistema legado.
Alguem poderia me orientar de como seria feito a criação de um login utilizando uma tabela já existente do sistema legado? E principalmente a parte da views.py para autenticar.
Answer: Você precisa criar um authentication backend customizado, veja https://docs.djangoproject.com/en/dev/topics/auth/customizing/
Você pode copiar e colar o modelo do django, e especificar no seu modelo o nome da tabela e de cada coluna
Na verdade pode não ser tão simples assim, eu não sei como as senhas são armazenadas no sistema legado, na verdade você nem precisa criar uma tabela, você pode criar a sua própria logica de autenticação, como por exemplo autenticar em um sistema remoto como AD/LDAP


Problema com namespace e include

Question: Bom dia. Tenho um webservice feito em slimPHP . Tenho uma classe que se chama control, e ele pertence ao namespace controllers, ela funciona normalmente sem problemas , mas surgiu a necessidade de importar um classe para manipulação de PDF , so que quando dou o include ele da erro , dizendo que não encontrou a classe solicitada. Alguém sabe o que pode ser ? Segue a classe.
<?php

 namespace controllers {

  include "fpdf/fpdf.php";

   class Control {

    //Atributo para banco de dados
    private $PDO;

    /*
      __construct
      Conectando ao banco de dados
     */

      function __construct() {
        $this->PDO = new \PDO('mysql:host=localhost;dbname=infoged', 'root',
              '*******'); //Conexão
                  $this->PDO->setAttribute(\PDO::ATTR_ERRMODE, \PDO::ERRMODE_EXCEPTION); //habilitando erros do PDO
    }

    public function PDF($romaneio) {
       $pdf =  new \fpdf\FPDF("P", "pt", "A4");
       $pdf->AddPage();
    }
  }
 }

Erro:
Fatal error:  Class 'fpdf\FPDF' not found in
C:\xampp\htdocs\infoged\services\controllers\Control.php on line
142

Inicio do arquivo FPDF

 define('FPDF_VERSION', '1.81');

class FPDF {

    protected $page;               // current page number
    protected $n;                  // current object number
    protected $offsets;            // array of object offsets
    protected $buffer;             // buffer holding in-memory PDF
    protected $pages;
 .....
 }

Comment: Acredito que se você postar o erro que da, irá ajudar bastante jovem.
Comment: Beleza , postando!
Comment: É bem provável que o erro seja porquê você efetuou o `include` dentro do `namespace`, incluindo as classes FPDF no mesmo, enquanto você tenta acessar a classe no `namespace \fpdf`. Tente dar o `include` antes do `namespace`.
Comment: Quando dou o include antes do namespace ele me retorna esse erro.```Fatal error:  Namespace declaration statement has to be the very first statement in the script in```
Comment: O que possui dentro do arquivo fpdf.php? Poderia postar apenas a inicialização da classe neste arquivo por favor?
Comment: Então tira esses `{}` do `namespace`.
Comment: Bruno postei. Anderson vou fazer o teste sem as chaves .
Comment: Quando coloquei sem as chaves ele me retornou outro erro. Embora eu ainda ache que seja um erro relacionado ao namespace ou include, pois se eu dou um include em outra classe normal esse pacote funciona perfeitamente. O erro agora eh : ```Message: Constant FPDF_VERSION already defined```
Answer: A forma correta seria:
<?php namespace controllers; // Deve ser a primeira linha do arquivo e não se deve abrir chaves

include "fpdf/fpdf.php";

class Control {

    /*...*/

    public function PDF($romaneio) {
        // Classe está definida no escopo global, por isso se deve passar o \FPDF. Se tentar inicializar sem o \, é como se estivesse no escopo desse namespace, mas ela não está e por isso se deve chamar com \.
        $pdf =  new \FPDF("P", "pt", "A4");
        /*...*/
    }
}

Obs.: Ocultei partes desnecessárias do código para demonstrar apenas onde deve corrigir. Faça a alteração e veja se vai funcionar. Caso queira chamar a classe FPDF apenas, em vez de \FDPF, você deve incluir no arquivo fpdf/fpdf.php no início do arquivo namespace controllers;.
Comment: Estava com o comentário incorreto. Atualizei o texto, agora a explicação está correta e mais clara.
Comment: Isso mesmo , ele localizou a classe, agora consigo manipular. Muito obrigado Bruno.

Problema com namespace e include


Linear mixed effect models - Why is time specific variance under estimated?

Question: Suppose you fit a linear mixed effect model
$$y_{ij} = \mu + \beta^T x_{ij} + u_i + \epsilon_{ij}$$
where $u_i \sim N(0,\sigma^2)$ and $\epsilon_{ij} \sim N(0,\nu^2)$.
I have noticed that when fitting this model in R using synthetic data (that is specifically generated to follow the above model) using the lme package:
<code>  model <- lme(y~1+x1+x2,data=dat,~1|id/time)
  id = i, time=j
</code>
the package will yield extremely accurate estimates for $\sigma^2$, but $\nu^2$ will be systematically under estimated. Examples
$$\sigma = \nu = 1 \implies  \hat{\sigma}\approx 1, \hat{\nu} \approx 0.91$$
$$\sigma = \nu = 0.5 \implies  \hat{\sigma}\approx 0.5,  \hat{\nu} \approx 0.43$$
This problem does not seem to improve with sample size.
I'm curious if there's an intuitive reason for this, and if it can be corrected for?
Comment: what is id and time? "~1|id/time" should generate the Z matrix for random effect. Do you know what kind of Z will be?
Comment: $id$ is an identifying factor for each individual. $time$ is the time for observation $j$ of individual $i$. eg.$id =1,2,...n$ while $t=1,2,...,T$. My aim with that code is to include two random intercepts, one for $i$ and one for $i:T$, which I thought is what $id/time$ constructs for $Z$?
Comment: Did you do simulation? otherwise how can you know the values of parameters? From the model, you just have one random intercept. I guess change "~1|id/time" to "-1|id" or "~1|time" you will get correct $\hat v$.
Comment: Yes, I did the simulation. What do you mean "-1|id" or "~1|time"?  "-1|id" does not seem to work in lme (unless Im misunderstanding). I can get correct estimate for "time" if I exclude "id" from the model formula, but when "id" is included time is under estimated. Is there a way to correct without removing id?
Comment: should be "~1|id". typo. i = id, j = time?
Comment: Yes, sorry that was not clear.
Comment: Then should be :~1|id". Do not know how did you generated data? Are you sure that data are generated according to your specified model strictly?
Comment: ~1|id fits an intercept for each $i$ giving an estimate of $\sigma$, but not of $\nu$? For some reason lme does not allow me to include intercepts for $1|id$ and $1|id:time$ separately, so I am forced to use $1|id/time$. If I replace $time$ by a factor which is unique to all observations, say $ep$, then I get the same results by fitting $list(~1|id,~1|ep)$ as for $~1|id/t$.
Comment: I am fairly certain the generation is correct. The model is pretty simple, but I will double check.
Comment: For mixed model $Y=X\beta + Z\gamma +\epsilon$, try hard to understand matrix Z. Could you write down Z that you used to generate you data?
Comment: $Z$ is of the form (for $T=3):
$ \begin{bmatrix}
1 & 1 & 0 & 0\\
1 & 0 & 1  & 0\\
1 & 0 & 0  & 1
\end{bmatrix}

with $ \gamma = [u_i, \epsilon_{i1}, \epsilon_{i2}, \epsilon_{i3}]^T $
Comment: page 4 on this https://www.stat.cmu.edu/~cshalizi/490/10/mixed-effects/mixed-effects-01.pdf
Answer: The mathematical formulation and the R code you present do not match. Namely, the mathematical formulation is for a simple random intercepts models, whereas the code for a model with nested random intercepts. That is, the model behind the code is: $$\left \{ \begin{eqnarray} \begin{array}{l} y_{ijk} = \mu + \beta^T x_{ijk} + u_i + b_{ij} + \epsilon_{ijk},\\ u_i \sim \mathcal N (0, \sigma_u^2), \;\; b_{ij} \sim \mathcal N (0, \sigma_b^2), \;\; \epsilon_{ijk} \sim \mathcal N (0, \sigma^2). \end{array}\end{eqnarray}\right.$$
The code behind the model you've written in mathematical formulation should be something like:
<code>model <- lme(y ~ time + x1 + x2, data = dat, random = ~ 1 | id)
</code>
Comment: Hey! Thanks for your response. In the results from running your code I get:
 Formula: ~1 | i
        (Intercept) Residual
StdDev:    1.002582 1.002047

Here, is 1.0025 the estimate of $\sigma_u^2$, while 1.002047 the estimate of $\sigma^2$? That would make a lot of sense.


Do Scale Shapes change with different tunings?

Question: All the scale diagrams I've seen are shown in E standard tuning.

Would the scale shape change in another tuning, like Drop D for example? Alternatively, do I just keep the same scale shape regardless of tuning?
Note: I'm showing a bass guitar diagram because that's what I play, but I imagine this would be the same for most fretted instruments.
Comment: For a quick overview of the effect from tuning on scales and chords have a look here, wehre you can customize your tuning, too: [Example for C-Major scale tuned half step down](https://chord.rocks/bass-guitar/tuning-half-step-down/scales/c-major).
Answer: Yes, they would change. Try to think of the notes not on the fret, but on the specific position on the string. So, if you lower the tuning on that string, let's say by a whole tone, the note that was on the 5th fret would move up to the 7th fret. Similarly if you tune your strings up.
If you tune the whole bass/guitar a whole tone lower, then the fingerings wouldn't change per se, but everything would be a whole tone lower. But if you change the tuning a lot, like you tune your E to a D, and leave the other 3 strings as they are, then the fingerings would change a bit more.
The same goes if you try some other tuning as well (like for instance tuning in fifths like Tim said)
Comment: You say "Of course" as if this answer would be apparent to everyone, or as if another possible answer didn't exist.
Comment: @Dedwards don't take it personally, I didn't mean any offense by it
Comment: Thank your for altering your statement; please don't assume the answer to a question is as obvious as you see it to be.
Answer: Yes, scale and chord shapes do change if you use an alternate tuning.  If the open note was an "E" but has been changed, then all of the other notes on that string will move to other positions and you'll need to play a different "shape" to reach those new positions.
In this sort of situation the best thing to do is get a piece of paper and draw a new fretboard diagram with the notes written in their new positions, and the scale's notes circled to remind you of the new shape.  With a bit of practice you'll re-learn the new intervals for the tunings you use frequently.
Answer: Yes. If the guitar was tuned to just drop D, then all the scale notes on the bottom string would need to be fretted 2 frets higher. With other tunings, because the open string notes have changed, and also changed in relation to each other then adjustments would have to be made all over.
The same scenario would happen on bass, although the most usual change of tuning seems to be a fifth between strings instead of a fourth, thus changing the fingering and fretting of scale notes.


How to tell which variable is more meaningful when modeling the relationship between several predictors and outcome variable?

Question: I'm facing a problem in which I need to figure out two things:

which predictor, out of several relevant ones, is the most meaningful one in its effect/predictive power over a predicted variable.
the order of meaningfulness (from most meaningful to least) across those different predictors.

As I do not have an a-priory hypothesis about this investigation, I thought I should be doing some sort of multiple regression analysis. Then, perhaps I should be extracting the terms from the model and see which one is the most meaningful. I already know that going by p-value isn't the right way. Then what is?
Example
Let's say that I want to investigate which factors affect the well-being of city residents. I sample people (residents) from both New York City and San Francisco, and ask them to rate:

Their general satisfaction from their city.
How clean their city (in their opinion)
How good the level of education in schools
How good the public transportation.

The way I see this, there are 3 relevant variables here (cleanliness, education, and transportation) that may be related to overall satisfaction. I want to model this relationship, and conclude how different NYC from SF is. For example, when it comes to overall satisfaction, is the impact of education > transportation > cleanliness in NYC, whereas in SF transportation > education > cleanliness?
Here's some toy data to demonstrate.
my_df <- structure(list(location = c("sf", "nyc", "nyc", "sf", "nyc",
                                  "nyc", "nyc", "nyc", "nyc", "sf", "nyc", "sf", "sf", "sf", "nyc",
                                  "sf", "sf", "nyc", "nyc", "nyc", "sf", "sf", "sf", "sf", "sf",
                                  "nyc", "sf", "sf", "nyc", "sf", "nyc", "nyc", "nyc", "sf", "nyc",
                                  "nyc", "nyc", "sf", "nyc", "sf", "sf", "nyc", "nyc", "nyc", "nyc",
                                  "nyc", "nyc", "nyc", "nyc", "sf", "nyc", "nyc", "sf", "sf", "nyc",
                                  "nyc", "nyc", "nyc", "sf", "sf", "nyc", "sf", "nyc", "nyc", "sf",
                                  "nyc", "sf", "sf", "nyc", "nyc", "nyc", "nyc", "sf", "nyc", "nyc",
                                  "nyc", "sf", "sf", "nyc", "nyc", "nyc", "nyc", "sf", "sf", "nyc",
                                  "nyc", "nyc", "sf", "sf", "sf", "nyc", "sf", "sf", "sf", "nyc",
                                  "nyc", "nyc", "nyc", "nyc", "nyc"),
                     satisfied = c(5, 1, 7, 5,
                                   7, 1, 5, 5, 5, 7, 7, 4, 1, 3, 5, 6, 7, 7, 6, 4, 4, 5, 6, 5, 5,
                                   7, 5, 6, 5, 4, 7, 7, 5, 5, 4, 7, 7, 5, 6, 6, 3, 6, 5, 7, 5, 7,
                                   6, 5, 4, 3, 6, 5, 7, 3, 5, 5, 7, 5, 6, 7, 7, 7, 7, 5, 4, 7, 6,
                                   7, 7, 6, 6, 6, 5, 7, 5, 4, 6, 4, 7, 5, 6, 6, 5, 5, 6, 7, 6, 5,
                                   1, 5, 2, 7, 7, 7, 7, 7, 1, 3, 7, 7),
                     clean = c(4, 1,
                                         7, 3, 4, 1, 6, 6, 7, 4, 5, 1, 1, 1, 4, 6, 6, 1, 6, 1, 4, 2, 2,
                                         7, 3, 5, 2, 4, 1, 1, 4, 6, 3, 5, 1, 4, 5, 2, 5, 5, 4, 5, 4, 7,
                                         3, 6, 5, 4, 5, 4, 5, 4, 5, 1, 5, 2, 6, 5, 7, 6, 3, 7, 5, 6, 4,
                                         6, 6, 5, 5, 5, 4, 1, 4, 4, 5, 1, 3, 1, 2, 2, 6, 4, 3, 6, 7, 7,
                                         5, 2, 4, 1, 3, 1, 5, 3, 5, 5, 1, 1, 5, 6),
                     edu = c(5,
                                             1, 7, 4, 4, 1, 6, 6, 6, 4, 5, 4, 4, 3, 3, 5, 4, 1, 1, 3, 5, 6,
                                             5, 5, 3, 6, 2, 4, 4, 4, 6, 3, 4, 7, 1, 4, 7, 5, 6, 5, 5, 5, 4,
                                             7, 3, 7, 6, 5, 5, 4, 5, 3, 4, 4, 4, 7, 5, 4, 6, 6, 4, 7, 4, 2,
                                             5, 6, 6, 7, 6, 7, 3, 3, 2, 6, 6, 2, 5, 3, 6, 5, 6, 4, 4, 5, 6,
                                             7, 3, 3, 4, 5, 4, 1, 3, 4, 4, 6, 5, 1, 4, 6),
                     transportation = c(1,
                                              1, 7, 5, 7, 1, 4, 6, 6, 6, 6, 1, 1, 1, 5, 5, 4, 7, 6, 6, 7, 5,
                                              2, 7, 3, 6, 1, 4, 7, 5, 6, 7, 4, 3, 2, 6, 4, 2, 6, 5, 4, 7, 6,
                                              7, 3, 7, 4, 4, 5, 4, 6, 3, 5, 2, 7, 3, 7, 7, 7, 6, 7, 7, 7, 5,
                                              3, 5, 4, 7, 6, 6, 4, 2, 4, 4, 5, 6, 5, 2, 6, 2, 6, 6, 3, 4, 7,
                                              7, 7, 4, 5, 4, 5, 3, 7, 5, 7, 7, 7, 1, 6, 6)),
                row.names = c(NA, -100L),
                class = c("tbl_df", "tbl", "data.frame"))

library(magrittr)
library(effectsize)

my_df %>%
  lm(satisfied ~ clean*location + edu*location + transportation*location, data = .) %>%
  effectsize() %>%
  plot()
#> Warning: It is deprecated to specify `guide = FALSE` to remove a guide. Please
#> use `guide = "none"` instead.

Created on 2021-08-15 by the reprex package (v2.0.0)
To compare between cities, I added interaction terms between each edu/clean/transportation variable and location variable. Then I used effectsize::effectsize() to get the estimates from the model. But what can I conclude from those estimates?
If I completely got this wrong, please advise what other path I should take for tackling this problem.
Thanks!
Comment: I think looking at variation (chi^2 statistic) and penalizing by degrees of freedom looks to be the best procedure. But this is subjective. see the anova function in the rms package.
Comment: My preference is to apply analysis of variance to the model
Comment: @JTH, thanks. Could you please hint how you would then conclude about the *meaningfulness* of each predictor?
Comment: @JTH, thank you for recommending the `{rms}` package, I didn't know about it. I've tried wrapping my head around its functions, but it's all very new to me. I'll be thankful if you could provide an example of how *you* would utilize `rms` functions to do ANOVA over the data I showed in the post. I could then pick up from such an example.
Comment: @JTH, I took a stub with `rms`. Please see my answer below. Could you provide feedback please? I also didn't find how to follow your suggestion with chi-square and df.
Comment: @Emman I would suggest looking at the `residuals` of the model. The residuals can tell you a lot whether the model is effective or not.
Answer: I believe you're looking for a metric of (relative) variable importance (see also this thread). Many available methods rely on the decomposition of the $R^2$ to assign ranks or relative importance to each predictor in a multiple linear regression model. A certain approach in this family is better known under the term "Dominance analysis" (see Azen et al. 2003). Azen et al. (2003) also discuss other measures of importance such as importance based on regression coefficients, based on correlations of importance based on a combination of coefficients and correlations. A general good overview of techniques based on variance decomposition can be found in the paper of Grömping (2012). These techniques are implemented in the R packages relaimpo, domir and yhat.
Here, I'm going to illustrate a method that is model-agnostic (i.e. it can be applied to a variety of model types) and has intuitive appeal: Variable importance based on permutation. The idea is very simple:

Decide on a performance metric that is important to you. Examples include: Root mean square error (RMSE), mean absolute error (MAE), $R^2$ etc. This also is somewhat dependent in the model type.
Calculate the metric on your dataset, $M_{orig}$. This serves as baseline performance metric.
For $i = 1, 2, \ldots, j$:
(a) Permute the values of the predictor $X_i$ in the data set.
(b) Recompute the metric on the permuted data and call it $M_{perm}$.
(c) Record the difference from baseline using $imp(X_i)=M_{perm} - M_{orig}$.

Do this repeatedly, say 1000 times, and take the average of the importance values. Intuitively, the permutations break the relationship between the predictor $X_i$ and the outcome. The larger the change in the performance metric, the higher the predictors' importance. More information can be found in this chapter of an online book by Christoph Molnar.
The R package vip implements this procedure (see the documentation (PDF) for more information). The following code applies the idea to your dataset. I chose the $R^2$ and the mean absolute error (MAE) as performance metrics and permute each predictor 1000 times:
library(vip)
# The model
mod <- lm(satisfied ~ clean*location + edu*location + transportation*location, data = my_df)

# Calculate permutation-based importance with r-squared as metric
set.seed(142857) # For reproducibility
p_r2 <- vip::vi(mod, method = "permute", target = "satisfied", metric = "rsquared", pred_wrapper = predict, nsim = 1000)
p_r2

  Variable       Importance  StDev
  <chr>               <dbl>  <dbl>
1 transportation     0.198  0.0492
2 clean              0.177  0.0465
3 edu                0.0462 0.0237
4 location           0.0449 0.0250

# Calculate permutation-based importance with mae as metric
p_mae <- vip::vi(mod, method = "permute", target = "satisfied", metric = "mae", pred_wrapper = predict, nsim = 1000)
p_mae

  Variable       Importance  StDev
  <chr>               <dbl>  <dbl>
1 transportation     0.166  0.0413
2 clean              0.144  0.0400
3 location           0.0396 0.0214
4 edu                0.0368 0.0219

According to the $R^2$, permuting transportation leads to the largest change in $R^2$, followed by clean. Using the mean absolute error shows a similar ordering with transportation and clean being most important while location and edu being least important.
References
Azen R, Budescu DV (2003): The Dominance Analysis Approach for Comparing Predictors in Multiple Regression. Psychological Methods 8:2, 129-148. (link)
Grömping U (2012): Estimators of relative importance in linear regression based on variance decomposition. Am Stat 61:2, 139-147. (link)
Comment: Thanks, this is new to me and looks interesting and relevant, and well explained. Is there a way to compare between cities (levels of `location`) within this framework? Essentially, I want to know if the order of `Importance` values in `p_r2`/`p_mae` remains the same or changes when we examine each city on its own. Is there any way to do that?
Comment: Great overview, thanks! Whilst the permutation method is attractive due to its universality, I wonder whether it can lead to misleading results because it breaks the dependency structure among the predictors. I have tried it on http://users.stat.ufl.edu/~winner/data/worldagprod.dat with the model `Output~Population.in.agriculture+Workstock+Land.equivalent+Fertilizer.consumption`. Interestingly, the permutation method yields results very similar to standardized coeffizients (betasq metric in *relaimpo*), but differes in rank order and relative difference from the $R^2$ decomposition methods.
Answer: The following answer is some sort of an ignorant attempt to use {rms} package, following @JTH's suggestion. I got to say that this is the first time I'm using this package, and I have very minimal understanding of what I'm doing. Hence, I ask that anybody who can -- please provide feedback!
I've followed the procedure described in this chapter.
my_df <- structure(list(location = c("sf", "nyc", "nyc", "sf", "nyc",
                                     "nyc", "nyc", "nyc", "nyc", "sf", "nyc", "sf", "sf", "sf", "nyc",
                                     "sf", "sf", "nyc", "nyc", "nyc", "sf", "sf", "sf", "sf", "sf",
                                     "nyc", "sf", "sf", "nyc", "sf", "nyc", "nyc", "nyc", "sf", "nyc",
                                     "nyc", "nyc", "sf", "nyc", "sf", "sf", "nyc", "nyc", "nyc", "nyc",
                                     "nyc", "nyc", "nyc", "nyc", "sf", "nyc", "nyc", "sf", "sf", "nyc",
                                     "nyc", "nyc", "nyc", "sf", "sf", "nyc", "sf", "nyc", "nyc", "sf",
                                     "nyc", "sf", "sf", "nyc", "nyc", "nyc", "nyc", "sf", "nyc", "nyc",
                                     "nyc", "sf", "sf", "nyc", "nyc", "nyc", "nyc", "sf", "sf", "nyc",
                                     "nyc", "nyc", "sf", "sf", "sf", "nyc", "sf", "sf", "sf", "nyc",
                                     "nyc", "nyc", "nyc", "nyc", "nyc"),
                        satisfied = c(5, 1, 7, 5,
                                      7, 1, 5, 5, 5, 7, 7, 4, 1, 3, 5, 6, 7, 7, 6, 4, 4, 5, 6, 5, 5,
                                      7, 5, 6, 5, 4, 7, 7, 5, 5, 4, 7, 7, 5, 6, 6, 3, 6, 5, 7, 5, 7,
                                      6, 5, 4, 3, 6, 5, 7, 3, 5, 5, 7, 5, 6, 7, 7, 7, 7, 5, 4, 7, 6,
                                      7, 7, 6, 6, 6, 5, 7, 5, 4, 6, 4, 7, 5, 6, 6, 5, 5, 6, 7, 6, 5,
                                      1, 5, 2, 7, 7, 7, 7, 7, 1, 3, 7, 7),
                        clean = c(4, 1,
                                  7, 3, 4, 1, 6, 6, 7, 4, 5, 1, 1, 1, 4, 6, 6, 1, 6, 1, 4, 2, 2,
                                  7, 3, 5, 2, 4, 1, 1, 4, 6, 3, 5, 1, 4, 5, 2, 5, 5, 4, 5, 4, 7,
                                  3, 6, 5, 4, 5, 4, 5, 4, 5, 1, 5, 2, 6, 5, 7, 6, 3, 7, 5, 6, 4,
                                  6, 6, 5, 5, 5, 4, 1, 4, 4, 5, 1, 3, 1, 2, 2, 6, 4, 3, 6, 7, 7,
                                  5, 2, 4, 1, 3, 1, 5, 3, 5, 5, 1, 1, 5, 6),
                        edu = c(5,
                                1, 7, 4, 4, 1, 6, 6, 6, 4, 5, 4, 4, 3, 3, 5, 4, 1, 1, 3, 5, 6,
                                5, 5, 3, 6, 2, 4, 4, 4, 6, 3, 4, 7, 1, 4, 7, 5, 6, 5, 5, 5, 4,
                                7, 3, 7, 6, 5, 5, 4, 5, 3, 4, 4, 4, 7, 5, 4, 6, 6, 4, 7, 4, 2,
                                5, 6, 6, 7, 6, 7, 3, 3, 2, 6, 6, 2, 5, 3, 6, 5, 6, 4, 4, 5, 6,
                                7, 3, 3, 4, 5, 4, 1, 3, 4, 4, 6, 5, 1, 4, 6),
                        transportation = c(1,
                                           1, 7, 5, 7, 1, 4, 6, 6, 6, 6, 1, 1, 1, 5, 5, 4, 7, 6, 6, 7, 5,
                                           2, 7, 3, 6, 1, 4, 7, 5, 6, 7, 4, 3, 2, 6, 4, 2, 6, 5, 4, 7, 6,
                                           7, 3, 7, 4, 4, 5, 4, 6, 3, 5, 2, 7, 3, 7, 7, 7, 6, 7, 7, 7, 5,
                                           3, 5, 4, 7, 6, 6, 4, 2, 4, 4, 5, 6, 5, 2, 6, 2, 6, 6, 3, 4, 7,
                                           7, 7, 4, 5, 4, 5, 3, 7, 5, 7, 7, 7, 1, 6, 6)),
                   row.names = c(NA, -100L),
                   class = c("tbl_df", "tbl", "data.frame"))

library(rms, warn.conflicts = FALSE)
#> Loading required package: Hmisc
#> Loading required package: lattice
#> Loading required package: survival
#> Loading required package: Formula
#> Loading required package: ggplot2
#>
#> Attaching package: 'Hmisc'
#> The following objects are masked from 'package:base':
#>
#>     format.pval, units
#> Loading required package: SparseM
#>
#> Attaching package: 'SparseM'
#> The following object is masked from 'package:base':
#>
#>     backsolve

model_fit <- rms::ols(satisfied ~ clean*location + edu*location + transportation*location, data = my_df)

my_datadist <- rms::datadist(my_df)     ## apparently, we need these two lines
options(datadist = "my_datadist")  ## otherwise we get an error with `summary(model_fit)`
                                   ## I learned it from here: https://stackoverflow.com/a/41378930/6105259

plot(summary(model_fit))

Created on 2021-08-16 by the reprex package (v2.0.0)

As far as I understand this output, we see the effect each predictor carries on the outcome variable, with 95% CI. Thus, for example, we can conclude that clean has a greater effect over satisfied than edu has.

Does this make sense?
Originally, I was interested in the interaction between each edu/clean/transportation and location, as I want to learn how the relationships between predictors and outcome change between cities. But here, as far as I can see from this output, the interaction isn't reflected in terms of effect on satisfied.

UPDATE

Following @JTH's comment, I'm adding another plot:
plot(anova(model_fit))

The info underlying this plot is here:
library(dplyr, warn.conflicts = FALSE)
library(tibble)

anova(model_fit) %>%
  as_tibble(rownames = "factor") %>%
  mutate(across(3:6, round, 4))
#> # A tibble: 14 x 6
#>    factor                          d.f.     `Partial SS` MS      F       P
#>    <chr>                           <anov.r> <anov.rms>   <anov.> <anov.> <anov.>
#>  1 "clean  (Factor+Higher Order F~  2        11.8209      5.9105 3.4587  0.0356
#>  2 " All Interactions"              1         0.0000      0.0000 0.0000  0.9969
#>  3 "location  (Factor+Higher Orde~  4         4.8955      1.2239 0.7162  0.5830
#>  4 " All Interactions"              3         4.8821      1.6274 0.9523  0.4188
#>  5 "edu  (Factor+Higher Order Fac~  2         4.0844      2.0422 1.1951  0.3073
#>  6 " All Interactions"              1         3.1038      3.1038 1.8163  0.1811
#>  7 "transportation  (Factor+Highe~  2        15.3207      7.6604 4.4828  0.0139
#>  8 " All Interactions"              1         0.0476      0.0476 0.0279  0.8678
#>  9 "clean * location  (Factor+Hig~  1         0.0000      0.0000 0.0000  0.9969
#> 10 "location * edu  (Factor+Highe~  1         3.1038      3.1038 1.8163  0.1811
#> 11 "location * transportation  (F~  1         0.0476      0.0476 0.0279  0.8678
#> 12 "TOTAL INTERACTION"              3         4.8821      1.6274 0.9523  0.4188
#> 13 "TOTAL"                          7        90.9761     12.9966 7.6055  0.0000
#> 14 "ERROR"                         92       157.2139      1.7088     NA      NA

UPDATE 2

Addressing @EdM's comment, here is a print of anova(model_fit) without converting it to a tibble.
anova(model_fit) %>%
  round(., 4)
#>                 Analysis of Variance          Response: satisfied
#>
#>  Factor                                                   d.f. Partial SS
#>  clean  (Factor+Higher Order Factors)                      2    11.8209
#>   All Interactions                                         1     0.0000
#>  location  (Factor+Higher Order Factors)                   4     4.8955
#>   All Interactions                                         3     4.8821
#>  edu  (Factor+Higher Order Factors)                        2     4.0844
#>   All Interactions                                         1     3.1038
#>  transportation  (Factor+Higher Order Factors)             2    15.3207
#>   All Interactions                                         1     0.0476
#>  clean * location  (Factor+Higher Order Factors)           1     0.0000
#>  location * edu  (Factor+Higher Order Factors)             1     3.1038
#>  location * transportation  (Factor+Higher Order Factors)  1     0.0476
#>  TOTAL INTERACTION                                         3     4.8821
#>  REGRESSION                                                7    90.9761
#>  ERROR                                                    92   157.2139
#>  MS      F    P
#>   5.9105 3.46 0.0356
#>   0.0000 0.00 0.9969
#>   1.2239 0.72 0.5830
#>   1.6274 0.95 0.4188
#>   2.0422 1.20 0.3073
#>   3.1038 1.82 0.1811
#>   7.6604 4.48 0.0139
#>   0.0476 0.03 0.8678
#>   0.0000 0.00 0.9969
#>   3.1038 1.82 0.1811
#>   0.0476 0.03 0.8678
#>   1.6274 0.95 0.4188
#>  12.9966 7.61 <.0001
#>   1.7088

Comment: Again, this is subjective, but my preference is for Chi^2 - DF, which is the x-axis of `plot(anova(model_fit))`. The reason the penalization is important is because you can end up in situations where the # categories determines the factor relevance.
Comment: It might help for you to get the [course notes](http://hbiostat.org/doc/rms.pdf) and [book](https://hbiostat.org/rms/) associated with the `rms` package. Besides showing examples of how to do this evaluation of variable importance, they also show the dangers of putting too much trust in that evaluation. See the sections on "Bootstrapping Ranks of Predictors" in Chapter 5 of both references. Chapter 4 (in both) is a superb summary of strategies for regression modeling, which I recommend that you read carefully and keep as a guide to doing this type of work.
Comment: Try `plot(anova(model_fit))`, which graphically shows the technique I mentioned in my other comment.
Comment: The way you printed the `anova.rms` object, you lost some text. For each of the individual variable names (`clean`, `location`, `edu`, `transportation`) the corresponding row in the printout includes calculations for the "Factor + Higher Order Factors"; in this case, the variable _plus_ all interactions involving it. That's why the `Partial SS` and `MS` values don't agree in those rows. The p-value for each of those rows agrees with the p-value in the corresponding row of the plot, which also includes the interactions. Negative plot values mean chi-square is less than the corresponding df.
Comment: I agree with @JTH that ($\chi^2$ - df) is a reasonable measure here. Packages like `rms` can do different processing on an object returned by a regression depending on whether they perform `print()` or `plot()` or `summary()` on it. In this case, ($\chi^2$ - df) is most easily seen in the plot; you would need to do more calculations to get it from the printout.
Comment: "To see how this set of predictors changes when we look at one city ... versus the other" you need to look at the _interaction_ terms. In this case, none of them is significant, so you can't reliably say that cities differ. An advantage of that plot is precisely that it _doesn't_ depend on choice of reference level: it's an overall measure for that predictor or interaction term. If interactions are significant, you can use `Predict()` and `contrast.rms()` to get into details of how the cities differ.
Comment: @cdalitz with OLS, each $\chi^2$ value in the plot is the product of the corresponding [partial F statistic](https://stats.stackexchange.com/q/157102/28500) times its df. Subtracting the df penalizes for the number $p$ of fitted predictors included in that estimate, as $p$ is the expected value of $\chi^2$ under the null. So I suppose you could consider it analogous to Cohen's $f^2$ but with penalization. This nicely generalizes to regressions other than OLS, with $\chi^2$ values from a Wald statistic on each set of coefficients and covariances.
Comment: @JTH, thanks. I updated my answer. Looking at the plot (the x axis), we could have thought that some values are negative (to the left of `0`). However, I think that this is just an issue with the graphics. Still, should we go by the `Partial SS` or the `MS` column? For the interaction terms, both columns hold the same values, but this isn't the case for the other terms. I'm a bit confused...
Comment: @EdM, thank you! your comments are very helpful. I've updated the post and added the raw print of `anova(model_fit)` (with some rounding).
Comment: @EdM, thanks! One thing I don't understand though -- part of my original answer was to see how this set of predictors *changes* when we look at one city (`nyc`/`sf`) versus the other. But when I plot with `plot(anova(model_fit))`, what do I see here? Is it supposed to be according to `location`'s reference level? That is, if we tweak using `relevel(location, ref = ...)` then the plot should be somewhat different? I can say that I tried releveling but the plot remains the same.
Comment: @EdM Do these values measure effect size or statistical significance? Or is in this particular use case a monotonous relationship between the ANOVA p values and Cohen's $f^2$, which measures how strongly each variable improves the fit?
Answer: As all your variables are numeric and limited to the same range of values, directly comparing the absolute values of the coefficients (as you did in your answer) is one possible way to estimate effect size.
It might be, however, that for one variable the answers are only in the range 1-3, while they are for another variable in the range 1-5. If both variables are equally important, the second will have a smaller coefficient due to its wider range. To overcome this problem, you could compute standardized coefficients, aka "beta coefficients". There are convenience functions for their computation in addon packages, but with base R you can achieve the same by standardizing your data before doing the model fit:
my_df.scaled <- scale(my_df)

In addition, you should also check, whether all coefficients are significantly different from zero.
Answer: I'd look at the residuals for determining the effectiveness of a model. Plots of residuals versus other quantities are used to find failures of assumptions. The most common plot, especially useful in simple regression, is the plot of residuals versus the fitted values. A null plot would indicate no failure of assumptions. Curvature might indicate the fitted mean function is incorrect. Residuals that seem to increase or decrease in average magnitude with the fitted values might indicate nonconstant residual variance. A few relatively large residuals may be indicative of outliers, cases for which the model is somehow inappropriate.
Assumptions of a linear model are:

Linear relationship: There exists a linear relationship between the independent variable, x, and the dependent variable, y.

Independence: The residuals are independent. In particular, there is no correlation between consecutive residuals.

Homoscedasticity: The residuals have constant variance at every level of x.

Normality: The residuals of the model are normally distributed.

If one or more of these assumptions are violated, then the results of our linear regression may be unreliable or even misleading.
Using the given data, I build a simple linear regression model given below.
# required libraries
library(caret)
library(ggplot2)
library(magrittr)

# split the train dataset into train and test set
set.seed(2021)
index <- createDataPartition(my_df$satisfied, p = 0.7, list = FALSE)
df_train <- my_df[index, ]
df_test  <- my_df[-index, ]

# Model building
lm_model <- lm(satisfied ~., data = my_df)
# Make predictions and compute the R2, RMSE and MAE
predictions <- lm_model %>% predict(df_test)
data.frame( R2 = R2(predictions, df_test$satisfied),
            RMSE = RMSE(predictions, df_test$satisfied),
            MAE = MAE(predictions, df_test$satisfied))
         R2      RMSE       MAE
1 0.4513587 0.9956456 0.8224509
# Residuals = Observed - Predicted
# compute residuals
residualVals <- df_test$satisfied - predictions
df.1 <- data.frame(df_test$satisfied, predictions,
                   residualVals)
colnames(df.1)<- c("observed","predicted","residuals")
head(df.1)
  observed predicted   residuals
1        5  4.443428  0.55657221
2        7  6.930498  0.06950225
3        4  3.626674  0.37332568
4        3  3.538958 -0.53895795
5        5  5.083349 -0.08334872
6        7  6.097199  0.90280068
ggplot(data = df.1, aes(x=predicted, y=residuals))+
  geom_point()+
  xlab("Predicted values for satsified")+
  ylab("Residuals")+
  ggtitle("Residual plot")+
  theme_bw()

Discussion
To understand the strength/weakness of a model, relying on a single metric is problematic. Visualization of model fit, particularly residual plots in the context of linear regression model, are critical to understanding whether the model is fit for purpose.
When the outcome is a number, the most common method for characterizing a model's predictive capabilities is to use the root mean squared error (RMSE). This metric is a function of the model residuals, which are the observed values minus the model predictions. The mean squared error (MSE) is calculated by squaring the residuals and summing them. The RMSE is then calculated by taking the square root of the MSE so that it is in the same units as the original data. The value is usually interpreted as either how far (on average) the residuals are from zero or as the average distance between the observed values and the model predictions. Another common metric is the coefficient of determination, commonly written as R2. This value can be interpreted as the proportion of the information in the data that is explained by the model. Thus, an R2 value of 0.45 in the above model, implies that the model can explain less than half of the variation in the outcome/dependent/response variable, satisfied variation. Simply put, the above model is not good. It should be noted that R2 is a measure of correlation and not accuracy.
Comment: Thanks. Could you explain further how this method helps you to learn about which predictor is more meaningful compared to the others?
Comment: @Emman I've added a discussion. Let me know if it helps or not. If not, be descriptive on what you would like to discuss.
Comment: thanks for adding the discussion, While it is informative and generally relevant, it doesn't address the initial question that led to this thread: how can we compare *between* individual predictors? Your answer discusses the model as a whole, but this is not what I seek to understand. I simply want to know whether in a given city, residents value education more than transportation, and whether in a different city it might be the other way around. It is *my* assumption that predicting satisfaction can lead to such conclusions. How can we utilize your method to arriving at such bottom line(s)?
Comment: @Emman sadly but you'll have to figure out the relationship between predictors on your own. Mine and others who have given varied options should be seen as pointers/directions, that can be used to make sense of relationships you want to understand/comprehend/prove.


How do proxy providers have such a massive amount of IPs?

Question: I was looking at a proxy provider this evening and it said:

Our IPv4 database consists of over 500,000 IP addresses from all 6 major continents with 50 Mbps to 10 Gbps bandwidth.

I'm just wondering, how do these proxy providers get such a huge amount of IP addresses?
Answer: A proxy provider is just a kind of ISP. They can just request addresses from the relevant RIRs (ARIN, RIPE NCC etc). If they need 500.000 addresses, they get them.
Of course these days the RIRs don't have any IPv4 addresses anymore, but before they ran out it was just "show that you need them, and you get them".
This is also how it works for IPv6, of which there is plenty.
Comment: That makes sense. So, entering the market as a proxy provider nowadays must be nearly impossible, unless you're providing ipv6 IP addresses.
Comment: Indeed, unless you have a few million to spare to buy that many IPv4 addresses from someone else.


Restrict IIS site with Windows Authentication folder permissions

Question: I created two IIS sites which both have Windows Authentication and Directory Browsing enabled.
For the first one, let's call it "admin", I set no special permissions.  I navigated to the site and was prompted for credentials. I logged in with an administrator account on the machine and was able to browse the folder.  Perfect.
For the second one, let's call it "test", I created a new Windows user and removed it from ALL groups including "Users".  I created a new folder on the drive and gave this new user permissions to the folder.  When I access this site via a browser and login with the new account, I am able to browse the folder as expected.  However, I am also able to browse the "admin" site above with this new account.  This is not expected, as the only users that have permissions to the folder on the drive are "CREATOR OWNER", "SYSTEM", "Administrators" and "Users".  My new user is not a member of any of those groups.
Why is this happening? What is the proper way, with IIS, to secure a folder with a local account, but ensure that account has no access to other folders?
Answer: Just because your second user is not a member of the Users group, it doesn't mean it is not a part of the Users group during runtime.
Take my user Peter:
 net user peter

 Local Group Memberships      *Guests
 Global Group memberships     *None

just a member of Guests, not Users.
But when I ask for my security context during runtime:
 whoami /groups

 Group Name                             Type             SID
 ====================================== ================ ============
 Everyone                               Well-known group S-1-1-0
 BUILTIN\Guests                         Alias            S-1-5-32-546
 BUILTIN\Performance Log Users          Alias            S-1-5-32-559
 BUILTIN\Users                          Alias            S-1-5-32-545
 NT AUTHORITY\INTERACTIVE               Well-known group S-1-5-4
 CONSOLE LOGON                          Well-known group S-1-2-1
 NT AUTHORITY\Authenticated Users       Well-known group S-1-5-11
 NT AUTHORITY\This Organization         Well-known group S-1-5-15
 NT AUTHORITY\Local account             Well-known group S-1-5-113
 LOCAL                                  Well-known group S-1-2-0
 NT AUTHORITY\NTLM Authentication       Well-known group S-1-5-64-10

We can see that I am in Users as well as in a whole bunch of other special groups.
To limit files to certain users, you have to remove the permissions for the Users group.


Question on proving validity in predicate logic

Question: Using the proof rules of predicate logic prove the validity of the following sequent:
$$ \forall X  \exists Y(P(X)\lor Q(Y))\vdash \exists Y \forall X(P(X)\lor Q(Y)) $$
I have been trying to prove this for quite a long time but nothing at all seems to work.Any guidelines on how to go about it would really help. Thanks.
The Question is taken from Exercise 2.3 Q9(f) from the book Logic in computer science by Michael Huth and Mark Ryan.
Comment: Hint: I don't know the precise system of logic in your book, but it may help if I explain why this is valid. If $\exists Y\,Q(Y)$ is valid, then the right-hand side of the sequent is already valid using any witness for $\exists Y\,Q(Y)$ as a witness. If not then, $\forall Y\lnot(Q(Y))$ is valid, but then the left-hand side of the sequent is equivalent to $\forall X\,P(X)$, which means the right-hand side of the sequent follows from the left-hand side using any $Y$ as a witness. So the structure of the proof is a case analysis on $\exists Y\,Q(Y)$.
Answer: I looked at your book and the system it uses seems to be the same as the Fitch system I like to use.  The key to this proof is to set it up as a Proof by Contradiction:


Pytesseract - output is extremely inaccurate (MAC)

Question: I installed pytesseract via pip and its result is terrible.
As I searched for it, I think I need to give it more data
but I can't find where to put tessedata(traineddata)
since there is no directory like ProgramFile\Tesseract-OCR using Mac.
There is no problem with images' resolution, font or size.
Image whose result is 'ecient Sh Abu'
Because large and clear test images work fine, I think it is a problem about lack of data.
But any other possible solution is welcomed as long as it can read text with Python.
Please help me..
Comment: Can you expand on what you mean by "runs really poor?" It sounds like you're not getting the correct results, and you're not sure where to put the training data?
Comment: That's what I exactly meant. Even if I put a large enough and clear image, it doesn't recognize it properly. Since I found no answer for my problem during hours of searching but putting data, I questioned where should I put data but I'm not sure what's the reason for this. And thank you for trying to help me, sorry for my poor English..
Answer:
I installed pytesseract via pip and its result is terrible.

Sometimes you need to apply preprocessing to the input image to get accurate results.

Because large and clear test images work fine, I think it is a problem about lack of data. But any other possible solution is welcomed as long as it can read text with Python.

You could say lack of data is a problem. I think you'll find morphological-transformations useful.
For instance if we apply close operation, the result will be:

The image looks similar to the original posted image. However there are slight changes in the output images (i.e. Grammar word is slightly different from the original image)
Now if we read the output image:
English
Grammar Practice
ter K-SAT (1-10)

Code:
import cv2
from pytesseract import image_to_string

img = cv2.imread("6Celp.jpg")
gry = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
opn = cv2.morphologyEx(gry, cv2.MORPH_OPEN, None)
txt = image_to_string(opn)
txt = txt.split("\n")
for i in txt:
    i = i.strip()
    if i != '' and len(i) > 3:
        print(i)

Comment: Thanks for your help!
Comment: @Windowsill make sure to search image skewing and correction if you are working with the rotated images.


Let n be any positive integer. Find all primitive pythagorean triples which have 2^n as a side.

Question: Q. Let n be any positive integer. Find all primitive pythagorean triples which have 2^n as a side.
So here is what I have;
We see that 2^n = 2∙2∙2∙∙∙2, n times, and is clearly an even number. Then we see 2^n can be the side x, for where x=2st, by our definition of primitive pythagorean triplet. That is 2^n=2st ⇒ 2^n-1=st ⇒
not sure if I am headed in the right direction, and if I am where to go from here.
Answer: Hint:  So now you have $st=2^{n-1}$, with $s$ and $t$ relatively prime and of opposite parity.  Can you figure out $s,t$ from that?  There are not many choices.
Comment: It would have seemingly endless answers for any increasing n? I know a simple example n=2 => s=1, t=2 as part of the 3,4,5 triangle but how can I tell if there is a definitive amount.
Comment: Next try $n=3$.  We need $st=4$.  There are not many ways to factor $4$, especially with one of the factors odd.  For $n=4$, you have to factor $8$ with one of the factors odd.
Comment: So the only way to do would be with s=4, t=1, and then for 2^4 = 16 , the only way to do so would be s=8, t= 1? So all P.P.T.'s have the form of s=2^(n-1) and t=1?
Comment: and we know that the only integers that divide 2^n are 2 and 1
Comment: No need to use $n-1$ instead of $n$, but yes :)
Comment: That is right.  You get $s=2^{n-1},t=1$ giving a triangle of $2^{2n-2}-1,2^n,2^{2n-2}+1$.  For $n=3$ this is $15,8,17$ and $n=4$ gives $63,16,65$
Answer: $$ a = k\cdot(m^2 - n^2)   ,\ \, b = k\cdot(2mn) ,\ \, c = k\cdot(m^2 + n^2)\\
\implies b=2^n\\
\implies mnk=2^{n-1}\\
\implies k=\dfrac{2^{n-1}}{mn}$$
Thus,
$$a=\dfrac{2^{n-1}}{mn}\cdot(m^2 - n^2)=\dfrac{2^{n-1}m}{n}-\dfrac{2^{n-1}n}{m}\\
\implies b=\dfrac{2^{n-1}m}{n}+\dfrac{2^{n-1}n}{m}$$
Then, since $m$ and $n$ relatively prime and of opposite parity, you can figure out their values. Substitute the result into the above equation.
Comment: Not sure I understand what's going on here.. are you taking multiples of the general equation by k?


Is a "weighted" covariance/variance calculation still positive semi definite?

Question: Suppose I want to calculate a sample variance/covariance matrix:
cov(x) = $ \frac{\sum _{i=1} ^k (x_i - \overline{x})(x_i - \overline{x})^T}{k} $
I was wondering, if I calculate
$ \frac{\sum _i n_i(x_i - \overline{x})(x_i - \overline{x})^T}{k}$ , where n$_i>0$ are scalar
Would the resulting matrix still be positive semi definite?
Thank you for any insight on this.
Answer: Yes. Each $S_i = (x_i-\bar x)(x_i - \bar x)^T$ is a p.s.d. matrix, so any combination with non-negative weights will be p.s.d., as well.
To wit, consider, for an arbitrary compatible (same dimension) $a\in \mathbb{R}^p, p = {\rm dim}\, x$, the quadratic form $a^T S_i a$:
$$
a^T S_i a = a^T (x_i-\bar x)(x_i - \bar x)^T a = [a^T (x_i-\bar x)] \, [(x_i - \bar x)^T a] = [a^T (x_i-\bar x)]^2 \ge 0
$$
since the term in the square brackets is simply a scalar. The worst thing that can happen is that the vector $a$ is orthogonal to $x_i-\bar x$.
Comment: Is that because every vector times its transpose would lead to a p.s.d matrix? Is that always the case? I thought that just the overall sum $ \frac{\sum _{i=1} ^k (x_i - \overline{x})(x_i - \overline{x})^T}{k} $ was p.s.d.
Comment: yes. I extended the answer to elaborate on this point.


Multiplicações contidas em textos

Question: Como fazer as multiplicações contidas em um campo varchar? (para obter a área, no caso)
+------------------------------------------------------+
|                     DESCRIÇÃO                        |
+------------------------------------------------------+
|Chapa Térmica 605x745 Speed Chemfree                  |
|Chapa Térmica 665 X 910 Ryobi Fuji                    |
|Chapa Convencional P 4001 605x745 Speed IBF           |
|Chapa Térmica 665 X 910 X 030 sem Processo Ryobii IBF |
|Chapa Térmica 800x1030 Komori Chemfree                |
+------------------------------------------------------+

Buscando uma solução alternativa, encontrei uma campo com o tipo de um código para as chapas, o qual contém as dimensões das mesmas:
+--------------+------------------------------------------------------+
|     CODE     |                    DESCRIPTION                       |
+--------------+------------------------------------------------------+
| T605745SPCHE | Chapa Térmica 605x745 Speed Chemfree                 |
| T665910RYJUF | Chapa Térmica 665 X 910 Ryobi Fuji                   |
| C605745SPIBF | Chapa Convencional P 4001 605x745 Speed IBF          |
| T665910RYIBF | Chapa Térmica 665 X 910 X 030 sem Processo Ryobii IBF|
| T8001030KBF  | Chapa Térmica 800x1030 Komori Chemfree               |
+--------------+------------------------------------------------------+

Creio que meu único problemas agora é como tratar os códigos com 6 ou 7 números a serem multiplicados. Alguma sugestão?
Answer: Não sei em qual linguagem está trabalhando então vou tentar ser genérico na resposta:
Comece separando as linhas, e em casa linha utilize uma expressão regular para reconhecer o formato, como por exemplo:
\b[0-9]\+x[0-9]\+

Essa é uma expressão regular do tipo sed, pode ler mais em
https://www.gnu.org/software/sed/manual/html_node/Regular-Expressions.html
A expressão que escrevi diz o seguinte: obténha uma expressão iniciada por um espaço em branco (\b) seguido de um ou mais dígitos ([0-9]+), seguida de um x, e em seguida de um ou mais dígitos.
Pode então separá-las com um split no símbolo x, ou seja, separe em dois elementos, o anterior e o posterior ao x, daí os converta a números e simplesmente vai ter a área calculada.
Comment: Obrigado @AndréClaudinho. A Linguagem que estou usando é MSSQL 2008, está identificada na `TAG`. Consegui um resultado.
Answer: Com essa nova coluna, as coisas ficaram bem mais fáceis. Fiz o seguinte:
'Area' = (CONVERT(decimal,SUBSTRING(table.CodItem,2,3))*

        CASE WHEN SUBSTRING(table.CodItem,8,1) LIKE '[0-9]' THEN
            CONVERT(decimal,SUBSTRING(table.CodItem,5,4)) ELSE
            CONVERT(decimal,SUBSTRING(table.CodItem,5,3)) END)/1000000

A função CONVERT e a divisão por /1000000 eu utilizei apenas para conversão de mm² para m². Obrigado!

Multiplicações contidas em textos


Pythagorean triples and Pell's equations

Question: We all know that Pythagorean triples are diophantine equations of order 2. So my question is can the solutions to ALL Pell's equations of the form $X^2-DY^2=1$ be derived from the Pythagorean triples? Is there an algorithm to do it?
P.S: I found an interesting paper (details below) that deals with the solubility of negative Pell's equation in terms of Primitive Pythagorean Triples. I was wondering if an algorithm be derived from it for even the normal Pell's equations.
A. Grytczuk, F. Luca and M. Wójtowicz, "The Negative Pell Equation and Pythagorean Triples"
Comment: Yes and yes.  See https://math.uchicago.edu/~may/VIGRE/VIGRE2008/REUPapers/Yang.pdf for instance.
Comment: I am sorry, but that paper has no relation between Pythagorean triples and Pell's equations. I know how to solve a Pell's equation using simple continuous fractions but I wonder if there is a shortcut using Pythagorean triples.
Comment: The Pythagorean triples are found by generating the rational points on the curve $x^2+y^2 =1$.  You can put a $D$ in there and do the same thing, if you like.
Answer: Pythagorean triples correspond to the special case of rational solutions for $D=-1$ and relate to rational points on the unit circle.
If $X^2+Y^2=1$ then $(X^2-Y^2)^2+(2XY)^2=(X^2+Y^2)^2=1$ and $X^2-Y^2$ and $2XY$ give another solution.
The Pell equation is looking for integer points on $X^2-DY^2=1$. Once one solution is found note that $(X^2+DY^2)^2-D(2XY)^2=(X^2-DY^2)^2=1$ so that $(X^2+DY^2)$ and $2XY$ make a further solution.
Note that there are finitely many integer points on the unit circle - $(0, \pm 1)$ and $(\pm 1, 0)$ while the group of rational points is not finite and not finitely generated (geometrically there are rational points at arbitrarily small angles with the real axis).
On the other hand for Pell's equation, which is looking for integer points, and with positive $D$ (negative $D$ is trivial), the group generated by the solutions is finitely generated (though an infinite abelian group: if solutions exist - we can always generate more as above). The existence of a fundamental unit from which all solutions can be derived is the interesting result in this case - and that is the productive approach.
You know, as you comment, the continued fraction methods here. If you read the literature you will see that fundamental solutions can be surprisingly large in relation to $D$. It would be a real surprise if something as simple as deriving solutions from Pythagoras would work as a general procedure - how would you account for this phenomenon?

Just to note, on the new solutions from old theme:
If $X^2-DY^2=1$ and $W^2-DZ^2=1$ then we can consider $$(X+Y\sqrt D)(W+Z\sqrt D)=(XW+DYZ)+(XZ+YW)\sqrt D$$ and we find that $$(XW+DYZ)^2-D(XZ+YW)^2=(X^2-DY^2)(W^2-DZ^2)=1$$
This works in the Pythagorean case for $D=-1$
Comment: I know that for the negative Pell's equations to be soluble, then a Primitive Pythagorean triple satisfying a certain criterion should exist as given in this paper \\

A. Grytczuk, F. Luca and M. Wójtowicz, "The Negative Pell Equation and Pythagorean Triples".

So I was thinking maybe something similar for normal Pell's equation exist. I agree that for certain values of $D$, the solutions can be very large like $D=97$.
Comment: @WilburWestwood Always useful if you have references to cite them in the actual question - the first theorem is that the Pell equation $X^2-DY^2=1$ is solvable ifff there is a primitive Pythagorean Triple $A^2+B^2=C^2$ and positive integers $a$ and $b$ such that $D=a^2+b^2$ and $|aA-bB|=1$. Then (second theorem) $X=|aA+bB|, Y=C$ is a solution of the Pell equation. So this enables them to build Pell solutions from primitive Pythagorean Triples - but they don't show how to hit a specified value of $D$. If this is your question, please sate it more fully.
Comment: I understand the need of references. But I found it after I posted the question. So it was not in the original Q. I will emend it now. Also it is for the Pell equation of the form $X^2-DY^2=-1$ not the normal Pell's equation.
Comment: @WilburWestwood The $-1$ version of the Pell's equation is not so different. There is still a "fundamental unit" for which odd powers give $-1$ solutions and even powers give $+1$ solutions - if you look at the identities I gave you will see how two $-1$ solutions (possibly he same) can be combined to give a $+1$ solution.