How can I serve a "410 Gone" HTTP status AND redirect my removed content?

Question: I have a load of content that is gone (don't ask). I would like to tell Google and other search engines that this resource is gone, please don't try to index it. This is what HTTP status code 410 looks good for.
This is great for search robots, but if it's a real person accessing the page, I'd rather they were redirected to the new content.
Luckily, all the gone URLs match a pattern.
Is it possible to have all those handled by issueing a 410, and a redirect of some sort?
(I worry that permanent redirects might affect SEO too?)
Put another way: what I want to say is "That content is gone, you should not link to it. There's new content here:..."
[EDIT: to express why this is Drupal related]
Drupal is my CMS. With any generic website I would find my own solution to present a page and the correct header. With Drupal there's usually "a module for that", sometimes an elegant way to solve a common problem, and sometimes a hideously inefficient way to square a circle known as "the Drupal Way" [joke].
Drupal has an in-built system for 404s that works great. You can set a page, and because it's got a url you can use the block module to put other useful stuff on it, e.g. a View.
So there's 2 questions:

Does Drupal (or a reliably contrib'ed module) have a way to issue 410s akin to the 404 system?
Does Drupal have a 302/301 redirect system that can work with path patterns (i.e. separate from node as these dead pages were from another module that has been removed)
Comment: Related: [How to deal with deleted posts: Returning 410 status and preventing path re-use](http://drupal.stackexchange.com/q/58192/16495).
Comment: This is really close to being off-topic, unless it is asking about a Drupal solution for generating 410's and/or a proper way to implement a 410 w/ redirect solution in Drupal.
Comment: @MPD if it's off-topic here, can't it be migrated? Might be OK on Stack Overflow.
Comment: @Mołot This question can be easily salvaged with some editing.
Comment: @mpd I've edited and requested re-open. Can you click reopen, too? (not sure how that process works)
Comment: @Molot, thanks, I'd read that, but as I say in my edit, the content was not nodes.
Comment: That's why I'm not indicating it's a duplicate. Sadly all ready-made (or close) solutions i see are for nodes.
Answer: You can always output HTTP 410 header and redirect human users using JavaScript in a body:
var delay = 10000; //Your delay in milliseconds
setTimeout(function(){ window.location = URL; }, delay);

JavaScript from this answer.
Note that by rfc2616:

The 4xx class of status code is intended for cases in which the client seems to have erred. Except when responding to a HEAD request, the server SHOULD include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. These status codes are applicable to any request method. User agents SHOULD display any included entity to the user.

Emphasis mine. So, JavaScript outputted inside HTML body should work. But of course it is not guaranteed it will.
Your content is not nodes, so you probably have implemented hook_menu. If not, you can alter existing implementation with hook_menu_alter. What you need is to:

Using access callback and/or access arguments, ensure user gets the access permission to deleted content. If not, he will see 403 or 404, as this is Drupal's default behaviour for unpublished / deleted data.
In page callback test if content is valid, deleted or never existed.

If valid, default to the callback you are using now.
If deleted, use drupal_add_http_header to add "410 gone" header as appropriate. Output nice human readable page (optionally with JavaScript redirect, search box or anything appropriate).
If never existed, either default to what you were using already, or implement 404 the way you implemented 410 above.

Google says:

We recommend that you always return a 404 (Not found) or a 410 (Gone) response code in response to a request for a non-existing page.
Comment: @artfulrobot Well, you can just output page with message, but it will be displayed under "gone" address. As far as I know (it was never officially verified) redirects with delay does not affect ranking in any way, works just like simply outputting page. On the other hand, nice message and "see this instead" link can be even more user friendly.
Comment: Thanks. So I'd set up a custom page in a module with a menu that matches this pattern, use `header` 410, and output text for the user. I don't want to us JS for the redirect because I'm not sure how that affects Google (and other SEO). There will be 10s or possibly 100 URLs that end up here...
Comment: 410 and 404 errors can potentially hurt a site.  Google Webmaster tools will warn you about this.  Assuming you have a replacement URL, a 301 is a better option in the long run, and there are Drupal ways to handle them.
Comment: @MPD On the other hand, sudden redirection from many different content to one place can result in a ban, too. That's what black SEO guys was doing to suddenly boost some pages. Only safe thing not to loose content, but that's out of the question now, right? At the very least 410 is honest. Damage is done already and the only choice is *how* will it hurt, not *if*. Seems OP have decided, so I'm doing my best to help him with his decision.
Comment: @MPD Google says: ["We recommend that you always return a 404 (Not found) or a 410 (Gone) response code in response to a request for a non-existing page."](https://support.google.com/webmasters/answer/181708?hl=en&ref_topic=1724951)
Answer: Here's a more Drupal-specific answer.
410 doesn't seem appropriate to me.
I would simply have the old URLs issue HTTP 301 redirects, which you could do with the Redirect module.
Google and other search engines would then remove the old addresses and pick up the new ones (eventually, depending on how often your site is indexed). You could even submit URL removal requests to Google for each old address if you want to try expediting the process.
If you don't want the new addresses to be indexed at all, then the new pages should use a robots 'noindex' <meta> tag, which you could do with the Metatag module.
Comment: 301 means "moved permanently". It's inappropriate if new content is not the same (logically) as the old, now gone one. So it's not OK in this case.
Comment: @Mołot If the OP thinks a redirect is appropriate, then a 301 (or even a 302) is the proper solution.  It also aligns closer with Google's recommendations.  I had a fairly answer written about this, but couldn't post it before it got put on hold.
Comment: Well, Google recommendation I linked under my answer says exactly opposite than MPD. And decision *if* is SEO thing. Drupal thing is *how* to do *what OP asks to*.
Comment: For time being I've gone with 301, I'll keep eye on Google WMT to see what it complains about (it seemed v unhappy about all the 404s). The Redirect module looks really useful, thank you.
Answer: We just did a major redesign that include a new URL structure.  We decided to to use the Redirect module as part of the plan.
One of its features is to go through the {watchdog} table, and look for Page Not Found errors (ie, 404s).  It then provides a UI for adding a redirect to another URL.  You also have the option for setting the status code.  It will default to using a 301.  From the definition:

10.3.2 301 Moved Permanently
The requested resource has been assigned a new permanent URI and any
  future references to this resource SHOULD use one of the returned
  URIs. Clients with link editing capabilities ought to automatically
  re-link references to the Request-URI to one or more of the new
  references returned by the server, where possible. This response is
  cacheable unless indicated otherwise.

According to Google, a 301 is the proper code to use when you want the user to goto a new page:

If you're moving that content to a new URL, you should 301 redirect
  the old URL to the new URL-that way when users come to the old URL
  looking for that content, they'll be automatically redirected to
  something relevant to what they were looking for.

and a 410 should be used for content that is gone forever:

If you're getting rid of that content entirely and don't have anything
  on your site that would fill the same user need, then the old URL
  should return a 404 or 410. Currently Google treats 410s (Gone) the
  same as 404s (Not found).

The Redirect module does not handle 410 generation, though.  Otherwise, it has been a good way to handle the new scheme w/o needing to manually generate Apache rules.
Comment: I had this written, but was unable to post it before it got five close votes.
Comment: Thanks, this looks like the right way to go. I'll look into that module.
Answer: I've thought a lot about using 410 (gone) for pages that we don't need. Remember backlinks are like money you never want to lose them. So only under extreme conditions should we use 410 response for gone pages.
If you want Google to not to index them and delete them all from their index exclude them in the Robots.txt( here).
So better redirect all of them using 301 to one common "Moved" page and let Google know about it( and not block using robots.txt etc). If search engines know about 301 moved then your backlink worth will be maintained.
410 response is too risky to use since once you've sent a 410 response to Google etc, if pathauto creates the same alias for another page then Google will simply ignore it. I don't think this type of error can be debugged using WMT.
I think this question is more suitable to webmasters.stackexchange.com


How to filter products according to price with different currencies?

Question: I have a products table which has all the necessary fields for any table that describes a product. I can add a product to this table with any price and currency e.g.:
name    , Price, Currency
football, 10   , US dollar
jacket  , 10   , Euros
shirt   , 20   , Pounds
watch   , 30   , Euros
pants   , 35   , Pounds

As you can see each product is added with its price and the currency that it was set to. my problem comes when I want to do a filter using PHP or Mysql directly (whatever works) for products that are in the price range of e.g. 5 to 20 dollars or 20 to 35 Pounds
something like this form:
From:
<input name="price_from" type="text" />
To:
<input name="price_to" type="text" />
Currency:
<select class="element select medium" id="element_3" name="currency">
  <option selected="selected"></option>
  <option value="1">Dollar</option>
  <option value="2">Pound</option>
  <option value="3">Euro</option>
</select>

And this will search all products regardless of the currency it is set to. How can i do that? i thought about converting the currency when the user saves the product but the problem with that is currency exchange rate keeps on changing endlessly. and selecting all the products then calculate the exchange rate then filtering depending the the rage that was given is not a good solution, especially when you have a very big database of products.
Comment: The "US Euro", the new global über-currency? ;P
Comment: how to use an online currency exchange api ?

if you can I ll post an answer for that
Answer: One idea could be to calculate the currency exchange rate at the moment you do the query. For example, you  want to select all items between 1 US Dollar and 2 US Dollar.  Use a web service to calculate those values in other currencies. That way you will have something like:
Range:

1 and 2 US Dollar
0.4 and 0.7 Pounds
3 and 5 US Euro

And use those values to query the table.
Comment: This is what I was thinking. Convert the values when querying, then sort, then print into a table.
Answer: i just realize now (me also wanted same thing to do) create a new column in price table and update it with a currency constant -- keeping in mind one primary currency (euro in these case) example:
lets estimate dollar as 1.04 of a Euro

price      currency      sort_price       sort_currency ( just to be
more clearly)
10           USD            10.4              EUR
12           Eur            12                EUR

and now we can sort easy by column sort_price all in eur
Next make 2 update command for mysql to update USD price in column sort_price and for EUR also.
Or can make a page in administration page to update sort_price by click of a button, and need php cod there to be made.
code:

if (isset($_POST['sort_price_act'])) {
        $price_new_sdf ='';
    $price_ron_select_query ="SELECT price, currency, id FROM table ORDER BY id ASC";

    $price_ron_select = mysql_query($price_ron_select_query) or die("PPPPPPP".mysql_error());

    while ($price_ssss = mysql_fetch_array($price_ron_select, MYSQL_BOTH)) {

        if($price_ssss['currency'] == 'RON'){

        $price_new_sdf = round($price_ssss['price'] / 4.81,2);

        mysql_query("UPDATE produse SET sort_price = '".$price_new_sdf."' WHERE id = '".$price_ssss['id']."'") or die(mysql_error());

        }
        if($price_ssss['currency'] == 'EUR'){

        $price_new_sdf = $price_ssss['price'];

        //echo $price_new_sdf.' EUR + id = '.$price_ssss['id'].'
';

        mysql_query("UPDATE produse SET sort_price= '".$price_new_sdf."' WHERE id = '".$price_ssss['id']."'") or die(mysql_error());

        //$sort_priceuri = mysql_query($sort_priceuri_query) or die("VVVVVV".mysql_error());
        }
    }

}

          <form action="<?php echo $_SERVER['PHP_SELF']; ?>" method="post" onSubmit="return confirm ('ACTUALIZE sort_price in EUR ?')">
        <input type="submit" name="sort_price_act" value="SORT NEW PRICE">
        </form>

ALl price updated works like a charm :)


Limit of upper and lower Riemann sum are the same

Question:
Let $f:[a,b]\to\mathbb{R}$ be continuous. Consider, for all $n\in\mathbb{N}$, the homogeneous partition $P_n$ of order $n$. Prove that $\lim_{n\to\infty}L(f, P_n) = \int_a^b f=\lim_{n\to\infty}U(f, P_n)$.

So what I have tried is that since $f$ is continuous, then it's integrable. Meaning,
$$L(f, P_n)\leq sup(\{L(f, P)\mid P\in\mathbb{P}_{[a,b]}\})  = \int_a^b f= inf(\{U(f, P)\mid P\in\mathbb{P}_{[a,b]}\})\leq U(f, P_n).$$
Thus, now we must show that $\lim_{n\to\infty}U(f, P_n) = \lim_{n\to\infty}L(f, P_n)$. Indeed,
\begin{align*}
        \lim_{n\to\infty}L(f, P_n)
        &= \lim_{n\to\infty}\sum_{k=1}^n m_k(f)\lambda([t_{k-1}, t_k]) \\
        &= \lim_{n\to\infty}\sum_{k=1}^n \left(m_k(f)\cdot\frac{b-a}{n}\right) \\
        &= (b-a)\sum_{k=1}^n \left(\lim_{n\to\infty}\frac{m_k(f)}{n}\right)
    \end{align*}
where $m_k(f) = inf\{f(x)\mid x\in[t_{k-1}, t_k]\}$ and $M_k(f) = sup\{f(x)\mid x\in[t_{k-1}, t_k]\}$. But from here, how do we get to the upper sums? I.e., how does
$$\lim_{n\to\infty}m_k(f) = \lim_{n\to\infty}M_k(f)?$$
Comment: What is $m_k(f), M_k(f)$? If these are $f$ evaluated at the left and right endpoints of your partition, then by uniform continuity of $f$ on closed, bounded intervals, these should be sufficiently close to each other when the partition is small enough, for all x.
Comment: @AryanDugar I already added what those mean.
Comment: OK, then the argument in my previous comment should work. Note that since $[t_{k-1},t_k]$ are compact intervals, there will be points $x_1, x_2$ in each interval such that $f(x_1) = m_k(f), f(x_2) = M_k(f)$. By the argument in my previous comment, $M_k(f)$ and $m_k(f)$ will be sufficiently close whenever your interval is small enough (so that $x_1, x_2$ are sufficiently close). Hope that helps!
Comment: @AryanDugar I get only the first thing you said, however, intuitively I understand that both should be so close when $n\to\infty$ but rigorously, however, how does one prove that? I think I'm making a dumb mistake or missing something, but right now I can't see it.
Answer: Consider an arbitrary interval given by your partition, $I_k = [t_{k-1},t_k]$. Now, let $\frac{b-a}{n} = \Delta(x)$. Now choose your favorite $\epsilon > 0$, and note by uniform continuity of $f$ on $I_k$ that there is some $\delta > 0$ such that $|x - y| < \delta $ implies $|f(x) -f(y)| < \epsilon$.
Now, since $n \rightarrow \infty$, there is some rank $N$ such that $m \geq N \rightarrow \Delta(x) = \frac{b-a}{m} < \delta$, since $\delta$ is fixed. But then, consider $I_k$: note that for all $x,y \in I_k, |x - y| \leq \Delta(x) < \delta$. However, this implies that for all $x,y \in I_k, |f(x) - f(y)| < \epsilon$. Now, from what I initially commented, $m_k(x) = f(x_1^k), M_k(x) = f(x_2^k)$ for some $x_1^k, x_2^k \in I_k$. Thus, specialising our finding, $|f(x_1^k) - f(x_2^k)| < \epsilon$. This gives us our claim.
Comment: Thanks. I really appreciate it! Now I see what I was missing.


BottomNavigationView does not hide on scrolling inside fragment, if fragment contains AppBarLayout

Question: I'm having a lot of problems with scrolling and detection of it inside fragments using coordinatorLayout in Android. I have a "MainActivity" that contains 3 fragments. In one of those fragments I have an Appbar that collapses when fragment is scrolled. I've managed to do that, but if I set scrolling behavior to allow that, my bottomNavigationView (which is found in mainactivity.xml) does NOT react to scrolling. Codes go something like this:
Fragment1.xml
<android.support.design.widget.CoordinatorLayout
...
...>
    <android.support.design.widget.AppBarLayout
 ...
 ...>
        <android.support.design.widget.CollapsingToolbarLayout
                 app:layout_scrollFlags="scroll|exitUntilCollapsed"
   ...>
            <android.support.v7.widget.Toolbar
     .../>
        </android.support.design.widget.CollapsingToolbarLayout>
    </android.support.design.widget.AppBarLayout>

   <android.support.v4.widget.NestedScrollView
        app:layout_behavior="@string/appbar_scrolling_view_behavior">
   </android.support.v4.widget.NestedScrollView>

</android.support.design.widget.CoordinatorLayout>

MainActivity.xml
<android.support.design.widget.CoordinatorLayout
.../>
    <FrameLayout
        app:layout_behavior="@string/appbar_scrolling_view_behavior"
 .../>
    <android.support.design.widget.BottomNavigationView
      app:layout_behavior="@string/hide_bottom_view_on_scroll_behavior"
.../>
</android.support.design.widget.CoordinatorLayout>

Now, this works well enough, in the sense that when I scroll inside fragment my AppBar collapses into the title (which is what I want), BUT my bottomNavigationView does not react to scrolling.
What I found out is that if i add the line
app:layout_behavior="@string/appbar_scrolling_view_behavior"

to the AppBarLayout xml declaration I get the bottomView to collapse on scroll event (when I scroll up it shows again). So basically either I have the ability to collapse appbar inside fragment OR I have the ability to hide BottomNavigationView when I detect a scroll event inside the fragment.
Any sort of help would be appreciated.
Comment: Does provided answer works for you, if not what is the issue?
Comment: does the things work as expected when there is no app bar layout?
Comment: Yes it works completely fine without app bar layout. Basic problem is that scroll flag is used in app bar layout. It is consuming scroll event required for bottom navigation behaviour.
Answer: In order to hide bottom navigation in reaction to scrolling inside the fragement
modify onCreateView method on fragment1.kt as follows
override fun onCreateView(
        inflater: LayoutInflater, container: ViewGroup?,
        savedInstanceState: Bundle?
    ): View? {

        ...

        // access the bottom nav which is on the main activity
        val bottomNav: BottomNavigationView = activity!!.findViewById(R.id.bottom_nav)

        // hide bottom nav when scrolling
        rvList.addOnScrollListener(object : RecyclerView.OnScrollListener() {

            override fun onScrolled(recyclerView: RecyclerView, dx: Int, dy: Int) {
                if (dy > 0 || dy < 0) {
                    bottomNav.visibility = View.GONE
                }
            }

            override fun onScrollStateChanged(recyclerView: RecyclerView, newState: Int) {
                if (newState == RecyclerView.SCROLL_STATE_IDLE) {
                    bottomNav.visibility = View.VISIBLE
                }

                super.onScrollStateChanged(recyclerView, newState)
            }
        })

        return view
}

In Java
@Override
public View onCreateView(LayoutInflater inflater, ViewGroup container,
        Bundle savedInstanceState) {
        ...

    // access the bottom nav which is on the main activity
    BottomNavigationView bottomNav = getActivity.findViewById(R.id.bottom_nav);

    // hide bottom nav when scrolling
    recyclerView.addOnScrollListener(new RecyclerView.OnScrollListener()
    {
        @Override
        public void onScrolled(RecyclerView recyclerView, int dx, int dy)
        {
             if (dy > 0 ||dy<0)
             {
                 bottomNav.setVisibility(View.GONE);
             }
        }

        @Override
        public void onScrollStateChanged(RecyclerView recyclerView, int newState)
        {
             if (newState == RecyclerView.SCROLL_STATE_IDLE)
             {
                  bottomNav.setVisibility(View.VISIBLE);
             }

             super.onScrollStateChanged(recyclerView, newState);
        }
    });

        return view;
 }

Update:
Extract the listener code to a separate function as given below (in Kotlin)
   public fun hideBottomNav(
        rvList: RecyclerView,
        bottomNav: BottomNavigationView
    ) {
        rvList.addOnScrollListener(object : RecyclerView.OnScrollListener() {

            override fun onScrolled(recyclerView: RecyclerView, dx: Int, dy: Int) {
                if (dy > 0 || dy < 0) {
                    bottomNav.visibility = View.GONE
                }
            }

            override fun onScrollStateChanged(recyclerView: RecyclerView, newState: Int) {
                if (newState == RecyclerView.SCROLL_STATE_IDLE) {
                    bottomNav.visibility = View.VISIBLE
                }

                super.onScrollStateChanged(recyclerView, newState)
            }
        })
    }

Comment: Thanks for the solution. It works but I think there must be some better solution for this problem. I need this behaviour in all of my 5 fragments of activity, then I have to apply this listener to all recycler views of 5 fragments. It would be great if there is some flag or anything else such that bottomnavigation is able to get scroll events (even if fragment uses app bar layout with scroll flags) as behaviour to hide/show bottomNav is already defined (just need to get scroll events directly). This functionality is present in Cricbuzz app, I wonder how they have done it
Comment: I think you may be able to extract the code to a separate function.
Comment: have you tried moving listener code to separate fn?
Answer: Update your library to 28.0.0 or above or the androidx / material-1.0.0 and change your BottomNavigationView like the below-
<BottomNavigationView
        android:id="@+id/navigation"
        .
        .
        .
        .
        app:layout_behavior="@string/hide_bottom_view_on_scroll_behavior" />

hope it will be helpful.


RC Children routes aren't resolving getting blank page

Question: In my app.module.ts I have:
<code>import {settingsRouting} from './../../pages/settings/settings_routing';

const appRoutes: Routes = [
  { path: '', component: HomePage }
];
export const appRoutingProviders: any[] = [];
export const routing: ModuleWithProviders = RouterModule.forRoot(appRoutes);
</code>
And in my settings_routing.ts I have:
<code>    const settingsRoutes: Routes = [
      {
        path: 'settings',
        component: SettingsPage,
        children: [
          {
            path: '',
            children: [
              { path: 'account', component: AccountSettings },
              { path: 'profile', component: ProfileSettings }
            ]
          }
        ]
      }
    ];

export const settingsRouting: ModuleWithProviders = RouterModule.forChild(settingsRoutes);
</code>
I'm getting a blank page when trying '/settings/account' and '/settings/profile', how can I fix this?
NB: I still want to show stuff from my master settingspage. I want it to have a  for it's children
Comment: Try rearranging your path something like this: `{ path: 'settings', children: [ { path: '', component: SettingsPage}, { path: 'account', component: AccountSettings }, { path: 'profile', component: ProfileSettings } ] }`
Comment: No luck, not working :(
Comment: anything in the console?
Comment: Nop. Actually I've moved the settings route config into the main app.module and now '/settings' is resolving' but not it's children...
Answer: Turns out, SystemJS started requesting files from '/profile/node_modules/' and it was failing silently.
Ensure you have 'baseURL: '/' in your systemJS config.


Why is regression about variance?

Question: I am reading this note.
On page 2, it states:

"How much of the variance in the data is explained by a given regression model?"
"Regression interpretation is about the mean of the coefficients; inference is about their variance."

I have read about such statements numerous times, why would we care about "how much of the variance in the data is explained by the given regression model?"... more specifically, why "variance"?
Comment: "[V]ariance" as opposed to what, the standard deviation? What is it that you think we ought to care about in regression? What are your typical goals in building a regression model?
Comment: The variance has different units than the quantity being modeled, so I've always found it hard to interpret the "proportion of variance explained by the model".
Answer: why would we care about "how much of the variance in the data is explained by the given regression model?"
To answer this it is useful to think about exactly what it means for a certain percentage of the variance to be explained by the regression model.
Let $Y_{1}, ..., Y_{n}$ be the outcome variable. The usual sample variance of the dependent variable in a regression model is $$ \frac{1}{n-1} \sum_{i=1}^{n} (Y_i - \overline{Y})^2 $$ Now let $\widehat{Y}_i \equiv \widehat{f}({\boldsymbol X}_i)$ be the prediction of $Y_i$ based on a least squares linear regression model  with  predictor values ${\boldsymbol X}_i$. As proven here, this variance above can be partitioned as:
$$ \frac{1}{n-1} \sum_{i=1}^{n} (Y_i - \overline{Y})^2 =
\underbrace{\frac{1}{n-1} \sum_{i=1}^{n} (Y_i - \widehat{Y}_i)^2}_{{\rm residual \ variance}} + \underbrace{\frac{1}{n-1} \sum_{i=1}^{n} (\widehat{Y}_i - \overline{Y})^2}_{{\rm explained \ variance}}
 $$
In least squares regression, the average of the predicted values is $\overline{Y}$, therefore the total variance is equal to the averaged squared difference between the
observed and the predicted values (residual variance) plus the sample variance of the predictions themselves (explained variance), which are only a function of the ${\boldsymbol X}$s. Therefore the "explained" variance may be thought of as the variance in $Y_i$ that is attributable to variation in ${\boldsymbol X}_i$. The proportion of the variance in $Y_i$ that is "explained" (i.e. the proportion of variation in $Y_i$ that is attributable to variation in ${\boldsymbol X}_i$) is sometimes referred to as $R^2$.
Now we use two extreme examples make it clear why this variance decomposition is important:

(1) The predictors have nothing to do with the responses. In that case, the best unbiased predictor (in the least squares sense) for $Y_i$ is $\widehat{Y}_i = \overline{Y}$. Therefore the total variance in $Y_i$ is just equal to the residual variance and is unrelated to the variance in the predictors ${\boldsymbol X}_i$.
(2) The predictors are perfectly linearly related to the predictors. In that case, the predictions are exactly correct and $\widehat{Y}_i = Y_i$. Therefore there is no residual variance and all of the variance in the outcome is the variance in the predictions themselves, which are only a function of the predictors. Therefore all of the variance in the outcome is simply due to variance in the predictors ${\boldsymbol X}_i$.

Situations with real data will often lie between the two extremes, as will the proportion of variance that can be attributed to these two sources. The more "explained variance" there is - i.e.  the more of the variation in $Y_i$ that is due to variation in ${\boldsymbol X}_i$ - the better the predictions $\widehat{Y}_{i}$ are performing (i.e. the smaller the "residual variance" is), which is another way of saying that the least squares model fits well.
Comment: @MichaelChernick, yes but in least squares regression (which I think the OP is talking about based on the linked slides), the mean of the predicted values equals the mean of the $Y$s, so you can just call it the sample variance of the predictions.
Comment: Macro, my point was that this decomposition occurs only if $\langle \mathbf y - \hat {\mathbf y}, \hat{\mathbf{y}} - \bar{y} \mathbf{1} \rangle = 0$ and so the "regression" inherently involves an orthogonal projection onto a space containing the constant vector. Note that we can easily "break" this decomposition by simply removing the constant vector from our model, which seems in conflict with your most recent comment.
Comment: @Macro, I think this is quite good, but I have 2 questions. (1) In your top equation, called "sample variance" you divide by $n$, shouldn't that be $n-1$? (2) I always understood $R^2$ to be related to the decomposition of the *sum of squares* (eg, that's what the linked proof is discussing), but you appear to be working w/ *mean squares* in your second equation, does this equality hold in general w/ mean squares?
Comment: @gung, thanks for the comments. (1) What I've written there is the maximum likelihood estimator of the variance. You may be right that the convention is that 'sample variance' refers to the unbiased estimator. If I replace all of the $n$s with $(n-1)$s, nothing would change. (2) All I've done is divide through the equality by $n$ so that the decomposition can be thought of in terms of decomposing variance rather than sums of squares.
Comment: @gung, re: (1) note that is the MLE of the variance in the normal case. In general, though, dividing through by $n$ rather than $n-1$ gives a smaller MSE in estimating the variance (whether or not MSE is the proper measure of performance is another question entirely).
Comment: @Macro, I can see you're right. I was thrown off by the last sentence in the paragraph after the 2nd equation. At that point in the standard lecture that this answer emulates, I note that $R^2=SS_{reg}/SS_{tot}$, & reading quickly, I was left w/ the impression you were saying $R^2=MS_{reg}/MS_{tot}$, which students always assume & which you need to explicitly point out is *not* true. I'm also used to seeing / stressing that *sample* variance divides by $n-1$. The situation w/ the OP, or other future readers, may be similar. I wonder if a clarifying note or word of warning is in order?
Comment: @gung, I'm not totally clear what you're defining as $MS_{reg}$. If it is $MS_{reg} = SS_{reg}/n$ (and analogously for $tot$) then it seems that identity is true, since you're just multiplying by $n/n=1$. Can you clarify?
Comment: @Macro, when you make an ANOVA table for (eg) a simple regression model, you have 3 rows: reg, res, & tot; & 5 columns: SS, df, MS, F, & p. SS is calculated as your equation w/o dividing, df is 1 for each regressor ($N-1-p$ for res, etc), MS is $SS/df$, F is $MS_{reg}/MS_{res}$, p is $1-cdf(F_{df_{reg},df_{res}})$. Students always assume that $R^2=MS_{reg}/MS_{tot}$, since we say that $R^2$ is the % of *variance* & MS's are variances of a sort, but you have to stress that $R^2=SS_{reg}/SS_{tot}$ *only*. I think the reason you're OK is because $n\neq df$.
Comment: Macro, I do not understand your most recent edit: *Therefore the "explained" variance may be thought of as the proportion of variance in $Y_i$ that is attributable to variation in $X_i$, sometimes referred to as $R^2$.* The explained variance, as you have denoted it in display equation is neither a proportion nor sometimes referred to as $R^2$.
Comment: @cardinal, some words got lost in that edit - I've fixed it. Thanks. (btw, the way it was phrased before that was technically incorrect also, so an edit was needed).
Comment: @gung, Thanks for clarifying. In this case, the mention of $R^2$ was only meant to place the variance decomposition in a familiar context but I think that going into a discussion of what $R^2$ _isn't_ may be an unnecessary digression - I appreciate the comments though.
Comment: 3 of my top answers un-upvoted within seconds.. To this very mature person: If your goal is to punish me, you're wasting your time- I really don't care. My reputation here has been established by positive contributions to the site in various capacities (giving statistical advice, editing posts/tag wikis, taking out the trash with flags/close votes), not by points. All you accomplish by this is lowering the answer score, potentially taking attention away from authoritative answers that could be useful to others. This can't possibly be a good use of your time and certainly doesn't help the site.
Comment: This is like my answer but perhaps a little bit better explained. Also I see a possible critque that could have been mention is that  I should have written the variation relative to the mean of Y.
Comment: I made the edit to my answer because Yb is needed for the variance decomposition to work properly.
Comment: Yes it was clear to me that she was referring to least squares regression.  Still a lot of what you wrote is just repeating what I said slightly differently.  I still gave you a +1.
Comment: Did you mean "(e.g., least squares)" or "(*i.e.*, least squares)"? The former gives the, perhaps somewhat mistaken, impression that least-squares is but one of many approaches that leads to such a variance decomposition.
Comment: @cardinal, the decomposition of the variance into those two parts is still possible without using least squares regression but most of the subsequent interpretation I give only applies when $\widehat{Y}_i$ is a least squares prediction (since $\overline{Y}$ will not necessarily be the mean of the predictions in general models), so that's why I called it an "e.g."
Comment: @cardinal, you're right. For some reason I had the idea that this decomposition makes sense in greater generality but after looking again at its proof I can see that it doesn't hold in general. I've edited a bit to make it clear I'm talking about least squares regression.
Comment: It is not unusual to still consider $1 - \|\mathbf y - \hat{\mathbf y}\|^2/\|\mathbf y - \bar y \mathbf 1\|^2$ as a measure of variation explained in more general models (e.g., [here](http://stats.stackexchange.com/q/7357/2970)), even though it doesn't have quite as nice of an interpretation as in this case. :)
Answer: I can't run with the big dogs of statistics who have answered before me, and perhaps my thinking is naive, but I look at it this way...
Imagine you're in a car and you're going down the road and turning the wheel left and right and pressing the gas pedal and the brakes frantically. Yet the car is moving along smoothly, unaffected by your actions. You'd immediately suspect that you weren't in a real car, and perhaps if we looked closely we'd determine that you're on a ride in Disney World. (If you were in a real car, you would be in mortal danger, but let's not go there.)
On the other hand, if you were driving down the road in a car and turning the wheel just slightly left or right immediately resulted in the car moving, taping the brakes resulted in a strong deceleration, while pressing the gas pedal threw you back into the seat. You might suspect that you were in a high-performance sports car.
In general, you probably experience something between those two extremes. The degree to which your inputs (steering, brakes, gas) directly affect the car's motion gives you a clue as to the quality of the car. That is, the more of your car's variance in motion that is related to your actions the better the car, and the more that the car moves independently of your control the worse the car is.
In a similar manner, you're talking about creating a model for some data (let's call this data $y$), based on some other sets of data (let's call them $x_1, x_2, ..., x_i$). If $y$ doesn't vary, it's like a car that's not moving and there's really no point in discussing if the car (model) works well or not, so we'll assume $y$ does vary.
Just like the car, a good-quality model will have a good relationship between the results $y$ varying and the inputs $x_i$ varying. Unlike a car, the $x_i$ do not necessarily cause $y$ to change, but if the model is going to be useful the $x_i$ need to change in a close relationship to $y$. In other words, the $x_i$ explain much of the variance in $y$.
P.S. I wasn't able to come up with a Winnie The Pooh analogy, but I tried.
P.P.S. [EDIT:] Note that I'm addressing this particular question. Don't be confused into thinking that if you account for 100% of the variance your model will perform wonderfully. You also need to think about over-fitting, where your model is so flexible that it fits the training data very closely -- including its random quirks and oddities. To use the analogy, you want a car that has good steering and brakes, but you want it to work well out on the road, not just in the test track you're using.

Why is regression about variance?


Is $Y^2-X^3+X$ a variety?

Question: My question is whether $F=Y^2-X^3+X \in k[X,Y] $ ($k$ :algebraically closed) is an irreducible variety or not. It suffices to show that F is irreducible as we have a plane curve. Writing F as $F=(y+f(x))(y+g(x)) $ we can conclude that $f^2(x)=(x^2+1)x=(x+i)(x-i)x$. Doesn't that mean that $V(F)$ is irreducible? I am confused because if you see geometrically the curve you will see it's a non-connected shape:
Isn't every variety suppose to be connected? Topologically every irreducible set is a connected set so doesn't that mean the same geometrically?
Comment: This is an elliptic curve. It is connected in the Zariski topology.
Answer: A real subvariety (irreducible nonempty algebraic subset) of $\Bbb A^k(\Bbb R)$ may be disconnected in the "classical" topology, as this shows.
Nonetheless it is connected in the Zariski topology, which is much coarser.
The closed sets in the Zariski topology are those defined by algebraic equations.
If you split this curve into two distinct pieces, $C_1$ and $C_2$
as illustrated above, then $C_2$ is contained in the Zariski closure of $C_1$. This means that any polynomial vanishing on $C_1$ also vanishes on $C_2$.
Comment: It appears disconnected because you're only looking at the $\mathbb R$-valued points. If you look at this equation over $\mathbb C$ it looks like a torus with a point or two missing.
Comment: I do understand that Zariski topology is essentially different from the euclidean but still i would think that the variety above, being irreducible would mean that the 2 shapes intersect somehow. I thought for example that they do intersect on the imaginary plane, difficult though it may be to imagine If no such thing occurs, is there any geometric interpretation to what a closed irreducible set in zariski topology looks like?

Is $Y^2-X^3+X$ a variety?


node user authentication with cookie

Question: I am going to write a server api that responsible for user authentication. From my limit knowledge and understanding, when a user login with username and password, session will be created and auth cookie with username will save into cookie. When the user request for the next page, the user information in the cookie will send to server and server will recognize it. so my question is what if another person manually copy the existing cookie info and create the same cookie in a browser of another computer? Will it skip the login stage? Can anyone explain in details how to prevent this in details? thanks
Answer: Yes, it will most likely skip the login stage. What you describe is a form of session hijacking, or cookie hijacking. Using cookies over unencrypted connection (ie. HTTP instead of HTTPS) is not a secure solution because anyone can steal and use the same cookie and this is usually enough to get full access with no need to authenticate. (It can be - though it usually isn't - made harder to exploit but not impossible.)
Soon there will be no reason not to use HTTPS (see my answer to other question for details).
In addition to making sure that no one can read the cookie (using HTTPS and HttpOnly) you also have to make sure that no one can guess the session ID (eg. it cannot be a sequential or small number or anything like that).
See also:

Session hijacking on Wikipedia
Session hijacking on OWASP
Session fixation on Wikipedia
Session fixation on OWASP
Session riding (XSRF, CSRF)

node user authentication with cookie


How to determine post-treatment weight with uneven time points?

Question: I am looking for suggestions on how to deal with uneven time points and missing data in a single-group repeated-measures design.
BACKGROUND: A group of 220 subjects went through a 4-month weight loss treatment, and were given wireless scales to weigh themselves daily at home. The primary outcome is change in weight from pre- to post-treatment (for which I would normally use a paired samples t-test on).
HOW TO DETERMINE POST-TREATMENT WEIGHT?
We have pre-treatment weights for everyone, but post-treatment weight is hard to determine since not all the subjects weighed-in at the same time after treatment (and some dropped out altogether).
APPROACH A: Creating "Time Windows" to Determine Post-Treatment Weight.
I've tried creating arbitrary time windows (1, 2, 4, or 6 weeks around the treatment end date), and then taking the median weight to determine "post-treatment weight." I found (all p's < 0.01):
1 week window: $n = 124$, $-9.55$% weight loss
2-week window: $n = 143$, $-7.52$% weight loss
4-week window: $n = 161$, $-6.39$% weight loss
6-week window: $n = 168$, $-6.30$% weight loss
However, this is problematic because it excludes dropouts. Furthermore, the bigger the window, the bigger the "n", but smaller the effect size (because subjects who did not weigh in regularly also lost the least weight) and less valid the weight in terms of being measured at the same time between subjects.
APPROACH B: Intention-to-Treat with Last Observation Carried Forward.
Just using the last observation of all the subjects, up to 2 weeks post-treatment:
$n = 220$, $-4.22$% weight loss
However, this is problematic because the dropouts are depressing the effect size (though retains the largest "n").
APPROACH C: MIXED MODELS?
I've read that mixed models can handle uneven time points and missing data, though I'm not super familiar with this procedure, and am not sure what are the fixed and random are effects here. My attempts to run it in SPSS 21 caused an "insufficient memory" error.
APPROACH D: SOMETHING ELSE? (Multiple Imputation/Regression/ANCOVA, etc.)
Answer: Industry standard for weight loss trials is often last-observation-carried-forward, though (as you've found), there are limitations to that type of data analysis.  Mixed models are likely your best bet; I have run them in R with similar data though, and have not tried them with SPSS.
Random effects and fixed effects (though there is some debate about whether they should be called as such) address whether the particular parameter (e.g., intercept, slope) can vary by case (random) or not (fixed).  Gelman & Hill's Data Analysis Using Regression and Multilevel/Hierarchical Models provides a fantastic overview and introduction to these types of models, which do handle missing data and uneven time points, as well as R code.


How did we know we can factor the following polynomial?

Question: Here is the equation I have $x^4 - 2 x^3 - 3x^2 + 4x + 2 = 0,$ the book said we can factor it to obtain $(x^2 - 2)(x^2 - 2x -1) = 0$ but I do not know how is this true, the rational root test does not say that $\sqrt{2}$ is a root. Can someone explain to me which theorem say this please?
Comment: That Rational Root Theorem can't tell you the square root of two is a solution since square root of two is irrational.
Comment: Well, the quartic formula will tell you exaclty what the roots are. For reference: https://en.wikipedia.org/wiki/Quartic_equation
Comment: $\sqrt 2$ is irrational ...
Comment: Piita, TurlocTheRed, Digitallis: I believe that the OP knows that, whence his/her trouble about how to *guess* that factorization. @Emptymind: am I wrong? I.e. do you just want to be sure that this factorization is true?
Comment: @AnneBauval no that was not my problem ..... My problem was how to factorize it and why it is factorizable.
Comment: To know that it's true, just multiply the factors to check. To guess those factors starting from the polynomial is a different harder question.
Comment: If you "know" that it can be factored, write it as $(x^2 + ax + b)(x^2 + cx + d)$ and see if that yields any solution.
Comment: In practice one would simply check in some CAS such as Maple/Mathematica whether it is reducible. Verifying is just expanding. Of course it depends on context (this would not make sense in exam/exercise setting).
Answer: If the rational root theorem fails, one potential route is to "conclude" (perhaps falsely, but nonetheless proceed as if you have, since you might as well try something) that you can only factor into two "nice" quadratics. Since the original polynomial is monic, we can go ahead and say, for some $a,b,c,d$,
$$
x^4 - 2 x^3 - 3x^2 + 4x + 2 = (x^2 + ax + b)(x^2 + cx + d)
$$
Expand the right-hand side, and compare coefficients. You'll get a system of equations:
$$\begin{cases}
a+c &= -2 \\
ac+b+d &= -3 \\
ad+bc &= 4 \\
bd &= 2
\end{cases}$$
From here, you should try to solve the system of equations. Of course, this is easier said than done, as you have no reason to suspect, a priori, that $a,b,c,d$ are "nice". However, again assuming they are nice (say, integers) should lead to the desired conclusion in this case.
For instance, if $a,b,c,d$ are integers, then $(b,d) \in \{(1,2),(2,1),(-1,-2),(-2,-1)\}$, and you can try to solve the system from there. Or you could use the fact that $a+c=-2$ to derive a substitution and perhaps clean the system up a bit. However you prefer to solve, you should arrive at the factorization with some trial and error.

Other routes than assuming the factorization into quadratics may exist, of course. Such as:

You could try to see if this polynomial matches any familiar forms. This doesn't seem to, though, and this is a skill more honed by experience than anything. (In general, for instance, you might easily notice the binomial coefficients arising - not in this case though.)

The quartic formula, but this is painful. You could try to find roots via any other method, too, that might seem plausible: numerical methods like Newton's method, for example. (They can at least provide guesses for roots that are easily checked.) From there, you could divide by $x-(\text{that root})$ and get a perhaps-simpler cubic, or iterate that method.

More obscure tricks/tools of note may be Ferrari's method for quartics, or Descartes' rule of signs and Sturm's theorem for discerning the types or locations of roots, if you're interested.
Comment: Note that if you guess a value for one of a,b,c,d one can very quickly compute whether this leads to a solution for the other three variables. In particular guessing c=0 quickly leads to the correct factorization in this particular example.
Answer: Any polynomial can be reduced to the depressed form with a linear substitution. In this case, the substitution $\,x = z + \dfrac{1}{2}\,$ results in a simple biquadratic, which is straightforward to factor:
$$
\begin{align}
x^4 - 2 x^3 - 3x^2 + 4x + 2 &= \left(z+\frac{1}{2}\right)^4 - 2 \left(z+\frac{1}{2}\right)^3 - 3\left(z+\frac{1}{2}\right)^2 + 4\left(z+\frac{1}{2}\right) + 2
\\ &= z^4 - \frac{9}{2} z^2 + \frac{49}{16}
\\ &= \frac{1}{16} \left(4 z^2 - 9 - 4 \sqrt{2}\right) \left(4 z^2 - 9 + 4 \sqrt{2}\right)
\end{align}
$$
Recognizing that $\,9 \pm 4 \sqrt{2} = \left(2 \sqrt{2} \pm 1\right)^2\,$, each factor can be written as a difference of squares:
$$
\begin{align}
\frac{1}{16} \left(4 z^2 - 9 - 4 \sqrt{2}\right) \left(4 z^2 - 9 + 4 \sqrt{2}\right) &= \frac{1}{16}\left((2z)^2 - \left(2 \sqrt{2} + 1\right)^2\right)\left((2z)^2 - \left(2 \sqrt{2} - 1\right)^2\right)
\\ &= \frac{1}{16} \color{blue}{\left(2z-2\sqrt{2}-1\right)}\color{brown}{\left(2z+2\sqrt{2}+1\right)}
\\ &\qquad\;\;\; \color{brown}{\left(2z-2\sqrt{2}+1\right)}\color{blue}{\left(2z+2\sqrt{2}-1\right)}
\end{align}
$$
Substituting back $\,2z = 2x - 1\,$ and grouping the colored pairs produces the given factorization.
Comment: One should note that the substitution to create the depressed form eliminates the $x^3$ term and such a substitution always exists. In this particular example it so happens that this substitution also eliminates the $x$ term which makes the new polynomial easy to factorize.
Comment: @quarague Right, this won't work for factoring quartics in general. But reducing it to a depressed quartic is a natural first step, even if just attempting to solve it by traditional methods. In this case, it turned out to be a big shortcut.
Answer: I guess there's a trick specifically for seeking roots of integer-coefficient quartics of the form $\sqrt m$ where $m$ is a nonsquare rational number. Note that if $\sqrt m$ is a root of $ax^4+bx^3+cx^2+dx+e$, then it must separately be a root of both $ax^4+cx^2+e$ and $bx^3+dx$. (This is because
$$
a\sqrt m^4+b\sqrt m^3+c\sqrt m^2+d\sqrt m+e = (am^2+cm+e) + (bm+d)\sqrt m
$$
can equal $0$ only if both $am^2+cm+e=0$ and $bm+d=0$, since $\sqrt m$ is irrational.) In particular, the roots of $bx^3+dx$ (other than $x=0$) are $\pm\sqrt{-d/b}$, and so one can plug this into $ax^4+cx^2+e$ to see if it happens to be a root as well. For the polynomial in the OP, this works because $\pm\sqrt2$ are the nonzero roots of $-2x^3+4x$ and just happen to be roots of $x^4-3x^2+2$ as well.
To phrase this as a one-line test for such roots of integer-coefficient quartics:
$$
\text{If } bd\ne0 \text{ and } ad^2-cbd+eb^2=0 \text{, then } \pm\sqrt{-d/b} \text{ are roots of } ax^4+bx^3+cx^2+dx+e.
$$


Find a red rectangle on the game screen

Question: I know there's been many topics similar to this one but none of them had a precise answer to what I'm searching for so please if anybody knows, and also I'm doing it in C#.
You all probably know (FPS) games and on the game screen with resolution say 1024x768 i need to find a red rectangle(which is the enemy) and move the mouse to it.
So my main problem is to find that red rectangle. OK so here's what I've tried so far:
I've tried the AForge and ran out of memory:
<code>ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0);
TemplateMatch[] matchings = tm.ProcessImage(image1.Clone(r,
        System.Drawing.Imaging.PixelFormat.Format24bppRgb), image2);
</code>
I've used CopyfromScreen to create a image1 and image2 is a template I have.
I've tried the LockBits, so I can create the 2 dimensional code array for a bitmap and find the code for color red and  try to ID if it's a rectangle but the idea seems very complicated been stuck here for a 4 days now.
The web is full of info on this but the more I go into the more I get confused :(
Anyway please ppl help me out here:
Comment: Making an aimbot is highly unethical
Comment: I really have hard time believing that AForge ran out of memory. How are you using it?
Comment: @GETah i've tried 2 ways : 1. http://www.aforgenet.com/framework/features/template_matching.html on this link, by changing value of the ExhaustiveTemplateMatching(0.485f) i am able to find something, but when i put it higher it finds nothing, when i bring it down it finds everything just too much. Can't find the middle in between. 2. and second approach i've tried to put the TemplateMatch[] matchings = tm.ProcessImage(image1.Clone(r, System.Drawing.Imaging.PixelFormat.Format24bppRgb), image2);
 I've put above code into double FOR search that gave me out of memory thing.
Comment: @PoweRoy Very assumptions, but it's not a Counter Strike, Counter Strike uses AI functions as oppose to the graphical approach I'm attempting. Using this kind of aimbot in CS would be a suicide :) I'd rather let my girlfriend play.
Comment: @MandahMr. Why are you cloning `image1` all the time? Can't you just use the original image?
Comment: @GETah, yeah i've been trying to take the smaller part of the screen so it would be more comparable to the template, since i'm getting too many matches with just TemplateMatch[] matchings = tm.ProcessImage( sourceImage, template ); this. Basic idea was to break up the big screen into small images and compare each to the template, my idea is that might be more accurate. But i'm open to any suggestions, still new to C#.
Comment: @MandahMr. Have you tried changing the similarity threshold? Your code is using 0 as a threshold and that does not seem right to me
Comment: @GETah, yes i've tried that and if i go too high it finds nothing and when i bring it down it finds just too many things along with the right thing, can't find exact value that would do the trick, and also this method takes around from 5 seconds upto 5 mins depending on the resolution of the image. I dont mind if it takes around 1 second but can't spare more. The idea is quite simple  there must be something out there :-) hehe
Comment: @MandahMr. Have you tried the recommended value of 0.9f?
Comment: @GETah, yes i just tried and in that case it finds nothing at 69f it finds too many and at 70f it find only 1 but in the wrong place, and at 71f it finds nothing. I've read about this method and it seems that it uses pixels and their pattern for comparison. Meaning that it might not be the most ideal for searching something in the game screen where image will be very much diverse in terms of pixel patterns.
Comment: @MandahMr. The threshold should be in [0..1]. Also, ExhaustiveTemplateMatching processes only grayscale (8 bpp indexed) and color (24 bpp) images.
Comment: @GETah, i was referring to this // (set similarity threshold to 92.5%)
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching( 0.925f ); just didn't put the "0." in front of 70f. I'm not sure maybe i'm using it wrong or maybe this isn't the method i should use ?? But that's what i have so far twinkling with the comparison value.
Comment: @GETah, Also, ExhaustiveTemplateMatching processes only grayscale (8 bpp indexed) and color (24 bpp) images. :: so you mean there's no way to make it work faster for 24bpp image ?
Answer: Well, first I need to say that this will be probably slow, so if the red rectangle is moving fast, you will need some other solution. C++, CUDA, etc...
First:
Save a image of the red rectangle.
Define the area of possible locations of the red rectangle.
Steps:

Capture the game image (you can use Graphics CopyFromScreen).Copy only the area that the red rectangle might be, to reduce processing time.
Use EmguCV MatchTemplate to find the position of the red
rectangle.
Move the mouse to the position.
Some thread sleep and Repeat 1...

To process the image use EmguCV
To control the mouse use MouseKeyboardActivityMonitor
Speedup note: EmguCV have some CUDA support now, so you can try to use the CUDA version of the method.
<code>        //You can check if a 8bpp image is enough for the comparison,
        //since it will be much more faster. Otherwise, use 24bpp images.
        //Bitmap pattern = "YOUR RED RECTANGLE IMAGE HERE!!"
        Point location = Point.Empty;
        //source is the screen image !!
        Image<Bgr, Byte> srcBgr = new Image<Bgr, Byte>(source);
        Image<Bgr, Byte> templateBgr = new Image<Bgr, Byte>(pattern);

        Image<Gray, Byte> src;
        Image<Gray, Byte> template;
        src = srcBgr.Convert<Gray, Byte>();
        template = templateBgr.Convert<Gray, Byte>();

        Image<Gray, float> matchFloat;
        matchFloat = src.MatchTemplate(template, TM_TYPE.CV_TM_CCOEFF_NORMED);
        if (debug)
        {
            //matchFloat.Save(@"match.png");
            //Process.Start(@"D:\match.png");
        }
        //Gets the max math value
        double max;
        double[] maxs, mins;
        Point[] maxLocations, minLocations;
        matchFloat.MinMax(out mins, out maxs, out minLocations, out maxLocations);
        max = maxs[0];

        if (max > threshold)
        {
        location = maxLocations[0];
        //Point the mouse... Do your stuff
</code>

Find a red rectangle on the game screen


Dependency Hell trying to install GCC on CentOS4

Question: I have a production CentOS 4.4 box running Virtuozzo VPSes, and I'm trying to install GCC inside one of them so I can install the latest PHP from source. I downloaded gcc and its dependencies from a CentOS mirror and I was able to get some of them installed, but am running into problems with glibc-common:

[root@hq gcc]# rpm -i glibc-common-2.3.4-2.43.i386.rpm
warning: glibc-common-2.3.4-2.43.i386.rpm: V3 DSA signature: NOKEY, key ID 443e1821
        file /usr/bin/gencat from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/bin/getconf from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/bin/getent from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/bin/iconv from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/bin/lddlibc4 from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/bin/locale from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/bin/localedef from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/bin/rpcgen from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/bin/sprof from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/libexec/pt_chown from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/sbin/build-locale-archive from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/sbin/rpcinfo from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/sbin/zdump from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25
        file /usr/sbin/zic from install of glibc-common-2.3.4-2.43 conflicts with file from package glibc-common-2.3.4-2.25

So I tried to upgrade instead:

[root@hq gcc]# rpm -Uvg glibc-common-2.3.4-2.43.i386.rpm
warning: glibc-common-2.3.4-2.43.i386.rpm: V3 DSA signature: NOKEY, key ID 443e1821
error: Failed dependencies:
        glibc-common = 2.3.4-2.25 is needed by (installed) glibc-2.3.4-2.25.i686
        glibc-common = 2.3.4-2.25 is needed by (installed) glibc-dummy-centos-4-2.3.4-2.25.swsoft.i386

It seems like there's a dependency loop where glibc-common depends on glibc, but I can't upgrade glibc until I upgrade glibc-common. I'm also not sure what to do with the glibc-dummy-centos package that Virtuozzo setup. I've searched for solutions but haven't made any progress.
Update:
If I try to install them both simultaneously I still got the error about the dummy package:

[root@hq gcc]# rpm -Uvh glibc-common-2.3.4-2.43.i386.rpm glibc-2.3.4-2.43.i386.rpm
warning: glibc-common-2.3.4-2.43.i386.rpm: V3 DSA signature: NOKEY, key ID 443e1821
error: Failed dependencies:
        glibc-common = 2.3.4-2.25 is needed by (installed) glibc-dummy-centos-4-2.3.4-2.25.swsoft.i386

But I Google'd that and found that that it's safe to remove (1, 2) from the RPM database. After doing that I was able to install the remaining RPMs.
Answer: Install both.
rpm -Uvh glibc-common-2.3.4-2.43.i386.rpm glibc-...

Or better yet, just use yum.
yum install gcc

Comment: Yet another reason I won't touch Virtuozzo with a ten-foot cattle prod.
Comment: yum isn't available in Virtuozzo VPSes. I'll update the question with the error I get when trying to install them at the same time.
Comment: Yeah, I'm not a big fan of it either. But half of our servers are running it, so I'm forced to deal with it. Your answer fixed the interdependency problem (I don't know how I missed something that obvious...), and then I found a way to deal with the dummy package. Thanks :)


Error : Ajax call to server inside jquery function

Question: I am trying to make an ajax call to a server in my local network installed with LAMP who is running wordpress. I am currently doing changes to files in a wordpress plugin.
I want to call a php file located into my server to get json data to build a chart from a sql request.
So I have ajaxStatistics.php where I coded the sql request and the json encode.
Now I am in statistics.js where I have my function to build a chart and this is where I do the ajax call :
jQuery(function() {
    /**
     * call the ajaxStats.php file to fetch the result from db table.
     */
    $.ajax({
        url : "192.168.1.100/wp-content/plugins/pluginname/ajaxStatistics.php",
        type : "GET",
        success : function(data){
            console.log(data);
            ...

The result is an Uncaught TypeError: Cannot read property 'ajax' of undefined
at HTMLDocument.<anonymous> (statistics.js:92)
    at i (jquery.js?ver=1.12.4:2)
    at Object.fireWith [as resolveWith] (jquery.js?ver=1.12.4:2)
    at Function.ready (jquery.js?ver=1.12.4:2)
    at HTMLDocument.K (jquery.js?ver=1.12.4:2)

Comment: What is the problem?
Comment: Please show us error output also
Comment: I edited with error message
Comment: Have you included jQuery?
Comment: try to replace $ with jQuey => jQuery.ajax ..
Answer: You are trying to access $ as jQuery before it gets initialized so to make it in proper way, you can use DOM load event of it like so...
$(document).ready(function(){
    function build_statistics() {
        $.ajax({
        url : "http://192.168.1.100/wp-content/plugins/pluginname/ajaxStatistics.php",
        type : "GET",
        success : function(data){
            console.log(data);
        }
    }
});

And then call it like
build_statistics()

Comment: It says net::ERR_CONNECTION_REFUSED now
Comment: I solved it by using : http:/wp-content/plugins/pluginname/ajaxStatistics.php instead of Ipadress/etc/... Anyway, thanks for your help guys !
Answer: That means that jquery has not been loaded.
Make sure that you have the script in your html, and also wrap the call to this function sendData inside a
$(document).ready(function(){
//do stuff
})

Comment: I have the script linked in my html page.I wraped with document.ready, it changed the error to : Uncaught TypeError: $ is not a function.
Comment: is jquery.min.js included before script or after ?
make sure linked script is after jQuery.min.js
Answer: It seems like your $ variable is unreachable in your code.
Try this:
jQuery(function() {
    /**
     * call the ajaxStats.php file to fetch the result from db table.
     */
    var $ = jQuery;

    $.ajax({
        url : "http://192.168.1.100/wp-content/plugins/pluginname/ajaxStatistics.php",
        type : "GET",
        success : function(data){
            console.log(data);
            ...

And also make sure to add a http or https in your url.
Comment: Thanks I think it worked since I got the error : GET http://192.168.1.100/192.168.1.100/wp-content/plugins/pluginname/ajaxStatistics.php 404 (Not Found), the problem now is the access of the file, right? It s strange that it show the Ip adress two times tho...
Comment: I changed with https, I have the error : connection refused, Do I have to change something on the server ?
Comment: can you show your current code somewhere, because from the above url i can see, that you inserted the IP twice.


Creating a dict using two lists but formatting them differently

Question: I am forced to ask on here after embarrassingly failing to think of a quick solution myself. So I have the following:
keys= ["car", "cat", "sky"]
value_a = ['red', 'furry','cloudy']
value_b = ['blue', 'puffy','rainy']
value_c = ['yellow', 'pet','clear']
all_data = [value_a ,value_b,value_c]

def csv_exporter(keys, all_data):
    dictionary = dict(zip(keys, all_data))
    df = pd.DataFrame(dictionary)
    print(df)

csv_exporter(keys, all_data)

and so then my output is:
      car    cat     sky
0     red   blue  yellow
1   furry  puffy     pet
2  cloudy  rainy   clear

but what I want is this
      car    cat     sky
0     red   furry  cloudy
1     blue  puffy  rainy
2    yellow  pet   clear

Also, I am creating a dictionary first then exporting as a dataframe because that works more smoothly for my application.
Answer: You need to zip all_data too:
def csv_exporter(keys, all_data):
    dictionary = dict(zip(keys, zip(*all_data)))
    df = pd.DataFrame(dictionary)
    print(df)

Results:
>>> csv_exporter(keys, all_data)
      car    cat     sky
0     red  furry  cloudy
1    blue  puffy   rainy
2  yellow    pet   clear

Comment: thanks champ. zip(*) unzips it right?
Answer: Have you tried a simple transform?
all_data_new=[]

for element in range(len(all_data[0])):
    value_new=[]
    for dataSet in range(len(all_data)):
        value_new.append(all_data[dataSet][element])
    all_data_new.append(value_new)

all_data_new >> [['red', 'blue', 'yellow'], ['furry', 'puffy', 'pet'], ['cloudy', 'rainy', 'clear']]


How to replace failed disk on Linux Raid (mdadm)

Question: I get an input/output error when I list drives with: fdisk -l
Failed to read extended partition table (offset=3886721022): Input/output error
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x7ac0eeb9

Device     Boot      Start        End    Sectors  Size Id Type
/dev/sda1             2048 3886718975 3886716928  1.8T fd Linux raid autodetect
/dev/sda2       3886721022 3907028991   20307970  9.7G  5 Extended

Partition 2 does not start on physical sector boundary.

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x7ac0eeb9

Device     Boot      Start        End    Sectors  Size Id Type
/dev/sdb1             2048 3886718975 3886716928  1.8T fd Linux raid autodetect
/dev/sdb2       3886721022 3907028991   20307970  9.7G  5 Extended
/dev/sdb5       3886721024 3907028991   20307968  9.7G fd Linux raid autodetect

Partition 2 does not start on physical sector boundary.

Disk /dev/md1: 9.7 GiB, 10389291008 bytes, 20291584 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/md0: 1.8 TiB, 1989864849408 bytes, 3886454784 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Then, /proc/mdstat:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[2]
      1943227392 blocks super 1.2 [2/1] [_U]
      bitmap: 15/15 pages [60KB], 65536KB chunk

md1 : active raid1 sdb5[2]
      10145792 blocks super 1.2 [2/1] [_U]

unused devices: <none>

Then: mdadm --detail /dev/md0:
/dev/md0:
        Version : 1.2
  Creation Time : Tue Mar 20 06:41:14 2018
     Raid Level : raid1
     Array Size : 1943227392 (1853.21 GiB 1989.86 GB)
  Used Dev Size : 1943227392 (1853.21 GiB 1989.86 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Nov 23 16:21:07 2023
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : impacs:0
           UUID : 619c5551:3e475969:80882df7:7da3f864
         Events : 18817061

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       8       17        1      active sync   /dev/sdb1

Then: mdadm --detail /dev/md1:
/dev/md1:
        Version : 1.2
  Creation Time : Tue Mar 20 06:41:40 2018
     Raid Level : raid1
     Array Size : 10145792 (9.68 GiB 10.39 GB)
  Used Dev Size : 10145792 (9.68 GiB 10.39 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Thu Nov 23 14:19:34 2023
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : impacs:1
           UUID : 1b9a0dc4:cc30cd7e:274fefd9:55266436
         Events : 70615

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       8       21        1      active sync   /dev/sdb5

I don't see /dev/sda there, how can I know if it is used in the RAID?
Answer: About the only clue I see is this:
/dev/sda1             2048 3886718975 3886716928  1.8T fd Linux raid autodetect

It appears that your /dev/sda partition table, and probably /dev/sda1, is so messed up that RAID doesn't try to keep either sda1 or the no longer available sda5 in the disk array. So I'd say that it's about 99% certain that /dev/sda is half of your RAID arrays, and that it is dead and needs to be replaced.


Examples of principal ideal local Artinian rings

Question: I want to find some examples of principal ideal local Artinian rings.
That is, a commutative ring $R$ with identity with a unique maximal ideal, and this ideal is generated by a nilpotent element. (There is not any idempotent element in a local ring except $0$ and $1$, so under the Artinian condition, the maximal ideal cannot be generated by an idempotent one, and hence must be generated by a nilpotent one).
I have found three examples:
(1) Quotient ring $\mathbf{Z}/ p^d \mathbf{Z}$ with $d \ge 2$ and prime $p$;
(2) Quotient ring $K [x]/ \langle x^2 \rangle$ with a field $K$;
(3) Localization $(\mathbf{Z}/ p^d q \mathbf{Z})_q$ of the quotient ring $\mathbf{Z}/ p^d q \mathbf{Z}$ on $q$ with $d \ge 2$ and distinct primes $p$ and $q$.
But I realized that (3) is isomorphic to (1).
For some application, I need more examples (finite ring is more useful for me). But I cannot figure out any one else.
Many many thanks!
Answer: Well, most generally, if you take any principal maximal ideal $M$ of a commutative ring $R$, and select a nonzero integer $n > 1$, then $R/M^n$ is going to be an example.
Considering that all local principal ideal rings look like that (a ring whose ideals are precisely the powers of its maximal ideal) it's unlikely you'll get a much broader class of examples.
The trick, I suppose, is to be able to choose so that $M$ does interesting things. For example, if $M^2=M$ you only get a field.  But if $R$ is a domain, then you're guaranteed that $M^{i+1}\subsetneq M^i$.
As it happens the DaRT query only yields two things, both of which you've covered already.  But you could take any of the results of the PIDs that aren't fields query and do the construction I'm describing.


What assumptions do I need to check to combine levels in a categorical predictor for use in a GLM?

Question: If I have a GLM with a number of explanatory variables, where one is a categorical variable with levels "no treatment" "treatment A" "treatment B"
Assuming that it is reasonable to combine the 2 treatment groups together experimentally, what statistical assumptions do I need to check?
I wish to include to include the predictor variable treatment with levels 0 and 1 in the model.
For ANOVA I would check that it was reasonable to assume similar in groups variance for treatment A and treatment B
Does that assumption apply to GLM and linear regression as well? are there others to consider?
Answer: To check whether you can join the levels, a simple likelihood ratio test will do (the model with joined levels is a submodel of the one where they are separate).
As usual, this test is conditional on the correctness of the larger model (though hardly anybody ever bothers to check that), so technically, you need to assert the assumptions of your splitlevels model first. Note that checking the similar variance between two levels is not enough! Since you have multiple explanatory variables, what you really should check it the feasibility of equal variances in each 'cell', i.e. in each combination of levels from all predictors. Once again, typically, it is hardly possible to check this.
Finally (if the LR-test gives you enough confidence to join the levels), you should check the correctness of your reduced model, i.e.: once again checking the equal variance in each new 'cell' (note that the previous check on the old set of cells does not imply that the variances will be equal in the 'joined' sets, although it is highly likely if the reduced model came up ok: the means were then rather close and the variances were equal in each set of two old cells that were joined).
In reality, once again, this kind of thing is really hard to check (lots of cells with typically small numbers of observations, depending on your field of research). But remember that the univariate checks (like you appeared to suggest) in no way validate the equality of variances per cell. They are just an unfortunately accepted practice of soothing the mind.


Send email when zpool drive dies?

Question: I couldn't access my media drive the other day. SSH'd into my media server to see what was up. ZFS pool wasn't even mounted. When I got it running again, I saw that it was degraded and one of the harddrives was UNAVAIL.
Got it fixed now, but I'd like an earlier warning if this happens again. Can I get it to send me an email when a harddrive goes offline/degraded/unavailable?
Presently running ZFS 0.6.5.6 on Ubuntu 16.04.2.
Answer: You can (and should) monitor your pool via zed, the ZFS event daemon
Alternatively, you can configure cron to regularly run zpool status -x, sending email when some output is printed. For example:
#!/bin/bash
exit_code=0
/usr/sbin/zpool status -x | grep -v "all pools are healthy" && exit_code=1
exit $exit_code

Comment: https://medium.com/@arunderwood/configuring-zfs-event-daemon-zed-on-ubuntu-16-04-ee9ac2623c86 Looks like what I want. Will try this soon, thanks!

Send email when zpool drive dies?


Como acessar informações de um arquivo .torrent?

Question: Gostaria de saber se é possível e como obter informações (magnet link, seedes, leechers, peers, tamanho, etc..) de um arquivo .torrent utilizando PHP.
Comment: A pergunta foi reaberta após a abertura de [discussão no meta](http://meta.pt.stackoverflow.com/questions/3739/amplo-demais-qual-o-problema-desta-pergunta). @ViníciusGobboA.deOliveira O fato de a resposta existir no Google não quer dizer que ela não tenha espaço aqui. A não ser que a pergunta seja ampla demais (por exemplo, caso a resposta seja a documentação completa de uma ferramenta) ou tenha outro tipo de problema. O simples fato de haver respostas em outros lugares não invalida a pergunta aqui.
Comment: @ViníciusGobboA.deOliveira então se existe o Google, nenhuma pergunta aqui no *SOpt* precisa existir. A ideia é agregar conteúdo relevante no site, e esta é uma dúvida que não existe por aqui ainda.
Comment: Outra coisa @ViníciusGobboA.deOliveira: favor não postar links para o lmgtfy (eles soam meio pejorativos, não acha?)
Comment: @Otto Mesmo que essa lib resolva o problema, ela está em Inglês e não apresenta nenhuma informação contextual de *como*  ou *porque* funciona. Se fazemos tanta questão que as pessoas queiram aprender as coisas, evitar que apenas "copiem e colem" trechos de código na internet é o melhor caminho. E aqui nós podemos ter respostas muito mais completas e técnicas.
Comment: Eu votei para reabrir e me arrependi. Basta utilizar o Google, vários resultados são apresentados, basta escolher. É um problema trivialmente resolvido com uma busca no Google.
Comment: @PauloRodrigues essa talvez seja uma discussão para o meta, que não pretendo levar adiante. Mas não, o Google não é capaz de responder todas as perguntas de desenvolvimento. Disso eu não tenho dúvida.
Comment: Google serve para a pessoa pelo menos começar uma lógica de desenvolvimento que esta faltando aqui, na minha primeira busca pelo assunto achei https://github.com/adriengibrat/torrent-rw
Comment: @Gabe repassei o link para ajudar como estava sem tempo para montar um exemplo plausível. e ser menos "legal" que o outro amigo que mandou pesquisar no google.
Answer: Um arquivo de torrent contém apenas metadados, que são informações sobre o arquivo de destino mas nenhuma informação de conteúdo deste arquivo final. É basicamente um dicionário de bencode, que como neste exemplo, contém essa estrutura:
{
     'announce': 'http://bttracker.debian.org:6969/announce',
     'info':
     {
         'name': 'debian-503-amd64-CD-1.iso',
         'piece length': 262144,
         'length': 678428672,
         'pieces': '841ae846bc5b6d7bd6e9aa3dd9e551559c82abc1...d14f1631d776008f83772ee170c42411618190a4'
     }
}

Apenas um dos exemplos que você pode encontrar por aí, com esta classe você consegue extrair estes dados de um arquivo torrent, basta implementar algo assim para visualizar:
require_once 'class.bdecode.php';

$torrent = new BDecode('arquivo.torrent');
$results = $torrent->result;

Onde $results contém a estrutura de dicionário que citei um exemplo mais acima.
Como você pode observar, somente alguns destes dados que você citou estão incluídos no arquivo como o nome, partes, tamanho e etc, então isso responde parcialmente sua pergunta. Em relação aos seeds, peers e etc, isso vai além dos metadados do arquivo, pois envolve todo o protocolo de comunicação P2P.
Answer: Existe a biblioteca libtorrent em C++.
Porém ela fornece também uma interface python caso voce seja mais confortável com esta linguagem.
import libtorrent
info = libtorrent.torrent_info('test.torrent')
for f in info.files():
    print "%s - %s" % (f.path, f.size)

Fonte: SO
Comment: @OlimonF. Quando respondi, no dia 29/04 as 17:59, não havia esta informação, pois ela foi editada 29/04 às 18:20
Comment: O OP quer uma resposta para PHP.


Probability Theory, Chernoff Bounds, Sum of Independent (but not identically distributed) r.v

Question: Is there any known bound on sum of independent but not identically distributed geometric random variables?
I have to show that the tail of the sum drops exponentially (like in the Chernoff bounds for the sum of iid geom. variables).
Formally, if $X_i \sim Geom(p_i)$, $X = \sum_{i=1}^n X_i$, and $E[X]=\Theta(n)$,
Is it possible to show that $\Pr(X < 2E[X]) > 1 - \delta ^n$, where $\delta < 1$?
Thank you in advance,
Michael.
Answer: Hoeffding's inequality can be used but the random variables need to be bounded to get meaningful bounds. http://en.wikipedia.org/wiki/Hoeffding%27s_inequality.
Comment: Thank you for the answer. But as I see, Hoeffding's inequality holds for almost surely bounded variables. Moreover, as I added in the question, it is known that $E[X]=\Theta(n)$, and in that case there is no such bound. For example, if $n-1$ variables are distributed $Geom(p=1)$, and the last is distributed $Geom(p=1/n)$, the probability $\Pr(X<2E[X])$ is not exponentially high, but constant. So, there is no such bound in general case when the variables are not identical...


Allowed hidden parallel octaves

Question: I have read that hidden direct octaves between two voices are not always forbidden, but that there are some circumstances in which they are acceptable. As I have understood it, the following general rules are observed:

If one of the voices moves by jump and the other by contiguous movement, then they are allowed.

Exception: in the case of bass and the soprano, they are not allowed if the soprano jumps and the bass moves contiguously.

If both voices move by jumping, then they are forbidden.

However, two cases have occurred to me about which the text I have read says nothing:

What if both voices move by contiguous movement?
What if one of the voices does not move?

Would hidden direct octaves produced in this two ways be valid?
Answer: You question is kind of confusing, because you're mixing terms.
Hidden and direct and synonymous. Both are used in describing movement to a perfect interval by similar motion when the first interval is different that the second. Like a sixth to an octave with both voices ascending.
Parallel is more specific within the class of similar motion. Both intervals will be of the same type. Ex. a perfect octave to a perfect octave. "Hidden parallel octaves" is mixing terms. It's either hidden octaves or parallel octaves.

What if both voices move by contiguous movement?

If you move to an octave, and both voices move in contiguous (step-wise) similar motion, then by definition the previous interval must be an octave and the motion is parallel octaves.

What if one of the voices does not move?

By definition this is oblique motion. It's not similar motion therefore there can't be direct or parallel anything.
A few important concepts that you probably want to consider - if you haven't heard them already - are...

Independence of lines. Parallel perfect intervals are avoided at least in part because the consecutive intervals are exactly the same. Such lines are merely a transposition. Lines in similar - or parallel imperfect intervals - are not literal transpositions. There a slight changes in the exact linear contour of the part and that provides some basic level of independence.
Emphasize interval variety. In Fux the rule of thumb limit was 3 times for any consecutive interval. So even parallel thirds and sixth have a "limit." This is sort of a specific statement about linear independence. In effect it means you need to put in some contrary and oblique motion for variety's sake.
Third and sixth help define the modal aspects of a key. Perfect fifths and octaves help define the important tonal degrees. In practical terms third and sixth are better in the interior of phrases and perfect fifth and octave for cadences.
Relative motion "rules" are of special concern with the outer voices, the bass and soprano. When one of the voices is an inner voice, tenor or alto, there is less sensitivity about relative motion.

Some times it's good to keep positive ideas like those in mind. Those are the things you strive to do instead of worrying about prohibitions.
Comment: How are lines in parallel imperfect intervals - e.g. minor 6ths - not literal transpositions of each other? Parallel 6ths in a diatonic scale I get. But minor 6ths?
Comment: @Dekkadeci, well sure imperfect intervals of only one type would be transpositions ...and that would be very out of step with tonal harmony.
Answer: As I read various counterpoint summaries, the only problematic case is the soprano jumping (and turning state's evidence?). This emphasizes the octave and sounds like a voice dropped out. The best check is to listen for a noticeable "thinning" of the sound.

Allowed hidden parallel octaves


Laravel 5.5 cannot make validator in a form Request class

Question: I want to check if a form input 'departement' is filled only if two 'villes' have the same name.
within controller this code wokds perfectly :
 $rules=[  'nom' => 'required', 'ville'=> 'required|exists:villes,nom'];
 $messages = [
      'depart.required' => 'Deux villes portent le même nom, preciser le
 département'];
  $validator = Validator::make($request->All(), $rules,$messages);
  $validator->sometimes('depart_id', 'required|exists:departs,id', function
  ($input) {
      return Ville::where('nom',$input->ville)->count()>1;
  });
  if ($validator->fails()) {
    return redirect('admin/etab/create')
            ->withErrors($validator)
            ->withInput();
    }

I put the same  code in a Form Request  class:
public function rules()
{
  $rules=[  'nom' => 'required', 'ville'=> 'required|exists:villes,nom'];
  $messages = [
      'depart.required' => 'Deux villes portent le même nom, preciser le
  département',
  ];
  $validator = Validator::make($this->All(), $rules,$messages);
  $validator->sometimes('depart_id', 'required|exists:departs,id', function
  ($input) {
      return Ville::where('nom',$input->ville)->count()>1;
  });
  return $validator;
}

I get "Type error: Argument 2 passed to Illuminate\Validation\Factory::make() must be of the type array, object given," I think error message is inadequate but I cannot find why this way does not work
Thanks ......
Comment: Don't override the framework classes, just, don't. Check out https://laravel.com/docs/5.5/validation#form-request-validation
Comment: The rules method in the FormRequest class should return an array, not a Validator object. That's the error you are seeing here.
Comment: @Robert I understand that rule is not the good place for my code but I can't find int documentation where in a FormRequest I should place my code...
Comment: @aprokryfos Sorry for bad explaining I do not override the framework Request class, I create a FormRequest class. (I edited post)
Answer: You can check out the FormRequest class  in vendor/laravel/framework/src/Illuminate/Foundation/Http/FormRequest.php and check what it does.
It contains these 2 method at the top:
/**
 * Get the validator instance for the request.
 *
 * @return \Illuminate\Contracts\Validation\Validator
 */
protected function getValidatorInstance()
{
    $factory = $this->container->make(ValidationFactory::class);

    if (method_exists($this, 'validator')) {
        $validator = $this->container->call([$this, 'validator'], compact('factory'));
    } else {
        $validator = $this->createDefaultValidator($factory);
    }

    if (method_exists($this, 'withValidator')) {
        $this->withValidator($validator);
    }

    return $validator;
}

/**
 * Create the default validator instance.
 *
 * @param  \Illuminate\Contracts\Validation\Factory  $factory
 * @return \Illuminate\Contracts\Validation\Validator
 */
protected function createDefaultValidator(ValidationFactory $factory)
{
    return $factory->make(
        $this->validationData(), $this->container->call([$this, 'rules']),
        $this->messages(), $this->attributes()
    );
}

So you can basically provide a validator method in your own FormRequest class to create a custom Validator object, that method will get the ValidatorFactory as param.
In your case you would not need to do this, because you just want to append the sometimes rule to a default validator. Looking at the code above, it checks for the existence of the withValidator method, if it exists, it is called:
    if (method_exists($this, 'withValidator')) {
        $this->withValidator($validator);
    }

You could create the FormRequest, make sure the rules, messages and authorize methods are properly used, e.g. rules and messages return arrays and authorize returns a bool.
Then create a withValidator method in which you attach the sometimes rule to the Validator.
/**
 * Do foo with Validator
 *
 * @param \Illuminate\Contracts\Validation\Validator $validator
 * @return void
 */
public function withValidator(Validator $validator)
{
    $validator->sometimes('depart_id', 'required|exists:departs,id', function {
        return Ville::where('nom', $this->input('ville'))->count() > 1;
    });
}

This way sometimes is attached to your validator before the validation is performed.
Answer: You don't put all the validation logic in the rules method like that. Only the rule definitions go there. All you need is this:
public function rules()
{
  return [
    'nom' => 'required',
    'ville'=> 'required|exists:villes,nom',
  ];
}

Laravel will handle the validation from there on out. You don't need to manually create a Validator class when using FormRequests.
Customizing the message involves creating a messages method within the class like so:
public function messages()
{
    return [
        'depart.required' => 'Deux villes portent le même nom, preciser le départemen'
    ];
}

Update
As for the sometimes rule, I'd suggest creating a Rule object and customizing how you need to check for 2 vills having the same name.
Rule classes
Comment: That's ok for checking if ville exists. But I want to check if departement exists only iif two vills have the same name. For that I need a sometmes rule....
Comment: Ok, but what I've posted is how the FormRequest class needs to be laid out. The way you have the validator instance and messages array within the `rules` method is wrong. Those need to be removed for the FormRequest class to work


alias in index controller

Question: I'm very sorry I ask directly.
My Database:
id | A | B | C | D
And this is my controller:
$this->db->select("*")
         ->select('A + B + C as total_X')
         ->select('total_X - D as remains_Y')
         ->from('my_table')
.........
.........
......... another codes

This is the problem:
Unknown column 'total_X' in 'field list'
I'm little bit confused how to get "remains_Y" the addition from A, B, and C as total_X and I need to get "remains_Y" from (total_X - D).
What is the right code/syntax to get the Y?
Thank you so much for the help.
Best Regards,
Kris.
Answer: Try this:
$this->db->select("*, A + B + C as total_X, A + B + C - D as remains_Y")
         ->from('my_table')

alias in index controller


Gson asking for Integer TypeAdapter

Question: I am trying to serialize/deserialize a quite big object (that's why I will not write the whole object here) from and to json using Gson.
I have created TypeAdapter to DateTime and LocalDate but it still goes wrong with exception:
java.lang.UnsupportedOperationException: Attempted to serialize java.lang.Class: java.lang.Integer. Forgot to register a type adapter?

But as I have read in Gson docs, Integer does not need a TypeAdapter so I have check all the inner-objects inside this looking for something which needs an adapter and I found these non-primitive types:

Sets, List, Array, Enum

I have looked for them all and it seems like no one of these needs a TypeAdapter.
I am using this code to serialize the object:
Gson gson = new GsonBuilder()
            //these adapters are already tested and everything went right
            .registerTypeAdapter(DateTime.class, new DateTimeJsonConverter())
            .registerTypeAdapter(LocalDate.class, new LocalDateJsonConverter())
            .create();

WStepWork returnValue = new WStepWorkBL().getStep(id, currentUserId);

String json = gson.toJson(returnValue);

Does anyone knows if I actually need a TypeAdapter for any of these four or what is possibly going wrong?
Thanks in advance.
EDIT:
I have looking for the error and it's caused because of this:
model.java:
@XmlAccessorType(XmlAccessType.NONE)
public class WExternalMethod implements java.io.Serializable {

    //many attributes
    private String[] paramlistName;

    private Class[] paramlistType;
}

mapping.hbm.xml:
<class name="org.beeblos.bpm.core.model.WExternalMethod" table="w_process_head_method">
    <property name="paramlistName"
            type="com.sp.hibernate.common.usertype.StringArrayUserType">
            <column name="paramlist_name">
                <comment>for APP synchro: parameter list to send</comment>
            </column>
        </property>
        <property name="paramlistType"
            type="com.sp.hibernate.common.usertype.ClassArrayUserType">
            <column name="paramlist_type">
                <comment>for APP synchro: parameter list types (java types)</comment>
            </column>
        </property>
</class>

Comment: Start removing elements from that object one by one. Or half each time. You'll get down to the field that creates the problem. Please thoroughly read the link about [mcve].
Comment: That's a great idea. It will take long time so I might delay on answering, thanks btw
Comment: Please create a [mcve], which means you take your complicated object and reduce it to such a small object that only shows your error and doesn't contain any other stuff. And you also include the exact code that registers and serializes that object.
Comment: Totally forgot the code, sorry about that. But the object, I just don't know what is producing the error so I don't know what should I write here.
Comment: Can you step into the library in your debugger (open source?) and find the exact iteration/input line that is causing the error?
Answer: It was a missunderstood. It was asking for a TypeAdapter to Class and not to Integer so I did:
public class ClassJsonConverter implements JsonSerializer<Class>, JsonDeserializer<Class> {

    public ClassJsonConverter(){
        super();
    }

    @Override
    public JsonElement serialize(Class src, Type srcType, JsonSerializationContext context) {
        String str = src.getName();
        return new JsonPrimitive(str);
    }

    @Override
    public Class deserialize(JsonElement json, Type type, JsonDeserializationContext context)
            throws JsonParseException {

        Class returnValue = null;
        try {
            String str = json.getAsString();
            returnValue = Class.forName(str);
            return returnValue;
        } catch (IllegalArgumentException e) {
            return null;
        } catch (ClassNotFoundException e) {
            String mess = "deserialize(): "
                    + (e.getMessage()!=null?". "+e.getMessage():"")
                    + (e.getCause()!=null?". "+e.getCause():"");
        }

        return returnValue;
    }
}

Gson asking for Integer TypeAdapter