Newton's Method vs Gradient Descent?

Question: Currently I am studying logistic regression.
I read online that Gradient Descent is a 1st-order optimisation algorithm, and that Newton's Method is a 2nd order optimisation algorithm.
Does that mean that Gradient Descent cannot be used for multivariate optimisation and that Newton's Method cannot be used for univariate optimisation? Or can Newton's Method be done using a 1st order Taylor polynomial and still be different from Gradient Descent?
These sites are causing me to question:

Univariate Newton's Method http://fourier.eng.hmc.edu/e176/lectures/NM/node20.html
Multivariate Gradient Descent  http://fourier.eng.hmc.edu/e176/lectures/NM/node27.html
Comment: You can certainly use both gradient descent and Newton's method for either univariate or multivariate optimization.
Comment: This means that gradient descent depends only on the derivative of a function, whereas Newton's method depends on the first and the second derivative. This terminology (1st order/ 2nd order) has nothing to do with univariate or multivariate optimisation.
Answer: Like in the comments stated; gradient descent and Newton's method are optimization methods, independently if its univariate or multivariate. Gradient descent only uses the first derivative, which sometimes makes it less efficient in multidimensional problems because Newton's method attracts to saddle points. Newton's method uses the curvature of the function (the second derivative) which lead generally faster to a solution if the second derivative is easy to compute. So they can both be used for multivariate and univariate optimization, but the performance will generally not be similar.
Comment: @HughPearse Yes. But to see why you have to consider the Newton-Raphson method to solve a system of nonlinear equations in several variables, using the derivative (i.e. the jacobian matrix). When you apply this method to an optimiztion problem, you are given a function $f$ of several variables, and you look for critical points, so the system do solve is $\frac{\partial f}{\partial x_i}=0$, $i=1\dots n$. Thus when you apply Newton's method to this system of equations, you need the second derivative.
Comment: So does that mean that when you use the 1st-order Taylor polynomial it is no longer Newton's method, that it becomes Gradient Descent?
Comment: in essence, the (directional) second derivative tells us how well we can expect a gradient descent step to perform and uses therefore a second -order Taylor approximation. Gradient descent just moves into the direction of the negative gradient, where the gradient refers to the derivative with respect to a vector.
Comment: @HughPearse You are confusing two related but different goals. With Newton's method you solve an *equation* by using the 1st derivative. When you *optimize*, however, you are not looking for a zero of a function, but for a zero of its jacobian. You then need one more derivative, and that's the hessian. See [Newton's method in optimization](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization).
Comment: @Jean-ClaudeArbaut I found [this site](https://thelaziestprogrammer.com/sharrington/math-of-machine-learning/solving-logreg-newtons-method) which shows a Newton's Method on a univariate function using 1st order Taylor polynomial. There is no Hessian or Jacobian until they need to perform multivariate optimisation. Are they using the wrong terminology?
Comment: @Steven31415 For Newton's method to converge, you need both the second derivative to *exist*, and the initial point must usually be quite close to the solution (the [Newton-Kantorovich theorem](https://en.wikipedia.org/wiki/Kantorovich_theorem) only guarantees local convergence). Gradient descent is much slower but also more robust. And often computing the hessian is too costly (if there are $n$ variables, the gradient has size $n$, but the hessian has size $n^2$). So it's not all black and white.
Comment: Fully agree on your comments @Jean-ClaudeArbaut. The overall conclusion though is that both optimization methods can be used in multivariate and univariate optimization, but are different in their way of finding solutions.
Comment: @Jean-ClaudeArbaut so just to be clear, in a Taylor polynomial with 3 terms (a constant, a linear term and a quadratic term). The first term (the constant) is always written as $$f(x)$$ Say we have a nonlinear cost function $$J(x)$$ in the Taylor polynomial is  $$f(x) = J^{\prime}(x), f^{\prime}(x) = J^{\prime\prime}(x)$$ or is it $$f(x) = J(x), f^{\prime}(x) = J^{\prime}(x)$$
Comment: @HughPearse See [Taylor's theorem for multivariate functions](https://en.wikipedia.org/wiki/Taylor%27s_theorem#Taylor's_theorem_for_multivariate_functions). Not as friendly as the one variable case...
Answer: Let us assume that we are trying to minimize $f(x)$ w.r.t $x$, with $f$ differentiable.
Let $\nabla f(x) = f'(x)$ be the first derivative of $f(x)$ w.r.t $x$, and $\nabla^2 f(x) = f'(f'(x)) = f''(x)$ be the second derivative of $f(x)$ w.r.t $x$.
To minimize $f(x)$ w.r.t $x$,  we can use either the Gradient Descent method or Newton's Method, both of them are iterative methods.
$\textbf{Gradient Descent method}$ has the form:
$$ x^{t+1} = x^{t} - \eta (\nabla f(x))$$
where $\eta $ is a step size parameter. Here we use the first only. Thus, it is called a 1st-order optimization method.
$\textbf{Newton's Method}$
$$ x^{t+1} = x^{t} - \eta (\frac{\nabla f(x)}{\nabla^2 f(x)}) $$
here we use the first derivative and the second derivative, thus this method is called a 2nd-order optimization method.
Answer:
Newton's method aims to find the roots of a function $g$. i.e., the $x$ values satisfying $g(x)=0$. The iterative update state is $x_{n+1} = x_n - J_g (x_n)^{-1}g(x_n)$, where $J_g$ is the Jacobian matrix.

Gradient descent aims at finding a (local) minimum of a function $f$. A necessary condition for a function $f$'s minimum at $x$ is $\nabla f(x)=0$ (note that it's not sufficient since the critical point can be a maximum or a saddle point too, assuming $f$ to be continuous and diferentiable) and the iterative update step is $x_{n+1} = x_n - \eta\nabla f(x_n)$, where $\eta$ is the learning rate.

Hence, the minimization problem $\underset{x}{min} \; f(x)$ can be thought of as a root finding problem $\{x \;|\;\nabla f(x) = 0\}$, assuming that a local minimum exists in some neighborhood, i.e, let $g  = \nabla f$ and solve with Newton, with iterative update step being $x_{n+1} = x_n - H_g (x_n)^{-1}\nabla f(x_n)$, where $H$ is the Hessian matrix, hence it becomes a second order method.

The Newton's method converges much faster often but it's computationally more expensive.

The next couple of animations demonstrate the convergence to (global) minimum value $(0,0)$ for the (convex) function $f(x,y)=x^2+y^2$ with gradient descent (with learning rate $=\eta = 0.05$, takes around $50$ iterations to converge, as shown below) and Newton's method (converges in a single iteration), starting with initial value $(3,3)$.

$f = \begin{bmatrix} x^2 \\ y^2 \end{bmatrix}$, $\nabla f = \begin{bmatrix} 2x \\ 2y \end{bmatrix}$ and $H_f = \begin{bmatrix} \frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x \partial y} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2} \end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}$

Gradient descent update:

$\quad \quad \begin{bmatrix} x_{n+1} \\ y_{n+1} \end{bmatrix} = \begin{bmatrix} x_{n} \\ y_{n} \end{bmatrix} - \eta \begin{bmatrix} 2x_{n} \\ 2y_{n} \end{bmatrix} = (1-2\eta)\begin{bmatrix} x_{n} \\ y_{n} \end{bmatrix}$

As can be seen from above, in this case, gradient descent converges to the minimum in a single step only when $\eta=0.5$.

Newton's method update:

$\quad \quad \begin{bmatrix} x_{n+1} \\ y_{n+1} \end{bmatrix} = \begin{bmatrix} x_{n} \\ y_{n} \end{bmatrix} - \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}^{-1}\begin{bmatrix} 2x_{n} \\ 2y_{n} \end{bmatrix}$
$\quad \quad = \begin{bmatrix} x_{n} \\ y_{n} \end{bmatrix} - \begin{bmatrix} \frac{1}{2} & 0 \\ 0 & \frac{1}{2} \end{bmatrix}\begin{bmatrix} 2x_{n} \\ 2y_{n} \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$
Gradient Descent
with $\eta=0.05$

with $\eta=0.25$

with $\eta=0.5$

Newton's Method vs Gradient Descent?


"Notice: unserialize() [function.unserialize]: Error at offset 10301 of 11637 bytes on views_db_object->load_row()"

Question: I use the E-Commerce Kickstart distribution. Almost always I get some error, every time I update and save any view.

Please could anybody tell me what is causing this error, or give me some suggestions?
Answer: I have encountered the same issue and thus far to resolve but I found this article.
http://www.zyxware.com/articles/3337/drupal-errors-drupal-views-notice-unserialize-function-unserialize-error-at-offset-1009-of-1036-bytes-in-views-db-
so...i researched some more and found this solution. I added this line to bootstrap.inc
$data = preg_replace('!s:(\d+):"(.*?)";!e', "'s:'.strlen('$2').':\"$2\";'", $variable->value);
$variables[$variable->name] = unserialize($data);

in the variable_init function
Answer: I know this was answered long ago, but this is to help out future Googlers.
All unserialize errors result from bad data in the database.  Data stored in a serialized format will look like this:
s:9:"my string";

s - datatype (in this case string)
9 - length of the data
"my string" - the data

In your case, the unserialize error is resulting from a view. If you have a database viewer/query program (i.e. phpmyadmin, dbForge Express, Navicat etc.) you can look at the views tables.  You'll find that the only one containing serialized data is the views_display table; the display_options field contains serialized data.  You can then execute a query such as the following:
SELECT vid, display_title, length(display_options)
FROM views_display
WHERE length(display_options)="11637"

vid, display_title are used to identify which record from the database is corrupted
display_options is the field containing serialized data, could be stored in database as longtext or a blob
views_display is the table we are searching as it is relevant to the error and contains serialized data
"11637" is the length of the data stored in the field and corresponds to the bigger number on the error

This particular query should return one result with the view id.  You have the option to delete the view or try to repair the data.
To repair the data copy the serialized data into a text editor that will tell you what column/character you are on. In your case, you would go to column/character 10301 (the first number in the error) and see if you can identify a mismatch (wrong datatype, incorrect length, incorrect syntax). Copy the corrected data back into the database.
Comment: This is a great answer because it actually explains what is causing the problem, which clarifies why the accepted solution works.
Answer: Recently I faced the same problem. Solutions above didn't look very reliable to run on production.. Luckily I had a dev server where the view was working fine so I did the following:
Export the view:

Login to my Drupal site dev server
Go to /admin/structure/views and export the view from there
Copy the exported view code

Import the view:

On the drupal site that has the view issue, goto: /admin/structure/views/import
And paste the view code there
Make sure to check "Replace an existing view if one exists with the same name"
Click Import
Do not forget to save!

The imported view code solved the issue for me and hope that works for you as well!


The order of residual field of $p$-adic field

Question: Let $L/ \Bbb{Q}_p$ be finite extension. Let $o$ be ring of integers of $L$.Let $π$ be a uniformizer, and $q$ be order of residue field.
I want to determine the order of $(o/\pi^no)^\times $.
I know the answer, which is $q^n-q^{n-1}$.
I guess this is just $\phi(q^n)$($Φ$ is Euler function),
but I don't know the reason why.
How can I prove  $#(o/\pi^no)^\times=q^n-q^{n-1}$?
Comment: It is clear from context that $\pi$ is a uniformiser.
Comment: The question is ill-defined. What is $n$ ? what is $\pi$ ? Are you sure you do not have extra assumptions on your extension ?
Comment: Your formula for $(o/\pi^n o)^\times$ is incorrect. Indeed, $o/\pi o$ is the residue field of $L$ and its order depends on the residual degree of the extension $L/\mathbb{Q}_p$.
Comment: I think in this case $q$ is meant to be the order of the residue field of $L$.
Comment: Doesn't this smell of induction? Is the case $n=1$ clear? Any thoughts how to relate the cases $n$ and $n+1$?
Comment: By the way, unless $q$ is a prime itself, $\phi(q^n) \neq q^n-q^{n-1}$.
Answer: Let $q$ be the cardinality of the residue field of $L$. Then, for example by Proposition 5.4.5 (vii) in Gouvea's excellent book, any element in $O/\pi^n O$ can be written as a sum
$$
a_0+a_1\pi+a_2\pi^2+\cdots+a_{n-1}\pi^{n-1},
$$
where $a_i\in O/\pi O$. Thus the order of $O/\pi^n O$ is $q^n$. Now suppose that a coset in $O/\pi^n O$ is a unit in $O/\pi^n O$. It is easy to see that any representative of this coset must have $a_0\neq 0$. Hence the non-units in $O/\pi^n O$ are precisely of the form
$$
a_1\pi+a_2\pi^2+\cdots+a_{n-1}\pi^{n-1}.
$$
In particular, there are $q^{n-1}$ of them.


Django generic foreign key reverse filter

Question: I have a model called "Comments" which uses a generic foreign key:
class CommentManager(models.Manager):

    def for_model(self, model):
        """
        QuerySet for all comments for a particular model (either an instance or
        a class).
        """
        ct = ContentType.objects.get_for_model(model)
        qs = self.get_query_set().filter(content_type=ct)
        if isinstance(model, models.Model):
            qs = qs.filter(object_pk=force_text(model._get_pk_val()))
        return qs

class Comment(models.Model):
    """
    A user comment about some object.
    """
    status = models.CharField(max_length=12, blank=True, null=True)
    sub_status = models.CharField(max_length=500, blank=True, null=True)
    comment = models.TextField()

    content_type = models.ForeignKey(
        ContentType,
        verbose_name=_('content type'),
        related_name="content_type_set_for_%(class)s")
    object_pk = models.TextField(_('object ID'))
    content_object = generic.GenericForeignKey(ct_field="content_type",
                                               fk_field="object_pk")

One of the things you can put comments on are Tickets:
class Ticket(CommonModel):
    type = models.ForeignKey(TicketType)
    priority = models.PositiveSmallIntegerField()
    status = models.ForeignKey(TicketStatus)
    access_serial_number = models.PositiveSmallIntegerField(null=True)
    parent = models.ForeignKey('self', null=True, related_name='child')
    submitted = models.DateTimeField()

I do a lot of filtering of Tickets - in the past all I did with Comments on Tickets is that when I displayed a Ticket, I used Comments.objects.for_model(ticket) to find all the Comments for it. But now what I want to do is find Tickets that have a specific text in the comment. There is no comment_set or equivalent with GenericForeignKey.
This is what I've come up with, but it's pretty horrible:
comment_ticket_ids = [int(c.object_pk) for c in Comment.objects.for_model(Ticket).filter(comment__icontains='error')]
tickets = Ticket.filter(status=open_status, id__in=comment_ticket_ids)

There must be a better way.
Comment: I add comments to tickets using the `GenericForeignKey` content_object, as in `comment = Comment(content_object=my_ticket, status="Ticket Created", comment="So and so created this ticket for grins").save()`
Comment: How are you relating `Comments` to `Tickets`? I've read through your question several times but I don't see the relationship in your code example.
Answer: Perhaps you could add something using .extra() to help. Instead of
comment_ticket_ids = [int(c.object_pk) for c in Comment.objects.for_model(Ticket).filter(comment__icontains='error')]
tickets = Ticket.filter(status=open_status, id__in=comment_ticket_ids)

You could attach the field id_wanted:
extra_select = {
 'id_wanted': 'CASE WHEN (SELECT COUNT(*) FROM tickets WHERE comments something) > 0 THEN TRUE ELSE FALSE END'
}

Then filter for tickets with the extra select:
tickets = Tickets.objects.filter(ticket__status=open_status, id_wanted__isnull=False).extra(extra_select)

It's not fully clear to me why you can't have a FKey relationship between these two models.
Comment: 1. We use a GenericForeignKey because you can attach Comments to things other than Tickets.
Comment: 2. I'm not clear how your suggestion works - it appears to be attaching a `id_wanted` to Comment and then expecting it to be there in a query that doesn't involve Comment?
Comment: I revised my answer - will delete if un-useful. I'm trying to translate a pattern I used, where I also had a content type relationship (allowing comments on many different objects) and used a `.extra()` clause in our queryset


$\varphi(x)=0$ for every continuous linear form implies that $x=0$

Question: Let $(E,||~~||)$ a normed vector space and $x\in E$.
I want to show show that  $$\varphi(x)=0~~\forall\varphi\in E'\Rightarrow~~x=0$$
Thank you in advance.
Comment: E is a vector space over $\mathbb{K}$, and E' is the space of continuous linear maps from E to $\mathbb{K}$,
Comment: What is $E'$? What is its relationship with $E$?
Comment: Should the proposition you want to prove be restated as follows?

$$\exists A \subseteq E : x \in A \wedge \phi(x) = 0 ~ \forall \phi \in E' \Rightarrow 0 \in A$$
Comment: @DonnieDarko For the future: usually you should write about your own attempts of solving the problem in your question.
Comment: @supinf that what i do usualy, but this one it looks trivial but i cant prove it,
Answer: This is usually done using the
Hahn-Banach Theorem.
Some further hints:
You start by defining a suitable linear functional on the linear hull that is spanned by $x$
(under the assumption that $x\neq 0$).
Then you can extend this linear functional to a functional in $E'$.
Comment: Yes sir .if we suppose that $x\neq 0$ then
by a corollary of Hahn-Banach theorem there exists a continuous linear form $\phi$ such that $\phi(x)=||x||$ and $||\phi||=1$ hence the contradiction because $\phi(x)\neq 0$


Different front page per language

Question: Is there any way to provide a different node ID per language for the front page setting in Drupal 8?
We need a different front page for a specific language. In Drupal 7, this was easy.
Answer: Technically, the config override API that config translation uses allows any kind of override of configuration.
The core config translation however only does translation, meaning it only allows to change things that are classified as user-facing texts and not technical strings.
What you want is usually referred to as localization, changing things based the language that isn't actually a "translation". There is a core issue to better support this, https://www.drupal.org/node/2275865, but it hasn't been updated in quite some time.
There are a few possible workarounds, though:

Since you mention nodes, in Drupal 8, translations of nodes have the same ID/URL. That means you could have your different front pages as translations of the same node. That would be problematic if you use Paragraphs for example, as that doesn't currently support translatable fields, so you'd be forced into the same structure.

You could use Page Manager to have different front pages (there is a per-language constraint necessary, which has a patch but has not yet been committed) or even just a view that selects a node based on a certain condition (like filtered by language of a certain type).

A custom callback that displays a certain node based on a bit of code.

Update:
4 years later, I had this use case too and still no activity on the core issue. As a workaround, I altered the config schema and made the front_page a label type to be able to translate it. Not fully tested yet, so beware of dragons:
function custom_config_schema_info_alter(&$definitions) {
  // Change the front page type to label so that it can be translated.
  $definitions['system.site']['mapping']['page']['mapping']['front']['type'] = 'label';
}

Comment: Thanks Berdir for the reply. I'm aware of the translation of nodes being the same id, that's exactly the reason for the question - one of the issues being Paragraphs ;)
I was indeed thinking along the lines of custom code, but was hoping for a solution in D8 that I overlooked :-p
I found a workaround for my problem for this project, but will look into Page Manager for future projects.
Comment: @tijsdb Please share your workaround.
Comment: @KariKääriäinen for this particular project we were able to sort it with the same node eventually. But for other projects I'd follow the answer from Berdir, and use a view which loads 1 node for the active language, or a custom piece of code, redirecting to the right node depending on the language.
Answer: Another way to have a homepage per language, or just about any config override per language is to use config override system. I created a ticket in config_override module to add language overrides support and there is a patch attached: https://www.drupal.org/node/2914987
In short, you would be able to create a folder in your codebase with language overrides, where each language will have it's own subfolder and config override will merge config keys from language overrides with the active configuration.
Example of usage:
Include language_override config key in settings.php
$config_directories['language_override'] = '../config_override/language';

Place an override file in 'language_override' folder following core's config_translation structure:
[language_override]/en/system.site.yml
page:
  front: /node/123

The patch is in review, but you could install the module and apply the patch easily to get this feature.
Answer: Make a view that provides a block for each language. Master has no output, Content: ID (< 1).
For Language set Rendering Language: Content language of view row (default).
If you want to display a full node on some language's front page, use Format -> Show: Content | Full content, and for Filter criteria use Content: ID (= 123).
Create a node with no content. Make this node your front page in Basic Site Settings -> Default front page. Translate this node for all languages.
Place the blocks at Content region, no title, configure them with language restriction and with Pages restriction, show only on the node you set as your front page.
If you are displaying a full node on one some blocks, include the title in block configuration. The title being shown then is not the node title, but the block title. This will be h2, that needs to be styled like h1.
This is trickier than I remembered, but possible.
Comment: I used this answer on my D8 multilingual site today. Thanks!

Different front page per language


How do I add a Commerce product type to a feature?

Question: I have a custom product type that I created in my local dev environment and I'm trying to export it into a feature.
However, I can't seem to find where the product type is stored in the Features module UI.  I found the permissions (Product: View Own Custom Products) and the field instances (commerce_product-custom-commerce_license_role) but I don't see the product type itself.
Instead, when I enable the feature on production, I get the following error message:

Warning in features rebuild of custom_module. No module defines permission
  "view any commerce_product entity of bundle custom_product_type_machinename".

When I export to production, the product type is not created.  Where are the Commerce product types in the Features UI?
Answer: As it turns out, product type export is not supported in the core Commerce module.  Instead, you need to install the Commerce Features module.
Comment: Hah, beat me to it yourself. ; )


Android PopupMenu "could not find class android.widget.PopupMenu'

Question: I want to use a popup menu in my app, which should be compatible with Android 1.6+. Therefore I am using this code (taken from Supporting Different Platform Versions) to distinguish between pre-Honeycomb (which has no PopupMenu) and Honeycomb+ to display either a PopupMenu or an AlertDialog:
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.HONEYCOMB) {
    PopupMenu popup = new PopupMenu(this, v);
    ...
} else {
    showDialog(DIALOG_ID);
}

This works fine with an emulated Android 2.1 (showing AlertDialog), 2.3.3 (showing AlertDialog) and 4.2.2 (showing PopupMenu). However it does not work with an emulated Android 1.6. I get these error messages:
E/dalvikvm(211): Could not find class 'android.widget.PopupMenu', referenced from method ...
W/dalvikvm(211): VFY: unable to resolve new-instance 50 (Landroid/widget/PopupMenu;) in L...;
W/dalvikvm(211): VFY:  rejecting opcode 0x22 at 0x0006
W/dalvikvm(211): VFY:  rejected L...;.... (Landroid/view/View;)V
W/dalvikvm(211): Verifier rejected class L...;
W/dalvikvm(211): Class init failed in newInstance call (L...;)

Why does Android 2.x behave as expected but Android 1.6 does not?
Comment: Why don't u use just a Alert Dialog ? with controlling API version
if (android.os.Build.VERSION.RELEASE.startsWith("1."))
                      .
                      .
                      .

else if(android.os.Build.VERSION.RELEASE.startsWith("2.0") ||
        android.os.Build.VERSION.RELEASE.startsWith("2.1"))
                      .
                      .
Comment: 1.6 doesnot support popup menu..bcoz it is added in Api level 11..
Comment: @Sam That's why I show an AlertDialog if API level is lower than 11...
Answer: Look at the documentation of this class : http://developer.android.com/reference/android/widget/PopupMenu.html
It has been added at API level 11.
Is 1.6 a real requirement for your app ? It is virtually non existent right now.
You have two solutions :
- Use a DialogFragment instead
- Create your own implementation based on PopupWindow (as suggested in the doc http://developer.android.com/training/backward-compatible-ui/older-implementation.html)
I would suggest to reconsider having 1.6 as the lowest target if you don't have a real reason to do so, it will create more trouble than it is worth for literally 0.1% of the install base. In any case, DialogFragment is the way to go for this kind of UI elements, and it is part of the compatibility library (so it is compatible with old versions of Android).


Como formatar os dados extraídos do datagridview para o excel

Question: <code>    private void button1_Click_1(object sender, EventArgs e)
    {
        Microsoft.Office.Interop.Excel._Application app = new Microsoft.Office.Interop.Excel.Application();
        Microsoft.Office.Interop.Excel._Workbook workbook = app.Workbooks.Add(Type.Missing);
        Microsoft.Office.Interop.Excel._Worksheet worksheet = null;
        app.Visible = true;
        worksheet = workbook.Sheets["Plan1"];
        worksheet = workbook.ActiveSheet;
        for (int i = 1; i < dGVPosicaoEstoque.Columns.Count + 1; i++)
        {
            worksheet.Cells[1, i] = dGVPosicaoEstoque.Columns[i - 1].HeaderText;
        }
        for (int i = 0; i < dGVPosicaoEstoque.Rows.Count - 1; i++)
        {
            for (int j = 0; j < dGVPosicaoEstoque.Columns.Count; j++)
            {
                if (dGVPosicaoEstoque.Rows[i].Cells[j].Value != null)
                {
                    worksheet.Cells[i + 2, j + 1] = dGVPosicaoEstoque.Rows[i].Cells[j].Value.ToString();
                }
                else
                {
                    worksheet.Cells[i + 2, j + 1] = "";
                }
            }
        }
    }
</code>
Preciso formatar os dados que estão sendo extraídos para o excel pois o mesmo não esta diferenciando (.)ponto de (,)virgula. Exemplo: O valor extraido esta desta forma (109.390.759) sendo que o correto seria (109.390,759).
Answer: Pelo que entendi, você quer converter um valor da lista dGVPosicaoEstoque no formato 109.390,759.
Supondo que este valor seja um double e esteja cadastrado como 109390.759. Neste caso você poderá utilizar o método String.Format (documentação aqui) para formatá-lo da forma que quiser antes de inseri-lo na planilha.
Veja um exemplo:
<code>double valor = 109390.759;

string valorFormatado = string.Format("{0:0,0.000}", valor);

Console.WriteLine(valorFormatado);    // valor impresso: 109.390,759
</code>


How to get Sinatra to auto-reload the file after each change?

Question: I am using
# my_app.rb
load 'index.rb'

and start the sever like this
ruby my_app.rb

but it never reload any changes I made in index page.
Did I miss anything here?
Answer: See the Sinatra FAQ,
"How do I make my Sinatra app reload on changes?"

First off, in-process code reloading in Ruby is hard and having a
  solution that works for every scenario is technically impossible.
Which is why we recommend you to do out-of-process reloading.
First you need to install rerun if you haven't already:
 $ gem install rerun

Now if you start your Sinatra app like this:
$ ruby app.rb

All you have to do for reloading is instead do this:
$ rerun 'ruby app.rb'

If you are for instance using rackup, instead do
  the following:
$ rerun 'rackup'
You get the idea.
If you still want in-process reloading, check out Sinatra::Reloader.
Comment: shotgun requires a minimal config.ru. If you don't have one already, create it and put "require './my_app'" (where my_app.rb is your ruby app) in it and you'll be set.
Comment: Small addition to for `rerun`. You are not limited with default options i.e. `rerun 'rackup'`. It's possible to run any server on any port. Here is example how to run Puma on port number 5678 — `rerun 'rackup -s puma -p 5678 app/sinatra/config.ru'`
Answer:
gem install sinatra-reloader

require 'sinatra'
require 'sinatra/reloader'

Note: it will reload only sinatra handlers (and, maybe some sinatra server configuration commands), but not custom files, which you have to reload manually.
UPD after 9 years: seems like it is already possible to reload other files using also_reload, dont_reload and after_reload -- https://github.com/sinatra/sinatra/pull/1150
Comment: Use the also_reload method to specify which files to reload:
`configure :development do |c|
  require 'sinatra/reloader'
  c.also_reload "./lib/*.rb"
  c.also_reload "./controllers/*.rb"
  c.also_reload "./init/*.rb"
end`
Comment: Hey @Nakilon, how would I go about re-loading such files manually? I am doing `require "./my-file"` without success
Comment: @kristianlm, `require` doesn't reload file, if it already was required. You `load` instead.
Comment: you are right. I have to re-save my `server.rb`, though, to get it to reload my external file. It doesn't check my external file alone, so I have to re-save my `sever.rb` every time I make a change to my external file. [shotgun](http://rubygems.org/gems/shotgun) seems to work better in this regard, but it also seems to re-load the database sessions.
Answer: You can use the rerun gem.
gem install rerun
rerun 'ruby app.rb'

OR if you are using rackup
rerun 'rackup'

Answer: gem install sinatra-reloader
require 'sinatra/base'
require "sinatra/reloader"

class MyApp < Sinatra::Base
  register Sinatra::Reloader

  get '/' do
    "Hello Testing1!"
  end
end

You may want to set environment variable to development and conditionally load the gem.
Answer: When you run the application with Passenger Standalone, just create a tmp/always_restart file:
$ touch tmp/always_restart.txt

See Passenger documentation for more info.
Answer: I like the Shotgun gem. If you're using a modular Sinatra app and have a config.ru file it's easy to run.
shotgun config.ru

Check the gem out here. It's fairly straight forward and no configuration needed.
Comment: Shotgun is nice but isn't compatible with ruby 3+.
Answer: On Windows, I am using my restart gem for this:
restart ruby my_app.rb

or, with rackup:
restart rackup

See here for more info, hope you find it useful.
Comment: the documentation states that it doesn't work on windows, i tried and it loads sinatra but doesn't reload on chenges
Answer: You could use guard-rack. Lifted from an article at dblock.org:
Add this to your Gemfile:
group :development do
  gem "guard"
  gem "guard-bundler"
  gem "guard-rack"
end

Then, create a Guardfile at the root of your project with this content:
guard 'bundler' do
  watch('Gemfile')
end

guard 'rack' do
  watch('Gemfile.lock')
  watch(%r{^(config|app|api)/.*})
end

Lastly, run Guard, like so: bundle exec guard, and rackup will reload every time.
Answer: If you only change your templates sinatra will always rerender them if you set your environment to development:
ruby app.rb -e development


Round number to decimal places

Question: I have a javascript in html body for a BMI Calculator. Everything is working so far on my own accord. I just need help with two things which i cant just figure out.
Firstly i cant get the output BMI calculation to 2 decimal places. I have tried using toFixed(2) and double with no luck.
And also When you input the first variables it highlights the comments in the table. When you do a second input (with different outcome) it highlights a new field and keeps the old one. (not refreshing itself) and also cannot figure that.
Script below

    <title>Body Mass Index (BMI) Calculator</title>

        <form name="bmiForm">

            Your Weight(kg): <input type="text" name="weight" size="11">

            Your Height(cm): <input type="text" name="height" size="11">

            <input type="button" value="Calculate BMI" onClick="calcBmi()">

            Your BMI: <input type="text" name="finalbmi" size="11">

                     Under 18 - Your Are     Very Underweight and Possibly Malnourished.

                     Under 20 - You Are     Underweight and Could Afford to Gain a Little Weight.

                     20 to 25 - You Have a     Healthy Weight Range for a Young and Middle-Aged Adults.

                     26 to 30 - You Are     Overweight.

                     Over 30 - You Are     Obese

        </form>

Comment: Do not bet here  and java is not javascript.
Comment: fair enough, but why not here, i have seen similar question asked and answered in regards to the decimal issue. I have tried to do the research before hand to even comming this far.
Comment: I may not solve your problem. But the reason for getting downvotes and  no answers is you are not demonstrated your problem briefly. you just dumped your code and asked for help. Please cut down your code and post the suspected code. If possible create a [fiddle](http://jsfiddle.net/). People love to solve  well understood question :) hope you understand.
Comment: For future reference, your title is awful. It should tell us something about your problem. Perhaps something like "round number to two decimal places" would be better.
Answer: toFixed worked for me
document.bmiForm.finalbmi.value = new Number(finalbmi).toFixed(2);

Also you need to set background of other trs to white as below
if(finalbmi < 18){
     document.getElementById('tr1').style.backgroundColor="grey";
}
else{
     document.getElementById('tr1').style.backgroundColor="white";
}

Click Here for Fiddle
Comment: thanks, i see what you ment, i was adding

    document.bmiForm.finalbmi.value = finalbmi.toFixed(2);

Round number to decimal places


Prove that the intersection of two intermediate subfields of a Galois extension corresponds to the product of its subgroups of its Galois group

Question: Let $F/K$ be a Galois extension and let $E_1, E_2$ be intermediate subfields of $F/K$ with corresponding subgroups $H_1,H_2$ of $Aut(F/K)$.
Prove that the intersection of $E_1$ and $E_2$ corresponds to $<H_1H_2>$,and $E_1E_2$ corresponds to the intersection of $H_1$ and $H_2$.
Answer:
Let $H = \langle H_1H_2\rangle$, then for any $\sigma \in H_1H_2$, and $x\in E_1\cap E_2$, $\sigma(x) = x$. So
$$
E_1\cap E_1\subset K^H
$$
Also, since $H_1 < H$, $K^H \subset K^{H_1} = E_1$. Similarly, $K^H \subset E_2$. Hence
$$
E_1\cap E_2 = K^H
$$
Note that $E_1E_2$ is, be definition, the smallest field containing both $E_1$ and $E_2$. Hence, if $H = H_1\cap H_2$, then
$$
E_1E_2 \subset K^H \Rightarrow H \subset Aut(F/E_1E_2) \qquad (\ast)
$$
Also, if $\sigma \in Aut(F/E_1E_2)$, then
$$
\sigma\lvert_{E_1E_2} = id \Rightarrow \sigma\lvert_{E_1} = id \Rightarrow \sigma \in H_1
$$
and similarly $\sigma \in H_2$. Hence,
$$
Aut(F/E_1E_2) \subset H \qquad (\ast\ast)
$$
By $(\ast)$ and $(\ast\ast)$, we see that
$$
E_1E_2 = K^H
$$
Comment: Thank you .. you helped alot
Comment: @Prahlad Vaidyanathan: Is it true that the Galois group of $E_1 \cap E_2$ is the subgroup $$ ? It is clear that $ \subset \ Gal (F/(E_1 \cap E_2 ).$ I don't know if the converse is true !
Comment: It follows from the Galois correspondence and what is proved in (1) above.


What method to calculate the change of an output by removing an input?

Question: I have a KPI with a formula to calculate,  what I do is actually calculate the value of the KPI for the whole population (2245 individuals or input), the formula is:
v1+v2+v3 / a1+a2+a3

Where vi and ai are the sum of all the values of the variable for every input, i.e for example v1 represents the sum of all the 2245 values of v1.
After that I want to know how does the KPI change if I remove one input from the population, so what I do is calculate vi and ai without that input, and then calculate the KPI again. (I do it for all inputs)
I do this with excel for the moment and it is quite manual, I want to know if there's a function in R or Python that would allow me to do that.
Any help would be appreciated, thank you.
Answer: If you load your columns into a pandas dataframe you can do the following:
import pandas as pd

#prepare toy dataset
n = 2245
v1, v2, v3, a1, a2, a3 = np.random.rand(6,n).tolist()
df = pd.DataFrame([v1,v2,v3,a1,a2,a3]).T
df.columns = ['v1','v2','v3','a1','a2','a3']

kpis = []
for i in range(n):
    #select all values except the current one
    new_df = df.loc[df.index.difference(pd.Index([i]))]

    #3xN matrices
    right = new_df.T.loc[['a1','a2','a3']]
    left = new_df.T.loc[['v1','v2','v3']]
    #squash them into 1xN and sum again to get a number
    left = (left.sum()).sum()
    right = (right.sum()).sum()

    kpi = left/right
    kpis.append(kpi)


How to handle with broken module installation

Question: I prepared a module that has some issues. I tried to install it, but while installing it crashed. I'd like to make some tiny changes and reinstall it, but the module is set as installed. When I go to uninstall this page is no longer available, because while module was installing it provided some changes, so uninstall page crashes. I know how to uninstall a broken module, but this procedure is pure evil and it takes a wile. Is there, so the changes are rolled back if the module has some issues?
Comment: Possible duplicate of [Manually disable a module](http://drupal.stackexchange.com/questions/72927/manually-disable-a-module)
Comment: valli's suggestion to edit the `config` table works form me. Better would be to have a database backup to which you can revert.
Comment: Not only config table but there's a key_value table that needs to be investigated to uninstall module. There might be more tables to remove, but these two are the most important. Still, it takes quite some time to clear those.
Answer: Before the module installation you export your site configuration with drush cex. You install your module and if it crash you import the saved configuration with drush cim.
Comment: makes sense, I've also found that devel module has option to reinstall module and it can handle with errors. It's a really fast process, probably there's a drush or drupal console command to reinstall module using devel.


Combining bins of FFT ouput

Question: I was trying to combine output of a $2n$ point Real FFT to generate custom FFT bins.
For example the FFT generates components at equally spaced frequencies $f_0,f_1,f_2 ...f_{n-1}$
$f_0$         = $0$ Hz
$f_{n-1}$     = SamplingFrequency/2
Instead of generating a linearly distributed bins, I want to create custom bins. For example:
How do I create a bin for frequency  $(f_0+f_1)/2$. Can simply average $bin_0$ and $bin_1$ ?
Comment: Hi Jean, it seems like your new to the mathematics part of stack exchange, for some basic information about writing math at this site see e.g. [here](http://meta.math.stackexchange.com/questions/5020/), [here](http://meta.stackexchange.com/a/70559/155238), [here](http://meta.math.stackexchange.com/questions/1773/) and [here](http://math.stackexchange.com/editing-help#latex).
You may find that better formatted questions are more likely to receive good answers!
Comment: Thanks for pointing out. Formatted!
Comment: If you write out words like "sampling frequency" and "bin", they get interpreted as a juxtaposition of variables names and formatted accordingly (e.g. italicized). You can include text in formulas using the `\text{...}` command. (Also you forgot to convert several of the indices to subscripts.)
Answer: Certainly nothing keeps you from avergaging bin $0$ and bin $1$; the question is what you're trying to calculate. The FFT calculates a discrete Fourier transform, and you're trying to interpret it as something continuous. That's possible, but it requires a careful interpretation.
One interpretation goes like this: The data you transform represent samples taken over a finite interval with equal spacing. The Fourier coefficients that the FFT computes are the discrete Fourier transform of the data, which you can interpret as the Fourier sum for the unique periodic function that interpolates the sampled values and is band-restricted by the Nyquist frequency. You can view these Fourier coefficients as the coefficients of delta peaks at the corresponding frequencies in a continuous Fourier integral. Now if you restrict the periodic function to a single period, you effectively multiply it by a rectangular window, and in Fourier space that corresponds to convoluting it with the $\operatorname{sinc}$ function. Thus each of the delta peaks gets replaced by a copy of the $\operatorname{sinc}$ function centred at the corresponding frequency. Since these $\operatorname{sinc}$ functions are $1$ at their central frequency and $0$ at the frequencies corresponding to all other coefficients, this gives rise to a continuous function that interpolates between the discrete Fourier coefficients. It's this function that has a well-defined interpretation, namely as the Fourier transform of a single period of the band-limited interpolation of your data.
Thus, to get something meaningful, you'd have to interpolate with $\operatorname{sinc}$ functions, so you'd need a full sum over all the coefficients to get a single interpolated value, not just a combination of the two adjacent coefficients. However, if your data are sufficienly well-behaved, it will often be a good approximation to do linear interpolation instead as you suggested.
Comment: @Jean: We had all already understood that. If you just want a recipe (with no justification and no guarantees that it is "the right" one), go ahead with your recipe, it's often a good approximation. If you want to get to the bottom of the thing, then make an effort and read joriki's answer, just as he made the effort to write it.
Comment: I am trying to create logarithmically spaced frequency bins instead of the linearly spaced bins that FFT creates by default.
Comment: Thanks for pointing out. I missed the last paragraph of his answer!
Answer: Just to add to add to joriki's answer: if what you are trying to do is a "resampling" of the discrete Fourier transform, and if the resampling factor is near one, a linear or cuadratic interpolation is often enough.
If you have a large and-or varying resampling factor, (in particular if you want to do some remapping to a logarithmic scale), where several bins of the original FFT are roughly mapped to a new single bin, a simple average of the original bins can suffice.
But, just as with other general resampling procedures, it's usually better to employ some "window function" to minimize aliasing effects (which amounts to performing a weighted average; the plain average would correspond to a rectangular window). Typically, in linear FFT to MEL-scale (or MEL -like, i.e., logarithmic frequency scale) a varying triangular window (with overlaping) is applied, which is often called a "filter bank"; don't let the name intimidate you, it's just a weighted average. See eg here or here or here.

Combining bins of FFT ouput


How can top principal components retain the predictive power on a dependent variable (or even lead to better predictions)?

Question: Suppose I am running a regression $Y \sim X$. Why by selecting top $k$ principle components of $X$, does the model retain its predictive power on $Y$?
I understand that from dimensionality-reduction/feature-selection point of view, if $v_1, v_2, ... v_k$ are the eigenvectors of covariance matrix of $X$ with top $k$ eigenvalues, then $Xv_1, Xv_2 ... Xv_k$ are top $k$ principal components with maximum variances. We can thereby reduce the number of features to $k$ and retain most of the predictive power, as I understand it.
But why do top $k$ components retain the predictive power on $Y$?
If we talk about a general OLS $Y \sim Z$, there is no reason to suggest that if feature $Z_i$ has maximum variance, then $Z_i$ has the most predictive power on $Y$.
Update after seeing comments: I guess I have seen tons of examples of using PCA for dimensionality reduction. I have been assuming that means the dimensions we are left with have the most predictive power. Otherwise what's the point of dimensionality reduction?
Comment: You are correct: there is no mathematical reason to suppose the top $K$ PCs of $X$ have any predictive power--just as there is no mathematical reason generally to suppose that any set of covariates $X$ has any relationship at all to a given $Y$.  It seems you might be referring to some statement you have encountered: exactly what does it say and who said it?
Comment: @whuber I guess I have seem tons of examples of using PCA for dimension reduction. I have been assuming that means the dimensions we are left with have the most predictive powers. Otherwise that's the point of reducing dimensions?
Answer: Indeed, there is no guarantee that top principal components (PCs) have more predictive power than the low-variance ones.
Real-world examples can be found where this is not the case, and it is easy to construct an artificial example where e.g. only the smallest PC has any relation to $y$ at all.
This topic was discussed a lot on our forum, and in the (unfortunate) absence of one clearly canonical thread, I can only give several links that together provide various real life as well as artificial examples:

Low variance components in PCA, are they really just noise? Is there any way to test for it?
Examples of PCA where PCs with low variance are "useful"
How can a later principal component be significant predictor in a regression, when an earlier PC is not?
How to use principal components analysis to select variables for regression?

And the same topic, but in the context of classification:

What can cause PCA to worsen results of a classifier?
The first principal component does not separate classes, but other PCs do; how is that possible?

However, in practice, top PCs often do often have more predictive power than the low-variance ones, and moreover, using only top PCs can yield better predictive power than using all PCs.
In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely related to ridge regression, which is a standard way of shrinkage regularization. Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well. See Why does shrinkage work? for the general discussion about bias-variance tradeoff and about how shrinkage can be beneficial.
In a way, one can say that both ridge regression and PCR assume that most information about $y$ is contained in the large PCs of $X$, and this assumption is often warranted.
See the later answer by @cbeleites (+1) for some discussion about why this assumption is often warranted (and also this newer thread: Is dimensionality reduction almost always useful for classification? for some further comments).
Hastie et al. in The Elements of Statistical Learning (section 3.4.1) comment on this in the context of ridge regression:

[T]he small singular values [...] correspond to directions in the column space of $\mathbf X$ having small variance, and ridge regression shrinks these directions the most. [...] Ridge regression protects against the potentially high variance
  of gradients estimated in the short directions. The implicit assumption is
  that the response will tend to vary most in the directions of high variance
  of the inputs. This is often a reasonable assumption, since predictors are
  often chosen for study because they vary with the response variable, but
  need not hold in general.

See my answers in the following threads for details:

What is the advantage of reducing dimensionality of predictors for the purposes of regression?
Relationship between ridge regression and PCA regression
Does it make sense to combine PCA and LDA?

Bottom line
For high-dimensional problems, pre-processing with PCA (meaning reducing dimensionality and keeping only top PCs) can be seen as one way of regularization and will often improve the results of any subsequent analysis, be it a regression or a classification method. But there is no guarantee that this will work, and there are often better regularization approaches.
Comment: Also relevant: onestop's answer to http://stats.stackexchange.com/questions/3561/dealing-with-correlated-regressors?lq=1
Comment: Thank you for collecting the references in your answer. Here is [another one](http://stats.stackexchange.com/q/141280/3277) recent. There is an answer with further links.
Comment: Thanks, @ttnphns! I have not seen that post, because it did not have a [pca] tag (I am closely following only a handful of specific tags). Actually, I am rather unhappy that there is a loose collection of 5-10 closely related threads, with no question and no answer being really perfect and no real duplicates between them. I would prefer to have one canonical thread that could be used for future references...
Comment: I've added the tag to that question. The "perfect" encyclodedic answer on that interesting theme is awaiting its author. :-) You might decide to become a one.
Answer: In addition to the answers that already focus on the mathematical properties, I'd like to comment from an experimental point of view.
Summary: data generation processes are often optimized in a way that makes the data suitable for principal component (PCR) or partial least squares (PLS) regression.

I'm analytical chemist. When I designing an experiment/method to measure (regression or classification) something, I use my knowledge about application and available instruments to get data that carries a good signal to noise ratio with respect to the task at hand. That means, the data I generate is designed to have large covariance with the property of interest.
This leads to a variance structure where the interesting variance is large, and the later PCs will carry the (small) noise only.
I'd also prefer methods that yield redundant information about the task at hand, in order to have more robust or more precise results. PCA concentrates redundant measurement channels into one PC, which then carries much variance and is therefore one of the first PCs.
If there are known confounders that will lead to large variance that is not correlated with the property of interest, I'll usually try to correct for these as much as possible during the preprocessing of the data: in many cases these confounders are of a known physical or chemical nature, and this knowledge suggests appropriate ways to correct for the confounders. E.g. I measure Raman spectra under the microscope. Their intensity depends on the intensity of the laser light as well as on how well I can focus the microscope. Both lead to changes that can be corrected by normalizing e.g. to a signal that is known to be constant.
Thus, large contributors of variance that does not contribute to the solution may have been eliminated before the data enters PCA, leaving mostly meaningful variance in the first PCs.

Last but not least, there's a bit of a self-fulfilling prophecy here: Obviously PCR is done with data where the assumption that the information carrying variance is large does make sense. If e.g. I think that there could be important confounders that I don't know how to correct for, I'd immediately go for PLS which is better at ignoring large contributions that do not help with the prediction task.
Comment: +1. This is an excellent addition, thanks for joining the discussion.
Comment: @amoeba: thanks for the kind words. As always, also your answer is very thorough. I do in fact rely on you taking care of [tag:pca]!
Answer: PCA is sometimes used to correct problems caused by collinear variables so that most of the variation in the X space is captured by the K principal components.
But this mathematical problem is of course not the same as capturing most of variation both in X, Y space in such way that unexplained variation is as small as possible.
Partial least squares tries to do this in the latter sense:
http://en.wikipedia.org/wiki/Partial_least_squares_regression
Answer: As other has pointed out, there is no direct link between top k eigenvectors and the predictive power. By picking the top and using them as basis, you are retaining some top energy (or variance along those axis).
It can be that the axis explaining the most variance are actually useful for prediction but in general this is not the case.
Comment: When you say "in general", do you mean in general in practice or in general in theory?
Comment: @amoeba In general because it is easy to construct a dataset where projecting the data on top k maximum variance axis is not predictive/discriminative.
Answer: Let me offer one simple explanation.
PCA amounts to removing certain features intuitively. This decreases chances of over-fitting.


Conectar e pegar um arquivo de uma FTP

Question: Estou tentando pegar um arquivo de uma FTP, porém não estou conseguindo receber e também não aparece nenhuma mensagem de erro, pelo contrario...
Abaixo o código que estou usando:
<?php

/* Declaração de variáveis */
$serverHost = 'ftp.teste.com.br';
$serverUser = 'teste';
$serverPass = 'teste';
$arquivoLocal = 'C:\xampp\htdocs\sistema\temp';
$diretorioRemoto = '/pub/cre/licenca/';
$arquivoRemoto = 'licencas00003935000121.dat';

/* Faz conexão com o servidor */
if (!$ftp = @ftp_connect($serverHost, 2321)) {
    echo "Erro ao se conectar com o servidor FTP...\n";
}else{
     echo "Sucesso ao se conectar com o servidor FTP...
\n";

}

/* Efetua autenticação no servidor */
if (!@ftp_login($ftp, $serverUser, $serverPass)) {
    echo "Erro ao efetuar autenticação no servidor FTP...\n";
    }else{
     echo "Sucesso ao se autenticar com o servidor FTP...
\n";

}

/* Definindo o modo passivo ligado */
ftp_pasv($ftp, true);
/* Acessando o diretório onde está o arquivo */
if (!@ftp_chdir($ftp, $diretorioRemoto)) {
    echo "Erro! Diretório não existe...\n";
    }else{
     echo "diretorio encontrado...
\n";

}
/**
* Copia o arquivo do servidor remoto para o servidor local
* Use FTP_ASCII para arquivos texto
* Use FTP_BINARY para arquivos binários
*/
if (@ftp_get($ftp, $arquivoLocal, $diretorioRemoto, $arquivoRemoto, FTP_ASCII)) {
    echo "Erro ao fazer download do arquivo...\n";
}else{
    echo "sucesso
";
    exit();
}

Comment: O `@` em `@ftp_connect`, `@ftp_login` etc. Serve para ocultar os erros, por isso eles não aparecem.
Comment: tirei todos, agora da essa mensagem de permissão! Já adicionei as permissões e nada!
 Warning: ftp_get(C:\xampp\htdocs\sistema\temp): failed to open stream: Permission denied in C:\xampp\htdocs\sistema\conectaftp.php on line 36

Warning: ftp_get(): Error opening C:\xampp\htdocs\sistema\temp in C:\xampp\htdocs\sistema\conectaftp.php on line 36
Answer: Consegui resolver da seguinte forma:
<?php

$local_file = 'temp/arquivo.dat';

$server_file = '/pub/licenca/licencas96668116000172.dat';

$ftp_server="ftp.com.br";

$conn_id = ftp_connect($ftp_server,2321)or die("Erro de conexão com
$ftp_server");

$ftp_user_name="teste";

$ftp_user_pass="teste";

$login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass) or die("Não foi possível realizar o Login");

ftp_pasv($conn_id, true) or die("Não foi possível mudar para o modo passivo");

if (ftp_get($conn_id, $local_file, $server_file, FTP_ASCII)) {

  echo "Transferência da licença foi executada com sucesso\n";

}
else{

  echo "Verifique e tente novamente\n";
}

ftp_close($conn_id);

?>

Comment: E o que mudou? O que estava errado? Como corrigiu? Por que a alteração resolveu o problema? Poderia completar a resposta com essas informações?

Conectar e pegar um arquivo de uma FTP


Create object type from a string

Question: The title may be very vague, I did not know how to put the problem, so please excuse me for this.
I have a python file, say module.py
class myClass(obj):
    """
    class Description
    """
    def myMethod(args):
        """
        method Description
        """
        # method definition

I have one xml file, say module_result.xml
<testsuite errors="0" failures="0" name="module.myClass-20161215114804" skipped="0" tests="1" time="0.000">
    <testcase classname="module.myClass" name="myMethod" time="0.000"/>
    <system-out>
        something
    </system-out>
</testsuite>

I have another python file, test.py from where I have to use module_result.xml to parse module.py
What I am doing in test.py (by giving hard coded path) is:
getdoc(module.myClass.myMethod)

I have to make this generic by reading names from xml, and those names are of type string
The argument type for getdoc() is object type, I am getting this error while using getddoc() with string:
str(object='') -> str
str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or
errors is specified, then the object must expose a data buffer
that will be decoded using the given encoding and error handler.
Otherwise, returns the result of object.__str__() (if defined)
or repr(object).
encoding defaults to sys.getdefaultencoding().
errors defaults to 'strict'.

working code in test.py:
import module
getdoc(module.myClass.myMethod)

But, this is hard coded, which is not acceptable in my case
I am extracting module/myClass/myMethod names from xml file
Lets say I got those names in variable by some logic as:
string_variable = "module.myClass.myMethod"

and I now want to do this:
getdoc(string_variable)

after doing this I am getting above mentioned error.
What can I do to convert string to object type?
Answer: Referencing two different questions from elsewhere
here and here
Let's say you want to instantiate an instance of "myClass" from "myModule".  To keep it simple, let's assume that "myModule" is already imported
import sys
import myModule
import myOtherModule

def create_object(modulename, classname):   # modulename and classname are strings
    mod = sys.modules[modulename]
    cls = getattr(mod, classname)
    inst = cls()
    return inst

Comment: I thought what you were asking was this: *given a string that represents the name of a class, how do I instantiate an instance of that class?*.  In any case, [this link](http://stackoverflow.com/questions/4821104/python-dynamic-instantiation-from-string-name-of-a-class-in-dynamically-imported) might we what you are looking for.
Comment: I did not get what you are trying to convey.. What is 'name' here?
Comment: I updated my answer again to reflect some new stuff I learned.
Comment: You got the question right the first time.. here I don't know the name of module/class/method....   They are just stored in some variable or list as string type (which was extracted from the xml file).  I will edit the question to make things more clear...
Comment: My updated answer is close to what you want.  It assumes the module is already imported, but you have the target class in string form.  e.g. `obj = create_object("myModule", "myClass")`  You can use the links I reference on how to dynamically import a module by string e.g.`__import__("myModule")`

Create object type from a string


Issues with doing a P2V on Exchange 2007

Question: I shutdown all Exchange services on the physical box and did a convert. Everything converted fine. Server booted up no issues and I could receive email from the outside world with no issues. But I am unable to send emails internally or externally. They just sit in the Outbox.
I also disabled the DHCPClient service as I didn't want the server to get an IP address from DHCP when it first came online. It came online with no IP, then I used console to assign the correct static IP address (same one on the physical server) and rebooted.
I ran the Exchange Troubleshooting Agent and it gave me the following errors:
The value for the '\MSExchangeIS Mailbox\Messages Queued For Submission' counter on server exchange is greater than zero (average value is 8.8) and it appears that 'MSExchangeMailSubmission' is failing to submit messages to at least one computer with the Hub Transport server role installed over the last minute.
Found 1 computers with the Hub Transport server role installed in the same Active Directory site as server exchange using local Active Directory site GUID 'ce8a4367-1bf6-4825-9cac-c4e2b115c450'.
Check Local Active Directory Site Hub Transport Server Role Health (Mailflow_CheckLocalBhHealth18.104.22.168.220.127.116.11)
I have one Exchange server with all the roles on it.
Has anyone see this before when doing a convert? Please help :(
Comment: Have you checked the IP addresses that your send and receive connectors are listening on?
Comment: Did you remove the original NICs from the system? Directions: http://forums.ngemu.com/hardware-discussion/113370-remove-non-present-devices-system.html
Answer: Turn the DHCP Client service back on. It is used to register DNS records as well as DNS lookups.
With that service off, you shouldn't be able to resolve any DNS names from that server, which would explain why no mail could be sent from it.
Comment: +1. http://support.microsoft.com/kb/264539/EN-US/
Comment: Can I +5 this? :-)
Answer: After hours of trying tons of difference fixes online I was finally able to resolve my exchange 2010 issue after migrating to vmware esxi 5. E-mails would go to drafts and I could also not delete them. E-mails were coming in fine but could not send. After several reboots and "fixes" I finally restarted the service "Microsoft Exchange Mail Submission" and all e-mail in queue started sending. Everything is working ok now and didn't even have to reboot.
Answer: Stop all exchange services on the vm.  Rename your log directories, ex mine were first and second storage group, i renamed them to ORIGFirst Storage group and ORIGSecond storage group.  I then re-created the Folders First and Second Storage group with nothing in them.  Restarted all exchange services and mail started flowing.


managability issues over using a single machine to host all sites, or a site per machine?

Question: So, a little background. I work for a company that has a number of extremely important, non-public facing websites. People's safety and livelihoods depend on these staying up. We have very little downtime, but there are always catastrophic situations that mean restoring from bare metal.
Our current setup is inadequate, but I'd like opinions on what I see as the potential options. We host everything internally on an incredibly nice vSphere setup. Right now, we have one monstrous Ubuntu instance that hosts everything--all sites, databases, assets, et cetera.
We backup every way you can imagine, and one of the benefits of the vSphere setup is that we can restore offsite if we have to, but having one massive machine means the restore time isn't insignificant.
I see two roads I can head down.

Simple redundancy. Migrating from this one machine to a web server, a SAN and a database server, and then either having redundant machines ready full time, or be able to spin them up quickly. This is what I'd traditionally expect to exist, but I don't know that it helps us that much. Restoring offsite means taking hours to get all sites back up, and it seems difficult to restore in a way that I could give preference to the most mission critical things. Internally, with vSphere, this doesn't seem like a massive advantage. But, this is fairly easy to maintain.
Split everything up with vSphere. Each site could be its own vSphere instance (or small set of vSphere instances to split out database/assets). This means more work maintaining a number of small servers instead of the one monolithic one, but it also means I could easily choose to restore Site A and Site B in a catastrophic situation, and leave the non-mission critical things for later. This also allows things to diverge software wise where necessary, which is both a positive thing and a negative thing.

Opinions? Am I ignoring an obvious option?
Comment: `People's safety and livelihoods depend on these staying up.` sounds like you need HA more than DR.
Answer:
Leverage VMware SRM, or at least VMware Replication. It will drastically reduce the amount of time it takes you to come online in a secondary datacenter. (Hyper-V Replica and HVRM are the equivalent in the Microsoft stack.
Separate your front-end from your back-end. It sounds like you need a web tier and a database tier.
Invest in proper load balancing for your front end. This can mean installing and configuring a multi-site Netscalar cluster, or configuring something like HAProxy.
Introduce redundancy in your database tier. You don't mention which DB product you're using, but many have high availability, replication, clustering, etc available. Use this.
Make your DR site a "warm site" where you have some servers running constantly, such as database mirrors. Then you don't have to restore them in a disaster, you just make them the active node.

vSphere makes backup and recovery easier by abstracting the hardware away, but it's no substitute for tried and true HA methods when availability is critical.
There's no reason to have a single vSphere instance per-website. This doesn't gain you anything.
Answer: I've seen the second setup more often than the first, especially when your DR site doesn't have the same capacity as your primary site. However, I think this question isn't really appropriate for SF, since it's very much opinion based (although I'm not going to flag it as such since I see some value to the question).


Efficiently find connected subgraphs (not necessarily induced) of a given order

Question: Given a connected undirected vertex-labelled graph, an edge in the graph, and some number n ≥ 2, is there an efficient algorithm to find all the connected (but not necessarily induced) subgraphs of order n that contain that edge?
I thought the obvious thing to do would be a sort of recursive traversal of the edges, starting from the given edge and taking care that no one path uses an edge more than once (though visiting the same vertex multiple times is permitted).  After traversing each edge I check how many vertices are on the path.  If it's less than n, I proceed with the traversal.  If it's equal to n, I yield the subgraph corresponding to the path and proceed with the traversal.  If it's greater than n, I stop.  However, this method is inefficient as it produces many of the same subgraphs over and over.
(If it matters, all vertices in the original graph are of degree 8, and I will be looking for subgraph orders up to 15.)
Comment: [The answer](http://stackoverflow.com/a/15659485/2144669) I posted to a [similar question](http://stackoverflow.com/questions/15658245/efficiently-find-all-connected-subgraphs/15659485#15659485) can be adapted to this problem by using edges instead of vertices (two edges are neighbors if they share an endpoint) and cutting off the recursion when the target number of vertices is reached.
Answer: David Eisenstat's comment to my question suggested adapting his answer to a similar question.  The following pseudocode is that adaptation, which I've verified works and doesn't generate any one subgraph more than once.  In it n refers to the order of the subgraphs to be returned (the test for which can be omitted if you want to return all subgraphs) and neighbours(e) means the set of edges which share an endpoint with e.  When the function is first called, the subgraphEdges parameter should contain the given edge, and the rest of the parameters should be set accordingly.
GenerateConnectedSubgraphs(edgesToConsider,
                           subgraphVertices,
                           subgraphEdges,
                           neighbouringEdges):
    let edgeCandidates ← edgesToConsider ∩ neighbouringEdges
    if edgeCandidates = ∅
        if |subgraphVertices| = n
           yield (subgraphVertices, subgraphEdges)
    else
        choose some e ∈ edgeCandidates
        GenerateConnectedSubgraphs(edgesToConsider ∖ {e},
                                   subgraphVertices,
                                   subgraphEdges,
                                   neighbouringEdges)
        GenerateConnectedSubgraphs(edgesToConsider ∖ {e},
                                   subgraphVertices ∪ endpoints(e),
                                   subgraphEdges ∪ {e},
                                   neighbouringEdges ∪ neighbours(e))


Prediction for differenced Time Series model

Question: I am working on a Time Series model, and the series appeared to be non stationary (presence of trend & seasonality), as I was using ARIMA to predict for my time Series , I performed first order differencing.
My question is now if I am predicting future values, are the predictions going to be differenced? Do I need to undo the differencing in any manner before forecasting?
On a side note, can I use an ARIMA (p,d,q) (P,D,Q) model instead of differencing to account for seasonality ?
Comment: The answer is _yes_, the predictions will be transformed and, if you try to do this manually, you will need to back-transform your model to get the correct forecasted values. The good news is that this process is fully automated in most statistical software so you won't have to do it manually.
Comment: @Digio , any suggestions on how to tackle it in R, or just the forecast function will take care of it in R, also any comments on the SARIMA vs ARIMA with differencing part in my question ?
Comment: I have reposted everything as an answer below.
Comment: I want to bump this question. I am not using any automated package and I have differenced my time series and now my model is predicting difference forecasts. If I have to un-difference it then what would be the solution? Would I have to forecast for all of the data I have (train + test), then concatenate forecasts for both train & test and then un-difference using cumulative sum and the 1st known value in the train dataset? Please let me know ASAP!
Answer: The answer is yes, the predictions will be transformed and, if you try to do this manually, you will need to back-transform your model to get the correct forecasted values. The good news is that this process is fully automated in most statistical software so you won't have to do it manually. An example using Hyndman's 'forecast' package would be:
<code># Without integrated term:
plot( forecast(Arima(y = WWWusage, order = c(1,0,1))) )

# With integrated term:
plot( forecast(Arima(y = WWWusage, order = c(1,1,1))) )
</code>
As you can see, in both cases the output forecast value in the back-transformed units, as opposed to:
<code># With manual differencing (non-automated way):
plot( forecast(Arima(y = diff(WWWusage), order = c(1,0,1))) )
</code>
which forecasts unintelligible values.
If you want to use a seasonal ARIMA(p,d,q)(P,D,Q) model you should do so on grounds of some model validation metric and not because you're trying to sidestep the integrated terms (which you probably won't anyway). The best thing to do would be to let function auto.arima select a model for you.
Comment: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/62768/discussion-between-digio-and-av-abhishiek).
Comment: thanks for the answer, I was wondering if we perform differencing it will remove or try to remove seasonality , then how can we use ARIMA(p,d,q)(P,D,Q) ? i.e can we use differencing along with seasonality ?
Comment: You must see differencing and seasonality as two different things. The integrated term removes _trend_, not seasonality. A seasonal model will likely require an integrated term as much as a non-seasonal.
Comment: ohh, that explains, but from [link] [https://www.otexts.org/fpp/8/1], it states  "Transformations such as logarithms can help to stabilize the variance of a time series. Differencing can help stabilize the mean of a time series by removing changes in the level of a time series, and so eliminating trend and seasonality." It seems to be a reputed site and thats why I am confused.
Comment: Yes, it's a book written by the person I cited earlier (Pr. Hyndman) but I think you misinterpreted what it says. Stationarity transformations such as logarithmising may create a "seasonally adjusted time series" (where seasonality exists) but the purpose of the differencing operator is to remove trend and can be applied just as well to non-seasonal series. It is nowhere implied that a seasonal model will never have to be differenced, e.g. auto.arima(JohnsonJohnson) returns an ARIMA(1,1,2)(0,1,0) model.
Comment: Ohh that explains it, is it possible that if we have time series which has only seasonality and no trend then we can directly  apply a seasonality Arima in forecast instead of differencing
Comment: Of course, try _plot(mdeaths)_.


How to work with catching errors

Question: I refreshing my Java skills, say we've got this code
public class HelloWorld extends Exception {
    public static int tenMultiplication(int x) {
        for (int i = 2; i < 11; i++) {
            System.out.println(x * i);
        }
        return x;
    }

    public static String scanExpression() {
        Scanner scan = new Scanner(System.in);
        String exp = "";

        do {
            System.out.println("Please enter a number:");

            try {
                exp = scan.nextLine();
                int result = Integer.parseInt(exp);
                tenMultiplication(result);
            } catch (NumberFormatException e) {
                System.out.println("Please enter a number: ");
            }
        } while (exp.matches("-?\\d+") || exp == "exit");

        return exp;
    }

    public static void main(String args[]) throws Exception {
       scanExpression();
    }
}

Program logic: Program asks for an input, and draws a row of multiplication table till 10; any time you can exit by typing "exit", everything else is an error.
Every time I write an incorrect number, it will simply catch an error and exit the program. What is the best way with going about iteratively catching errors if  you consecutively type non-Ints and not "exit" to exit the program? I tried putting
exp = scan.nextLine();
int result = Integer.parseInt(exp);
tenMultiplication(result);

But when trying to write an error here, it throws the error again, which defeats the point of my try { } catch blocks.
Comment: `exp == "exit"` - See: [How do I compare strings in Java?](https://stackoverflow.com/questions/513832/how-do-i-compare-strings-in-java)
Comment: Honestly, there are many good books and tutorials out there teaching such things. And unrelated: there is absolutely no point in calling a class HelloWorld to then extend Exception. Your class has to deal with exceptions - that doesn't mean at all that your example class should **be** an Exception!
Answer: You need to re-read the input in case NumberFormatException is caught, you can do it in a recursive way by by calling your function again
public static void scanExpression(Scanner scan) {

        System.out.println("Please enter a number:");

        try {
            String exp = scan.nextLine();
            if(exp.equals("exit"))
                System.exit(0);
            int result = Integer.parseInt(exp);
            tenMultiplication(result);
        } catch (NumberFormatException e) {scanExpression(scan);}

}

public static void main(String args[]) throws Exception {

    Scanner scan = new Scanner(System.in);
    scanExpression(scan);
}

Comment: This is THE answer.
Comment: Thank you. Just checked it out and worked flawlessly. Gave me some good other ideas about how to order your code!
Answer: After the catch block executes the program continues and goes to check the condition in the while statement. The while statement returns false for that condition so your program terminates. Plus you need to compare exp like this: exp.equals("exit").
Answer: Here are some changes I made to your loop.
  do {
      System.out.println("Please enter a number:");

      try {
          exp = scan.nextLine();
          int result = Integer.parseInt(exp);
          tenMultiplication(result);
      } catch (NumberFormatException e) {

      }
  } while (!exp.equals( "exit") );

I made your catch block empty. The reasoning is that if it does catch invalid input from the user, "Please enter a number:" would print twice.
Also, I modified your while loop condition. Previously you had exp == "exit". This doesn't work the way you expect it to work. When comparing String values, always use `String1.equals(String2) to see if String1 and String2 are equal.

How to work with catching errors


Using iptables without root privileges

Question: When running the following script as user ec2-user, I get the error message iptables v1.4.18: can't initialize iptables table filter: Permission denied (you must be root)
Script:
#!/bin/sh
# Offending IP as detected by mod_evasive
# Add the following firewall rule (block IP)
$IPTABLES -I INPUT -s $IP -j DROP

How can I run iptables as a non-root user to block a IP address?
NB: This script is usually called by mod_evasive
Comment: Be very careful with that, be sure that `$IPTABLES` var is the right absolute path to the binary.
Comment: @XavierLucas: The path is correct: "IPTABLES="/sbin/iptables"
Answer: Set the setuid bit on the script, so that it always runs as root.
chown root myscript
chmod u+s myscript

Comment: Hi Michael, when trying to run the script, I do get the error message: command not found. What permissions would I need to change? Any idea?
Comment: What command is not found?
Comment: The script itself: When I enter the command in bash: "sudo ban_ip.sh 126.96.36.199xx", I do get: "sudo: ban_ip.sh: command not found". But the script file is there.
Comment: @AlexR Why are you trying to sudo ? Also if it's not in your `PATH` you need to use the absolute path `./ban_ip.sh`.
Comment: @XavierLucas: With or without sudo: It does not work. I am in the same directory as the script when trying to execute it.
Comment: You are not using the **absolute path**. It doesn't matter in which directory you are.
Answer: Based on another comment you made here the issue you're experiencing is bash cannot find the script you are trying to run.
When you are running a script or command that isn't in one of the paths defined in your $PATH environment setting you need to provide the absolute or relative path for it. For example:

If the script is in /usr/local/bin you need to run /usr/local/bin/scriptname.sh.
If the script is in your home directory you need to run either /home/username/scriptname.sh or ~/scriptname.sh.
Alternatively you can change into that directory and call it with ./ like so:  ./scriptname.sh

Also you can update your $PATH environment setting with the path to the script by modifying .bash_profile, .bashrc, or .profile, depending on what env file you're using.
Answer: Just add sudo before the command:
#!/bin/sh
# Offending IP as detected by mod_evasive
# Add the following firewall rule (block IP)
sudo $IPTABLES -I INPUT -s $IP -j DROP

Comment: using sudo to gain root privileges is not the same as accomplishing the task without root privileges...
Answer: If your script is:
/drop.sh

and looks like this:
#!/bin/sh
exec /sbin/iptables -I INPUT -s "$1" -j DROP

Then add a line to your /etc/sudoers file that looks something like:
www-data ALL=(root) NOPASSWD: /drop.sh

NB if Apache is running as a user besides www-data you'll need to use that username here. This lets the Apache user run your drop.sh
Then finally a wrapper for mod_evasive:
#!/bin/sh
exec sudo /drop.sh "$@"

Comment: What's the point putting the script at the root of the filesystem ? Also, you should always use `visudo` wrapper instead of editing any sudoers file, that's why it's read-only by default.
Comment: You can put files wherever you want, but then you need to explain that all the other paths need to change.

Using iptables without root privileges


How to open shared folder with administrator previleges

Question: Using windows 7.
I am administrator.
I want to open folder that is share and I am able to Open the folder if I am logged in.
Now if Some user is logged in and I want to open that folder from that user's login, can I open the share folder to which user is not authorize, with my admin login?
Some thing like right click and open with admin.
Comment: Only down notes and no answer.
Answer: It's not supported mate.
https://support.microsoft.com/en-us/kb/938120

There are two network shares on a remote server. You use user
credentials to connect to one of the network shares. Then, you try to
use different user credentials to connect to the other network share.
In this scenario, you may receive the following error message: The
network folder specified is currently mapped using a different user
name and password. To connect using a different user name and
password, first disconnect any existing mappings to this network
share.

Two workarounds are suggested:

Method 1
Use the IP address of the remote server when you try to connect to the
network share.
Method 2
Create a different Domain Name System (DNS) alias for the remote
server, and then use this alias to connect to the network share.
After you use one of these methods, you can use different user
credentials to connect to the network share. In this situation, the
computer behaves as if it is connecting to a different server.


Why a form doesn't keep the value created during construction

Question: Here is a form using DI for $this->entity_type_manager
There is also a radio-button with Ajax (I have added the code as it is maybe the reason of my problem)
class myForm extends FormBase {

  private ?EntityTypeManagerInterface $entity_type_manager=NULL;

  public function __construct(Drupal\Core\Entity\EntityTypeManagerInterface $entity_type_manager) {

    $this->entity_type_manager=$entity_type_manager;
  }

  public static function create(ContainerInterface $container) {
    return new static(
      $container->get('entity_type.manager'),
    );
  }

  public function buildForm(array $form, FormStateInterface $form_state) {
    // keep the value in $form_state to be used in submit
    $form_state->set('entity_type_manager', $this->entity_type_manager);
    ...
    // Maybe the Ajax component is the problem
    $form['container'] = [
      '#type' => 'container',
      '#prefix' => '
',
      '#suffix' => '
',
      'my_radio' => [
        '#type' => 'radios',
        '#options' => ["option1","option2"],
        '#required' => TRUE,
        '#default_value' => $type_of_bo,
        '#ajax' => [
          'callback' => [$this, 'ajaxGetInvestmentHandler'],
          'wrapper' => 'ajax-wrapper',
          'event' => 'click input',
        ],
      ],
      ...
    ];
  }

  static public function ajaxGetInvestmentHandler(array $form, FormStateInterface $form_state) {
    return $form['container'];
  }

  public function submitForm(array &$form, FormStateInterface $form_state) {
    // $this->entity_type_manager is NULL
    // $form_state->get('entity_type_manager') is also NULL
    ...
  }

  ...

}

As you can see in the submit, $this->entity_type_manager is NULL and $form_state->get('entity_type_manager') is also NULL.
What is wrong?
Comment: "As you can see" - how can we see this? There isn't any reason why it should be null, how are you determining it is?
Comment: Sorry to not be clear enough. I meant that I have made some tests on those values and they are NULL (as showed by the somments in the submit). I agree this should not be NULL but it is... maybe it is coming from the AJAX?
Comment: Based on the code shown, it's not possible for `$this->entity_type_manager` to be NULL. The submitForm function cannot be called without instantiating the form, and the form requires the variable be passed to it to construct the form. That's why I'm asking how you diagnosed it to be NULL.
Comment: $this->entity_type_manager is also NULL in the validate function...
Comment: Because you're setting it to null... `$entity_type_manager=NULL;`
Comment: If I don't set it to NULL as a default value, I got `$entity_type_manager must not be accessed before initialization`
Comment: Then that's a different problem.
Answer: As you can see in the submit, $this->entity_type_manager is NULL
You need to declare the service property protected. Private properties can't be restored. See https://www.drupal.org/docs/drupal-apis/services-and-dependency-injection/dependency-injection-for-a-form#s-considerations-for-dependency-injection
.. and $form_state->get('entity_type_manager') is also NULL
The form_state is not cached between the first form build and the submit request. See buildForm: how differentiate between page reload and ajax callback?. And when you run the form build in later Ajax requests you overwrite the form state value with NULL.
Comment: This is magic. You save my day!!!