Spectral norm minimization

Question: I was reading the use of semidefinite programs to formulate the matrix norm minimization, but I am having trouble trying to understand it. I'd also like to understand it at a more intuitive level.

[Boyd and Vandenberghe: Convex optimization $\S$ 4.6.3]
Matrix norm minimization
Let $A(x) = A_0 + x_1 A_1 + \dots + x_n A_n$, where $A_i \in \mathbf{R}^{p\times q}$. We consider the unconstrained problem
$$
\textrm{minimize} \qquad
\|A(x)\|_2,
$$
where $\|\ \cdot\ \|_2$ denotes the spectral norm (maximum singular value), and $x \in \mathbf{R}^n$ is the variable. This is a convex problem since $\|A(x)\|_2$ is a convex function of $x$.
Using the fact that $\| A \|_2 \leq s$ if and only if $A^TA \preccurlyeq s^2 I$ (and $s \geq 0$), we can express the problem in the form
\begin{align}
\textrm{minimize} &\qquad s \\
\textrm{subject to} &\qquad A(x)^TA(x) \preccurlyeq sI,
\end{align}
with variables $x$ and $s$. Since the function $A(x)^TA(x) - sI$ is a matrix convex in $(x,s)$, this is a convex optimization problem with a single $q \times q$ matrix inequality constraint.

Where can I see what this fact is talking about? The only thing I read, based on Wikipedia, is that the $L_2$ norm of a matrix is $\|A\|_2 = \sigma_{\max}(A) \le \left(\sum_{i,j} |a_{i,j}|^2\right)^{\frac{1}{2}}$

Since $A^TA \preccurlyeq s^2I $ indicate that the matrix $A^TA - s^2I$ is negative semidefinite, if I have a $2\times 2$ matrix $A$, then

$\|A\|_2^2 \le s^2 \implies a_{11}^2 + a_{12}^2 + a_{21}^2 + a_{22}^2 \le s^2$

$A^TA - s^2I \preccurlyeq 0 \implies s^2(a_{11}^2 + a_{12}^2 + a_{21}^2 + a_{22}^2) - (a_{11}a_{12} + a_{21}a_{22})^2 \le 0 $
I spent a while looking at this expression, but I am still unsure how it explains the fact in point 1.

In layman's terms, is the expression saying that the $L_2$ norm of a matrix can only be lesser than its maximum eigenvalue ($s$?) if $A^TA - s^2I$ is negative semidefinite?
Comment: Related: https://math.stackexchange.com/q/2136401/339790
Comment: Does this answer your question? [Spectral norm minimization via semidefinite programming](https://math.stackexchange.com/questions/2136401/spectral-norm-minimization-via-semidefinite-programming)
Answer: As $\|A\|_2=\max_{x\neq 0} \frac{\|Ax\|_2}{\|x\|_2}$ by definition, for every $s\geq 0$, we have
\begin{align*}
\|A\|_2 \leq s &\iff \frac{\langle Ax,Ax\rangle}{\langle x,x\rangle}=\frac{\|Ax\|_2^2}{\|x\|_2^2}\leq s^2 \qquad &\forall x\neq 0\\
&\iff \langle Ax,Ax\rangle\leq s^2\langle x,x\rangle \qquad &\forall x\neq 0\\
&\iff \langle A^TAx,x\rangle- s^2\langle x,x\rangle\leq 0 &\qquad \forall x\neq 0\\
&\iff \langle (A^TA-s^2I)x,x\rangle\leq 0 \qquad &\forall x\neq 0\\
&\iff A^TA\preceq s^2I
\end{align*}

Spectral norm minimization


surface area of f(x,y) on a circular region

Question:
Find Surface area of $f(x,y) = xy$ over the region $\{(x,y) : x^2 + y^2 \le 16\}$

I know that I will probably need to convert to polar
I have some notes that show that if the area was a rectangle then I would do a double integral over that region and the function in the middle would be $\sqrt{ 1 +(\partial_x)^2 + (\partial_y)^2}$
If someone can show an example of how to do a similar problem with different numbers, that would be amazing
Answer: Note the magnitude of your surface normal is given as
$$
\sqrt{(f_x)^2+(f_y)^2+1}=\sqrt{y^2+x^2+1}
$$
and thus your surface integral is
$$
\int\int_{D}\sqrt{y^2+x^2+1}\mathrm dx \mathrm dy
$$
where $D$ is the disc of radius $4$ around the origin, all of which is all very convenient for polar, where your integral becomes
$$
\int_0^{2\pi}\int_0^4r\sqrt{1+r^2}\mathrm dr \mathrm d\theta=2\pi\int_0^4r\sqrt{1+r^2}\mathrm dr\\
=\pi\int_1^{17}\sqrt{u}\mathrm du
$$
Answer: The surface area is given by $$S_A=\iint_R \sqrt{1+(f_x)^2+(f_y)^2}dA.$$ In this case $R=\{ (x,y) \in \mathbb{R}^2:x^2+y^2 \leq 16 \}$, $f_x(x,y)=y$ and $f_y(x,y)=x$. Then $$S_A=\iint_{R} \sqrt{1+x^2+y^2}dA.$$ Using polar coordinates we get that $r \in [0,4]$ and $\theta \in [0,2\pi)$, and so $$S_A=\int_0^{2\pi}\int_0^4 r\sqrt{1+r^2}drd\theta.$$ It's easy to see that $$\int_0^x r\sqrt{1+r^2}dr=\frac{(x^2+1)^{3/2}}{3} \implies \int_0^4 r\sqrt{1+r^2}dr=\frac{17\sqrt{17}-1}{3}$$ and so $$S_A=\int_0^{2\pi} \frac{17\sqrt{17}-1}{3}d\theta=\frac{2\pi}{3}\left(17\sqrt{17}-1 \right).$$
Comment: Stupid question. How is the partial derivative not 2x and 2y?
Comment: Intuitively, the partial derivate of a multivariable function with respect to a variable (let's say $x$)  is justo to derivate the function "putting" all the variables that are not $x$ as constants. For example, if $g(x,y,z)=3xy-6z+\cos(z)$ then $g_z(x,y,z)=-6-\sin(z)$. Since the function you are working with is $f(x,y)=xy$, then $f_y(x,y)=x$, because $x$ acts like a constant in this case, the same with $f_x(x,y)=y$. Remember for example that the derivative of $x \mapsto 7x$ is $7$.
Comment: I realized that I was taking the partial derivative of the region's equation which is why I quoted 2x and 2y instead of taking partial derivative of the f(x,y)


Using group by in AutoMapper

Question: Can we use group by in AutoMapper with C# and Entity Framework? Can you show an example please?
CreateMap<x, y>()
            .ForMember(destination => destination.Id)
                options => options.MapFrom(source => source.Id)
            .ForMember(destination => destination.AktifMi,
                options => options.MapFrom(source => true));

I will be glad if you apply here
var y = _context.XXX.OrderByDescending(i => i.Id);
var ca = _mapper.ProjectTo<x>(y);

Comment: Show us the code without AM, with LINQ.
Comment: I added it at the bottom of the thread
Comment: That's not an group by, but yes, that would work. `ProjectTo` is just a `Select` statement.
Comment: I'm sorry but I didn't fully understand your answer.
Comment: can i group in automapper
Comment: yes you can pass a grouping or do group in automapper
Answer: you can do whatever you like in automapper.
You just need the correct model to map the source to your destination.
here is a sample snippet.

CreateMap<IGrouping<string, SourceModel>, DestinationModel>()
.ForMember(target => target.Users,
  source => source.MapFrom((src, dest) => {
      return BuilUsers(src);
  }));

  private IEnumerable<User> BuildUsers(IGrouping<string, SourceModel> source)
  {
      var result = new List<User>();

      foreach (var sourceUser in source)
      {
          result.Add(new User
          {
              UserName = sourceUser.UserName,
          });
      }

      return result;
  }

Let me know if you have any further question, Thank you.

Using group by in AutoMapper


Tomcat localhost access log does not contain failed requests

Question: Default Tomcat 8.5 logs do not appear to show any failed requests, or anything about the SSL handshakes. Specifically the localhost log and catalina.out
See this question for context about the type of failed request:
Intermittent Curl 35 error when using self signed certificate on Tomcat
This question seems to indicate that it is not possible....?
Tomcat access logs - are failed requests included?
I tried turning on debug, but still couldn't find anything related to a failed request.
Comment: Are you checking all logs? The `localhost_access.log` should contain the `500` errors. Application specific logs may end up in `localhost..log` or in application specific logs.
Comment: It's not actually a 500 server error, it doesn't seem to get that far. There is effectively no response from the server. As if it's failing during the SSL handshake. I'm using Tomcat with APR for SSL.
Answer: If you want to log the connection details, you need to increase to log levels of loggers org.apache.coyote and org.apache.tomcat.util.net, by putting:
org.apache.tomcat.util.net.level = FINE
org.apache.coyote.level = FINE

in $CATALINA_BASE/conf/logging.properties and increasing the log level of some handler, e.g.:
java.util.logging.ConsoleHandler.level = FINE

whose messages eventually end up in catalina.out.
This requires a server restart. You should be able to do the same thing through JMX (e.g. jconsole), without a restart.
Comment: I just had to set org.apache.coyote.http2.level to FINE and that gave me a stack of output. Not sure if it's exactly the one I'm looking for yet. But there is an EOF Exception in there, so good place to start.


Amazon RDS DB Migration - Replication Task cannot connect to AWS RDB but I can

Question: I am doing a proof of concept to migrate an existing SQL Server database to Amazon RDS. I set up the Amazon RDS database (I tried with SQL Server and Aurora - the result was the same in both cases).
To perform the migration, Amazon sets up the replication task. This task can connect to the source database (which is on the public internet), but always fails to connect to the Amazon database instance I have set up (SQL server, Aurora). However, I can connect to the Amazon database instance via a client tool (SQL Server Management Studio / MySQL client) without a problem!
So it seems that Amazon doesn't allow its own replication task to connect to the database, but allows external clients ??! That seems totally weird ?!
I used all default settings and haven't modified any network/security parameters.
If anyone can shed some light on this it'd be highly appreciated.
Here's the error message I got from the replication task:
<code>
Test Endpoint failed: Application-Status: 1020912, Application-Message: Cannot connect to ODBC provider ODBC general error., Application-Detailed-Message: RetCode: SQL_ERROR SqlState: HY000 NativeError: 2003 Message: [unixODBC][MySQL][ODBC 5.3(w) Driver]Can't connect to MySQL server on 'foobar.ap-southeast-2.rds.amazonaws.com' (110) ODBC general error.
</code>
Comment: Hi patchiboy, if the response below answered your question please upvote and accept it. That's the ServerFault's way to say thank you for the time and effort someone took to help you. Thanks!
Answer: Couple of possible reasons come to mind...

You talk about SQL Server (= Microsoft SQL Server) but the error message is about MySQL - are you sure you've got the correct engine configured?
I assume you use DMS (Database Migration Service) - if your DMS instance is in a VPC does the VPC have connectivity to the RDS? Is the DMS in the same VPC as the RDS?
Does the RDS security group permit access for the DMS instance?

I bet it's got something to do either with network configuration or some mismatch in the DMS vs RDS configuration. It's hard to tell with the information provided.
It may be beneficial if you post more details on how is the RDS configured (e.g. screenshot from the console) as well as how is the DMS instance configured (again a screenshot), showing the subnets, VPCs, etc. Otherwise it's hard to tell.
Another thing you can try is to spin up a small instance in the same subnet where DMS is, using the same security group as DMS has, login to it and try to access the RDS instance from there.

If it works it will be a problem with the DMS-vs-RDS configuration.
If it doesn't work it will be a networking problem.

Hope that helps :)
Comment: fwiw, the `(110)` code in the error message is `Connection timed out`.
Comment: @Michael-sqlbot thanks, that points to a network issue then.
Comment: For me, it was the third point: security group issue. Forgot to add DMS security group to Inbound Rules. Thanks


How to prove $x/2 > 2\log_2(x) +3$

Question: For sufficient large $x,  x/2 > k \log_2(x) $, here $k$ is conatant  but How can I show this rigorously?
The main problem is to show:  $x/2 > 2\log_2(x) +3$ after a sufficient large $x=x_0$, how can I prove this?
Comment: Number theory?  Calculus seems easier.  With $f(x)=\frac 12 x -2\log_2(x)-3$ can you show that $f(x)$ is, eventually, increasing?
Answer: First, look at the derivatives of each side of the reduced inequality: $\frac x2$ to $\frac12$, $2\log_2x+3$ to $\frac2{x\ln2}$. Clearly $\frac12>\frac2{32\ln2}$, so $\frac x2$ grows faster than $2\log_2x+3$ for $x>32$. Both these functions are strictly increasing, and the reduced inequality holds for $x=32$, so the reduced inequality, and thus the original inequality, must hold for sufficiently large x.
Answer: Let's define $f(x) = \frac{x}{2} - 2 \log_2(x) -3$
We have $f'(x) = \frac{1}{2} - \frac{2}{x\ln(2)}$
So, for $x> \frac{4}{\ln(2)}, f'(x) > 0$. This imply that $f$ is increasing for $x> \frac{4}{\ln(2)}$
But $f(30) = 15 - 2 \log_2(30)-3>0$, hence, $\forall x > 30, \frac{x}{2} > 2\log_2(x)+3$
Answer: The function $f(x)=\log_2(x)$ is concave and the line $g(x)=\frac x2$ cuts $f$ in the two points $(2,1)$ and $(4,2)$. It follows that for all $x\gt 4$ one has
$$\log_2(x)\lt \frac x2$$ You can take many examples for $x_0$ and $k$, for example $x_0=16$ so you have $\log_2(16)=4$ and $\frac{16}{2}=8\gt4+3$, hence for $x\gt 16$ you can take $k=1$

How to prove $x/2 > 2\log_2(x) +3$


How to interpret if a COX hazard ratio is dependent on 2 variables

Question: Hi so I'm interested in calculating HR for the following variables.
age, state and sex. Univariate calculation using R survival package for example looks something like this.
coxph( Surv(
    as.numeric(x[ ,time] ),
    as.numeric(x[ ,censor])
)~x$state , method="exact", data=x)
# if I run this analysis separately I get the following data

           beta HR (95% CI for HR) wald.test p.value
sex        -0.16    0.85 (0.62-1.2)      0.91    0.34
Age        0.0098         1 (0.99-1)       1.4    0.25
state      0.36          1.4 (1-2)       4.3   0.038

The thing is, if I run a multivariate analysis I get the following.
coxph( s_obj ~ x$state + x$sex+ as.numeric ( x$age) , method="exact", data=x)
                            coef exp(coef)  se(coef)      z      p
x$state                 0.404596  1.498696  0.177897  2.274 0.0229
x$sexm             -0.153171  0.857983  0.169687 -0.903 0.3667
as.numeric(x$age)  0.011899  1.011970  0.008561  1.390 0.1646

Likelihood ratio , p=0.04906

I'm trying to interpret this data. My hypothesis is that survival should be dependent on both state and sex.  A few questions, should I just dummy code state+sex and run a univariate?  Does the data above suggest that state alone independent of sex, effect survival?  If I want to truly show that survival is dependent on both state and sex, what would be the best way to do this?
Answer: Cox regressions and logistic regressions can have significant omitted variable bias. Omitting a predictor that is related to outcome from such models can bias the coefficients of the predictors that are included even if they are not correlated with the omitted predictor. This answer is one of several on this site that note this problem.
So it is generally unwise to do multiple single-predictor Cox or logistic regressions. The single Cox multiple regression is the best way to go.
That said, have you checked your model for linearity in the numeric predictors (Age and state) and for proportional hazards? Those tests are important for interpreting your results reliably. And how many events do you have? You don't want to have too few events for the number of predictors that you are evaluating.
Comment: thanks; there are 466 events.  Assuming that the data meets the criteria I still a bit confused as to how to interpret the multivariate results though - how would I go about testing whether survival is dependent on both sex and state?
Comment: @Ahdee you would look at the coefficients for them in the multiple regression model. `state` appears to be significant but `sex` doesn't seem to be; the whole model, however, is barely significant. If you think that `state` could have different effects on outcome depending on `sex` you could consider adding an interaction of `sex:state` to your model. A significant interaction term would imply that they are both important, but in a way that depends on the value of the other. With 466 events you shouldn't have to worry about overfitting when you add an interaction term.
Comment: great idea! I deleted a previous comment and added this edit: interaction not sig however when I added in another important covariate the interaction term  is now < .01 however the rest of the variable include state or sex on its own is not and the overall wald's state is also signficant.  Can I now safely say that the survival is dependent on the both of the variable and not each one on its own when taken into consideration the set of covariates I provided?  thanks.
Comment: @Ahdee the "not each one on its own" part of the statement can be misleading. As you set this up, the individual coefficient reported for `state` is its relation to outcome for females, and the coefficient reported for `sexm` is its relation to outcome when `state`=0. If your values for `state` are all well above 0, then that individual `sexm` coefficient doesn't represent a situation that occurs in practice. At typical `state` values you might have found a significant coefficient for `sexm`. That the relations of these two to outcome depend on each other's values, however, is correct.
Comment: thank you that was well put and it also helped me to understand more intuitively the changes in pvalue and risk as the covariates are added.


como acessar o json?

Question: Acho que a coisa mais simples é acessar uma chave e um valor no json, porém a semanas eu não estou conseguindo fazer isso... ele retorna o json certinho pelo ajax mas quando vou tentar acessar não da.
Obs: sou meio leigo e estou aprendendo.
segue codigo:
function getjson(){
console.log('Recuperando post');
$.post("sistema/getpost.php", 'get',
    function(post){
        // aqui é a cagada que estou fazendo (quando tiro o stringify ele nem aparece)
        var obj = JSON.stringify(post);

        var cont = obj.conteudo;
        var color = obj.cor;
        var like = obj.curtidas;
        var follow = obj.follow;
        var situacao = obj.situacao;

        // esse é o card onde contera os dados
        var posthtml = '

'+cont+'

'+follow+'

<button class="mdl-button mdl-button--colored ">chat_bubble523</button><button class="mdl-button mdl-button--colored " >favorite'+like+'</button>

';

        // aqui é em qual das paginas ele colocara o card
        $( "#tab-1" ).append(posthtml);

    }); }

Obs2: quando eu estava passando html direto não dava esse erro mas ai me disseram que não é viavel e nem funcional fazer isso ai eu mudei para ajax com json.
Agradeço qualquer ajuda
Comment: Não é `stringify` mas sim `parse`...
Answer: O JSON tem dois métodos:

stringify, para transformar um objeto com Primitivos numa String
parse, para transformar uma String em formato JSON num objeto

Assim o que deves usar é .parse() e não .stringify() pois o servidor retorna uma String JSON e não um objeto.
Contudo podias usar $.getJSON e simplificas a coisa:
$.getJSON('sistema/getpost.php', 'get', function(obj){
    var cont = obj.conteudo;
    // etc...

como acessar o json?


Converting expression to summation

Question: Apologies but I am having difficulty understanding how to convert a simple expression to a summation.
I have three parameters:
<code>x = final rating
y = major criteria
z = minor criteria
</code>
The final rating is the sum of the major criteria (which is multipled by 2) and the minor criteria. So something like: $x = 2y + z$
But there can be multiple $y$ and $z$. For example, if I had 2 major criteria and 3 minor criteria, the expression would look like: $x = 2y{_1}+2y{_2}+z{_1}+z{_2}+z{_3}$
So would the summation look like this?
$$x = \sum_{n=0}{2y_n + z_n}$$
Comment: Where is your upper bound?
Comment: Unless the number of major and minor criteria are the same, you need 2 summations. Also I'd take the $2$ out of the "major" summation, it can just be put infront. Lastly, you need at upper bounds on the summations.
Answer: the summation would look like:
$$x=2\sum_{i=1}^m y_i+\sum_{j=1}^n z_j$$
where $m$ is the number of major criteria and $n$ is the number of minor criteria. the lower limit starts at 1.
Comment: Thank you so much.

Converting expression to summation


Are points of the form $\overline{x} = (1,x_2)$ extremes of $f(x) = (x_1-1)^2x_2$?

Question:
Consider the function $$f(x) = (x_1-1)^2x_2$$
Consider the points of the form $\overline{x} = (1,x_2)$
a) Analyze the optimialty of first and second order for these points.
b) What we can say about $\overline{x}$ using these informations?
c) Use the expression of the function to obtain more conclusive
  informations about the characteristics of $\overline{x}$

a)
$$\frac{\partial f}{\partial x_1} = 2(x_1-1)x_2$$
$$\frac{\partial f}{\partial x_2} = (x_1-1)^2$$
$$\nabla f(1,x_2) = 0$$
$$\frac{\partial^2 f}{\partial x_1^2} = 2x_2$$
$$\frac{\partial^2 f}{\partial x_2^2} = 0$$
So the hessian at $(1,x_2)$ is
$$H(1,x_2) =   \begin{bmatrix}
    x_2 & 1  \\
    1 & 0
  \end{bmatrix}$$
$$\begin{bmatrix}
    a & b
  \end{bmatrix}\begin{bmatrix}
    x_2 & 1  \\
    1 & 0
  \end{bmatrix}\begin{bmatrix}
    a  \\
    b
  \end{bmatrix} = a^2 x_2^2 + 2ab$$
b)
this matrix is not definite positive or definite negative, so I cannot say that the points of the form $(1,x_2)$ are extreme points but I also cannot say they are not.
c) I think that this item is asking me to use the function to prove $(1,x_2)$ are not extreme points. Let's analyze $(1,x_2) + \lambda (d_1,d_2)$
$$f(x) = (1+\lambda d_1 -1)^2 (x_2+\lambda d_2) = \lambda^2 d_1^2 (x_2+\lambda d_2)$$
But small variations on $\lambda$ do not change the signal of $f$
Answer: $$\frac{\partial^2 f(x_1, x_2)}{\partial x_1 \partial x_2}=2(x_1-1)$$
$$\frac{\partial^2 f(1, x_2)}{\partial x_1 \partial x_2}=0$$
$$H(1,x_2) = \begin{bmatrix} 2x_2 & 0 \\ 0 & 0 \end{bmatrix}$$
Same conclusion as you for the front part.
Now to analyze its optimality.
Consider $x_2 > 0$. Then, we can draw a ball around $(1,x_2)$ such that the $y$-coordinate is always positive. Hence $f$ is nonnegative on that ball, hence it is a local minimum.
Similarly for $x_2<0$, we can draw a ball such that the $y$-coordinate is always negative, hence $f$ is nonpositive on that neighborhood. Hence it is a local maximum.
For $(1,0)$, whenever we draw a ball, we can always find $x_1 \ne 0$ and it is possible for $x_2$ to be positive or negative, hence it is neither local maximum nor minimum.
Comment: For $x_2>0$, if we think about $(1+h-1)^2(x_2+h)$ I can see that there's always $h$ that makes this positive, because I can always pick $h$ such that $00$. However I'm not convinced that this is a ball. It looks like that passes through $(1,x_2)$
Comment: $(1+h_1-1)^2(x_2+h_2)$, pick $h_1$ and $h_2$ such that $h_1^2 +h_2^2 < x_2^2$.


Airflow Dynamic Task Creation Stream Setting Not Working

Question: I have a complex DAG that basically repeats it's stream six times for six different sources. So I've been using a for loop to dynamically create my streams like this (small example):
sources = ['source_1', 'source_2', 'source_3', 'source_4', 'source_5', 'source_6']

for source in sources:

    source_task_1 = PythonOperator(
        task_id=source + '_create_emr',
        dag=dag,
        provide_context=True,
        retries=10,
        python_callable=execute_arl_source_emr_creation,
        op_args=[source])

    source_task_2 = BashOperator(
        task_id=source + '_starting_step_1',
        retries=10,
        bash_command='echo "Starting step 1 for ' + source + '"',
        dag=dag)

    source_task_2.set_upstream(source_task_1)

All of the tasks are successfully being created in the DAG because I can see them all on the Airflow UI but the strange thing is it's only linking the tasks in the stream for the first occurrence in the loop (source_1).
All of the other tasks have no upstream or downstream on them. I don't understand how this is possible since the first occurrence in the loop worked shouldn't they all work?
Here is the actual code I have (it's a very large DAG so I'm only showing the tasks not the python callable functions I am using inside the tasks...):
def create_emr_step_3_subdag(main_dag, subdag_id, source):
    subdag = DAG('{0}.{1}'.format(main_dag.dag_id, subdag_id), default_args=args)

    source_run_emr_step_3 = PythonOperator(
        task_id=source.sourceFullName + '_run_emr_step_3',
        dag=subdag,
        provide_context=True,
        retries=0,
        python_callable=execute_emr_step_3,
        op_args=[source_create_emr_task_id, source.sourceFullName])

    source_run_emr_step_3_waiter = PythonOperator(
        task_id=source.sourceFullName + '_run_emr_step_3_waiter',
        dag=subdag,
        provide_context=True,
        retries=10,
        python_callable=execute_emr_step_3_waiter,
        op_args=[source_create_emr_task_id, source.sourceFullName])

    source_run_emr_step_3_waiter.set_upstream(source_run_emr_step_3)

    return subdag

class DatalakeDigitalPlatformArlWorkflowSource:
    sourceShortName = None  # source_1, source_2, source_3, source_4, source_5, source_6
    sourceFullName = None  # SOURCE_1, SOURCE_2, SOURCE_3, SOURCE_4, SOURCE_5, SOURCE_6

    def getSourceShortName(self):
        return self.sourceShortName

    def setSourceShortName(self, sourceShortName):
        self.sourceShortName = sourceShortName

    def getSourceFulltName(self):
        return self.sourceFullName

    def setSourceFullName(self, sourceFullName):
        self.sourceFullName = sourceFullName

source_1 = DatalakeDigitalPlatformArlWorkflowSource()
source_1.setSourceShortName("source_1")
source_1.setSourceFullName("SOURCE_1")

source_2 = DatalakeDigitalPlatformArlWorkflowSource()
source_2.setSourceShortName("source_2")
source_2.setSourceFullName("HZN")

source_3 = DatalakeDigitalPlatformArlWorkflowSource()
source_3.setSourceShortName("source_3")
source_3.setSourceFullName("SOURCE_3")

source_4 = DatalakeDigitalPlatformArlWorkflowSource()
source_4.setSourceShortName("source_4")
source_4.setSourceFullName("SOURCE_4")

source_5 = DatalakeDigitalPlatformArlWorkflowSource()
source_5.setSourceShortName("source_5")
source_5.setSourceFullName("PP")

source_6 = DatalakeDigitalPlatformArlWorkflowSource()
source_6.setSourceShortName("source_6")
source_6.setSourceFullName("SOURCE_6")

sources = [source_1, source_2, source_3, source_4, source_5, source_6]

for source in sources:
    source_create_emr_task_id = source.sourceFullName + '_create_emr'

    source_create_emr = PythonOperator(
        task_id=source_create_emr_task_id,
        dag=dag,
        provide_context=True,
        retries=10,
        python_callable=execute_blah_source_emr_creation,
        op_args=[source_create_emr_task_id, source.sourceFullName])

    # source_starting_step_1
    source_starting_step_1 = BashOperator(
        task_id=source.sourceFullName + '_starting_step_1',
        retries=10,
        bash_command='echo "Starting step 1 for ' + source.sourceShortName + '"',
        dag=dag)

    # Get source Batch ID
    source_get_batch_id = PythonOperator(
        task_id=source.sourceFullName + '_get_batch_id',
        retries=10,
        dag=dag,
        python_callable=get_batch_id,
        op_args=[airflow_home + '/resources/batch-id-inputs/batchid_input.json', source.sourceFullName])

    # source_licappts
    source_sensor_licappts = OmegaFileSensor(
        task_id=source.sourceFullName + '_sensor_licappts',
        retries=10,
        filepath=airflow_home + '/foo/data/bar/blah/test/data',
        filepattern=source.sourceShortName + '_licappts_(.*).txt',
        poke_interval=3,
        execution_timeout=timedelta(hours=23),
        dag=dag)
    source_process_licappts = PythonOperator(
        task_id=source.sourceFullName + '_process_licappts',
        retries=10,
        dag=dag,
        python_callable=execute_d_landing_import,
        op_args=[source.sourceShortName + '_licappts_(.*).txt', 'get' + source.sourceFullName + 'BatchId'])

    # source_agents
    source_sensor_agents = OmegaFileSensor(
        task_id=source.sourceFullName + '_sensor_agents',
        retries=10,
        filepath=airflow_home + '/foo/data/bar/blah/test/data',
        filepattern=source.sourceShortName + '_agents_(.*).txt',
        poke_interval=3,
        dag=dag)
    source_process_agents = PythonOperator(
        task_id=source.sourceFullName + '_process_agents',
        retries=10,
        dag=dag,
        python_callable=execute_d_landing_import,
        op_args=[source.sourceShortName + '_agents_*.txt', 'get' + source.sourceFullName + 'BatchId'])

    # source_agentpolicy
    source_sensor_agentpolicy = OmegaFileSensor(
        task_id=source.sourceFullName + '_sensor_agentpolicy',
        retries=10,
        filepath=airflow_home + '/foo/data/bar/blah/test/data',
        filepattern=source.sourceShortName + '_agentpolicy_(.*).txt',
        poke_interval=3,
        dag=dag)
    source_process_agentpolicy = PythonOperator(
        task_id=source.sourceFullName + '_process_agentpolicy',
        retries=10,
        dag=dag,
        python_callable=execute_d_landing_import,
        op_args=[source.sourceShortName + '_agentpolicy_*.txt', 'get' + source.sourceFullName + 'BatchId'])

    # source_finished_step_1
    source_finished_step_1 = BashOperator(
        task_id=source.sourceFullName + '_finished_step_1',
        retries=10,
        bash_command='echo "Finished step 1 for ' + source.sourceShortName + '"',
        dag=dag)

    # source_starting_step_2
    source_starting_step_2 = BashOperator(
        task_id=source.sourceFullName + '_source_starting_step_2',
        retries=10,
        bash_command='echo "Starting step 2 for ' + source.sourceShortName + '"',
        dag=dag)

    source_run_emr_step_2 = PythonOperator(
        task_id=source.sourceFullName + '_run_emr_step_2',
        dag=dag,
        provide_context=True,
        retries=0,
        python_callable=execute_emr_step_2,
        op_args=[source_create_emr_task_id, source.sourceFullName])

    source_run_emr_step_2_waiter = PythonOperator(
        task_id=source.sourceFullName + '_run_emr_step_2_waiter',
        dag=dag,
        provide_context=True,
        retries=10,
        python_callable=execute_emr_step_2_waiter,
        op_args=[source_create_emr_task_id, source.sourceFullName])

    # source_elastic_search_check
    source_elastic_search_check = PythonOperator(
        task_id=source.sourceFullName + '_elastic_search_check',
        retries=10,
        dag=dag,
        python_callable=execute_get_advisor_batch_stage_status,
        op_args=['get' + source.sourceFullName + 'BatchId', source.sourceFullName])

    # source_finished_step_2
    source_finished_step_2 = BashOperator(
        task_id=source.sourceFullName + '_finished_step_2',
        retries=10,
        bash_command='echo "Finished step 2 for ' + source.sourceShortName + '"',
        dag=dag)

    # source_starting_step_3
    source_starting_step_3 = BashOperator(
        task_id=source.sourceFullName + '_starting_step_3',
        retries=10,
        bash_command='echo "Starting step 3 for ' + source.sourceShortName + '"',
        dag=dag)

    source_emr_step_3_subdag_task_id = source.sourceFullName + '_emr_step_3_subdag'

    source_emr_step_3_subdag = SubDagOperator(
        task_id=source_emr_step_3_subdag_task_id,
        dag=dag,
        retries=10,
        pool='entitymatching_task_pool',
        subdag=create_emr_step_3_subdag(dag, source_emr_step_3_subdag_task_id, source)
    )

    # source_finished_step_3
    source_finished_step_3 = BashOperator(
        task_id=source.sourceFullName + '_finished_step_3',
        retries=10,
        bash_command='echo "Finished step 3 for ' + source.sourceShortName + '"',
        dag=dag)

    # source_starting_step_4
    source_starting_step_4 = BashOperator(
        task_id=source.sourceFullName + '_starting_step_4',
        retries=10,
        bash_command='echo "Starting step 4 for ' + source.sourceShortName + '"',
        dag=dag)

    source_run_emr_step_4 = PythonOperator(
        task_id=source.sourceFullName + '_run_emr_step_4',
        dag=dag,
        provide_context=True,
        retries=0,
        python_callable=execute_emr_step_4,
        op_args=[source_create_emr_task_id, source.sourceFullName])

    source_run_emr_step_4_waiter = PythonOperator(
        task_id=source.sourceFullName + '_run_emr_step_4_waiter',
        dag=dag,
        provide_context=True,
        retries=10,
        python_callable=execute_emr_step_4_waiter,
        op_args=[source_create_emr_task_id, source.sourceFullName])

    # source_finished_step_4
    source_finished_step_4 = BashOperator(
        task_id=source.sourceFullName + '_finished_step_4',
        retries=10,
        bash_command='echo "Finished step 4 for ' + source.sourceShortName + '"',
        dag=dag)

    source_emr_termination = PythonOperator(
        task_id=source.sourceFullName + '_emr_termination',
        dag=dag,
        provide_context=True,
        retries=10,
        retry_delay=timedelta(minutes=5),
        python_callable=execute_emr_termination,
        op_args=[source_create_emr_task_id, source.sourceFullName])

    # source_successful
    source_successful = BashOperator(
        task_id=source.sourceFullName + '_successful',
        retries=10,
        bash_command='sudo aws sns publish blah blah blah',
        dag=dag)

    # finished_foo_bar_blah_workflow
    finished_foo_bar_blah_workflow = BashOperator(
        task_id='finished_foo_bar_blah_workflow',
        bash_command='echo "Finished foo_bar_blah_workflow"',
        dag=dag)

    ### Stream ###

    # Create EMR Cluster
    source_create_emr.set_upstream(starting_foo_bar_blah_workflow)

    # Step 1
    source_starting_step_1.set_upstream(starting_foo_bar_blah_workflow)
    source_get_batch_id.set_upstream(source_starting_step_1)

    source_sensor_licappts.set_upstream(source_get_batch_id)
    source_process_licappts.set_upstream(source_sensor_licappts)

    source_sensor_agents.set_upstream(source_get_batch_id)
    source_process_agents.set_upstream(source_sensor_agents)

    source_sensor_agentpolicy.set_upstream(source_get_batch_id)
    source_process_agentpolicy.set_upstream(source_sensor_agentpolicy)

    source_finished_step_1.set_upstream(source_process_licappts)
    source_finished_step_1.set_upstream(source_process_agents)
    source_finished_step_1.set_upstream(source_process_agentpolicy)

    # Step 2
    source_starting_step_2.set_upstream(source_finished_step_1)
    source_starting_step_2.set_upstream(source_create_emr)  # Don't run EMR steps until the EMR is created
    source_run_emr_step_2.set_upstream(source_starting_step_2)
    source_run_emr_step_2_waiter.set_upstream(source_run_emr_step_2)
    source_elastic_search_check.set_upstream(source_run_emr_step_2_waiter)
    source_finished_step_2.set_upstream(source_elastic_search_check)

    # Step 3
    source_starting_step_3.set_upstream(source_finished_step_2)
    source_emr_step_3_subdag.set_upstream(source_starting_step_3)
    source_finished_step_3.set_upstream(source_emr_step_3_subdag)

    # Step 4
    source_starting_step_4.set_upstream(source_finished_step_3)
    source_run_emr_step_4.set_upstream(source_starting_step_4)
    source_run_emr_step_4_waiter.set_upstream(source_run_emr_step_4)
    source_finished_step_4.set_upstream(source_run_emr_step_4_waiter)

    # Terminate EMR Cluster
    source_emr_termination.set_upstream(source_finished_step_4)
    source_successful.set_upstream(source_emr_termination)
    finished_foo_bar_blah_workflow.set_upstream(source_successful)

And as you can see the streaming is not working

Before my recent modifications to the file it was working just fine and can be seen here

I just did a ton of refactoring on my code and when I reloaded it I saw this error. I'm not sure what I did but I did a lot of find + replace all to rename things and I'm wondering if I messed something up in that process and maybe I just don't see the error in the code. But what makes me think that isn't the issue is because if that were the issue then why would my first source be correctly linked in it's stream?
Is it possible I've hit some sort of limit on the amount of tasks I can have in a single DAG?
Comment: I haven't been able to wrap my head around how this is even possible. If you think about it because the tasks are obviously all being created correctly (you can clearly see them all there) and so the for loop is working. Inside that for loop is the stream connections. There is no reason that code isn't being called. So I think this is an Airflow bug of some sort maybe. As a result I've had to eliminate the dynamic task creation from the for loop and manually put the code in for every source; something that grew my code by 1,398 lines.... not cool Airflow...
Comment: UPDATE: I just manually put the code in for every source and the same exact thing happened. The first source correctly links it's stream but the rest have no stream connections. This further validates my suspicion that this is an Airflow bug of some sort. Maybe I have too many tasks in one DAG? I'm going to try putting every source into it's own subdag and then see if that helps.
Answer: I think I found your mistake:
First, to rule out that this is not an airflow bug, I created a small DAG that creates 25 tasks for 7 sources and sets the upstreams, and everything works fine.
So I took your code, tried it and I experienced the exact same problem that you see.
Now I went ahead, commented out all the steps where you set the upstreams and put them back in step by step, and everything works fine until one puts in the very last line:
finished_foo_bar_blah_workflow.set_upstream(source_successful)

So I looked at the task finished_foo_bar_blah_workflow and as far as I can see, this task only needs to get created once, not for every source. So I put the code
# finished_foo_bar_blah_workflow
finished_foo_bar_blah_workflow = BashOperator(
    task_id='finished_foo_bar_blah_workflow',
    bash_command='echo "Finished foo_bar_blah_workflow"',
    dag=dag)

above the for source in sources: line, and voila, everything works fine.
Edit
So I looked into the upstream and downstream lists for the tasks that are supposed to be directly upstream of finished_foo_bar_blah_workflow and while finished_foo_bar_blah_workflow has only one task in the upstream_list (before I moved the code that creates that task, afterwards it correctly contains 7 tasks), all the task that are supposed to be directly upstream of that task, reference it in their downstream_list and and their upstream_list contains the tasks that should be there too. So this might be a bug concerning creating a task with the same task_id multiple times.
Comment: You fixed it man. Thank you so much. I can't believe I missed this... It seems so obvious now.


Quebra de linha em tabela HTML

Question: Gostaria de saber como posso fazer para quebrar a linha nesta tabela na versão mobile do site (@media screen and (max-width: 550px)):

Qnd. (ex: 01 un)
Comp. (ex: 1,2m)
Larg. (ex: 2,0m)
Alt. (ex: 1,5m)
Peso (ex: 3,0 kg)

<input type="text" onchange="calc()"></input>
<input type="text" onchange="calc()"></input>
<input type="text" onchange="calc()"></input>
<input type="text" onchange="calc()"></input>
<input type="text" onchange="calc()"></input>
<button onclick="this.parentElement.parentElement.outerHTML='';calc();">X</button>

Comment: A princípio a tag `input` não tem tag de fechamento ``.
Comment: Yuri não ficou muito claro, vc quer quebrar a linha dentro do input, quer quebrar as linha da THead, p que vc que dividir em outras linhas?
Answer: Quebrar linhas em uma tabela de forma responsiva é complicado porque a própria estrutura da tabela não permite isso.
Você poderia adicionar um display: block às células fazendo com que cada uma ocupe uma linha diferente, mas não creio que o resultado fique muito bom, veja:
É preciso visualizar em tela cheia e redimensionar a janela para menos de 550px.

@media screen and (max-width: 550px){
  td, th{
    display:block;
  }
}

Qnd. (ex: 01 un)
Comp. (ex: 1,2m)
Larg. (ex: 2,0m)
Alt. (ex: 1,5m)
Peso (ex: 3,0 kg)

<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<button onclick="this.parentElement.parentElement.outerHTML='';calc();">X</button>

<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<button onclick="this.parentElement.parentElement.outerHTML='';calc();">X</button>

Como a sua tabela não possui muitas colunas (apenas 6), minha sugestão é torná-la responsiva, se auto-ajustando à resolução da tela. Basta adicionar no CSS input: width: 100%;:

input{
   width: 100%;
}

Qnd. (ex: 01 un)
Comp. (ex: 1,2m)
Larg. (ex: 2,0m)
Alt. (ex: 1,5m)
Peso (ex: 3,0 kg)

<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<button onclick="this.parentElement.parentElement.outerHTML='';calc();">X</button>

<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<input type="text" onchange="calc()">
<button onclick="this.parentElement.parentElement.outerHTML='';calc();">X</button>

Uma correção no seu código, é que a tag <input> não possui tag de
  fechamento </input>.

Quebra de linha em tabela HTML


Equivalence relations, Cosets

Question: Let G be a group and for elements a,b (elements of)G let a R b mean that there exists an element x(element of)G such that a=xbx^(-1). Show that R is an equivalence relation on G.
Not really sure how to go about this one.
Comment: Arnold: you've asked $17$ questions now at MSE, but you haven't accepted any answers? FYI: It is considered good site etiquette (good manners) to accept a helpful answer after posting a question and receiving answers. You can accept only one answer per question asked, but shortly you will also be able to *upvote* as many helpful answers as you'd like. To *accept* an answer, simply click on the grey $\large \checkmark$ to the left of the answer you'd like to accept. It turns green when you click on it, and you receive $2$ reputation points each time you accept an answer.
Answer: You need to check the three conditions for an equivalence relation:

Check that $x \sim x$ under your relation $R$. So is there an element $a$ in every group $G$ such that $x=a x a^{-1}$?
Now check that if $x \sim y$, that is if $x=aya^{-1}$ that $y=bxb^{-1}$ for some $b \in G$. For this try solving for $y$ and then relabeling to find your $b$'s in terms of your $a$'s. Then have you found elements that make this work?
Now check that if $x \sim y$, that is if $x=aya^{-1}$ and $y \sim z$, that is if $y=bzb^{-1}$, that $x \sim z$. That means that $x=czc^{-1}$ for some $c \in G$. Try solving for $z$ in terms of $b$ and $y$ and then substituting that in your equation for $x \sim y$. Once you have that equation, can you find an element $c$ in terms of $a,b$ so that you have $x \sim z$?

Equivalence relations, Cosets


Calculus calculating work/energy

Question:
Above is a picture of the diagram I was given. I was told to find the amount of energy produced when the basin is filled. I believe that work is $W = fd$ where $f$ is weight. (please correct me on anything I got wrong, I am struggling to learn this). I then got the integral of $\frac{x^2}{40000}$ and from $x = 0$ to $x = 1000$ and multiplied that by $500$. I believe that this number is $f$ correct? My problem is finding $d$. I don't really know what it means by distance. I noticed most textbooks change the function into a function of $y$. Why is that? Any help would be greatly appreciated.
I just looked at the question again and I believe the only thing I missed is that the density is 64kg/cubic ft
Comment: Can you post the full text of the question? I think something might be missing, at a glance.
Answer: In following figure, that is basin which is full of water (assumption), consider cross section of it, and we want to find the volume of water, in layer of height $y$ (red area). This $y$ is determined with a $\dfrac{x}{40000}$, that's the length of the pool.
Let $\Delta y$ be the height of this layer, we choose it so thin that the slope of curve (bottom of basin), could be ignore. In this case the volume of water in this red part is
$$\Delta V=500ft.x.\Delta y=500.\sqrt{(4\times10^4.y)}.\Delta y$$
This volume of water has amount the potential energy
$$\Delta W=f.d=(\Delta M.g)y=\rho.\Delta V.g.y$$
The total work of water in the pool can be done is
$$W=\int_0^{25}10^5\rho.g.y^{\frac32}dy$$
Comment: Thank you very much for your reply. Could you please explain the notation a little bit? Mainly the x and delta y's. Thank you.

Calculus calculating work/energy


What "ladder sub {...}" means in perl?

Question: I'm reading tweetylicious source from github to study Mojolicious framework:
However I'm confused by below piece of code ladder sub .... What does it mean in Perl? It looks like not a regular Perl grammar.
Btw, I'm with Strawberry Perl 5.
# The rest of the routes are specific to logged in users, so we
# add a ladder to make sure (instead of making sure inside each route)
ladder sub {
    my $self = shift;
    return 1 if $self->session('name');
    $self->redirect_to('/login') and return;
};

Answer: It is a call to a subroutine called ladder that expects a code reference as its first argument. It is equivalent to
$tmpfunc = sub {
    my $self = shift;
    return 1 if $self->session('name');
    $self->redirect_to('/login') and return;
};
ladder($tmpfunc);

Comment: @Hao the code you're looking at is from 2011. The Mojolicious project is quite active and the API changes frequently. I've searched through the github repo and I can only find two references to `ladder`, when it was added in 2010. It has since been removed, but the changelog doesn't reveal when. I think you should look for more modern examples.
Comment: @Hao I went on IRC and asked in #mojolicious. The creator of the project says it probably went out in 2010 before version 1.0. So I would say it's safe to assume we shouldn't be using it today. The modern equivalent is called `under`, btw.
Comment: Thanks for answering. But how I fix the compilation error? I tried change  it to the example style, but then I blocked by cgi.pm:has description => 'Start application with CGI';
has usage => sub { shift->extract_usage };
Comment: @simbabque Got. Thanks for your time.
Comment: @Hao thank you. We're all just people here. I hope you're having fun learning Perl although you're experiencing a bit of a setback right now! :)

What "ladder sub {...}" means in perl?


Catch block not execting in C++ program

Question: I am running a c++ program with a catch block for all exceptions under Snow Leapard. The program is:
<code>#include <iostream>

using namespace std;

const int DefaultSize = 10;

int main()
{
   int top = 90;
   int bottom = 0;

   try
   {
      cout << "top / 2 = " << (top/ 2) << endl;

      cout << "top divided by bottom = ";
      cout << (top / bottom) << endl;

      cout << "top / 3 = " << (top/ 3) << endl;
   }
   catch(...)
   {
      cout << "something has gone wrong!" << endl;
   }

   cout << "Done." << endl;
   return 0;
}
</code>
I am using g++ version i686-apple-darwin10-g++-4.2.1 to compile the program: g++ -o test test.cpp.  The catch block is not executing. The output is: "top / 2 = 45
Floating point exception". No error message or Done is printed. The output is exactly the same as when the try-catch block is not present. Can someone tell me why the catch is not executing? I have gotten user-defined exceptions to work.
Answer: there is no default zero divide error in c++. So, you need to check if the bottom value is 0
<code>try{
    if(bottom==0)
         throw bottom; //Or any integer
   }

catch(int x)
{
     cout<<"Something has gone wrong";
}
</code>

Catch block not execting in C++ program


Coin tossing problem where three tails come in a row

Question: The problem is if I want to toss a coin n times, what is the number of ways possible when three tails comes in a row?
For example if I toss a coin for 4 times, the number of ways where 3 tails come in row is 3 i.e. TTTT, TTTH, HTTT. I need a generalized explanation for this.
Comment: Suggestion for finding the answer on the internet. Work out the answer for $n=3,4,5,6,7$ (easy to do by hand) and look for the sequence at http://oeis.org.
Comment: Didn't an almost identical question come up a day ago?
Answer: This is a classical problem.
We instead count the complement: The number of ways to have it so that no three tails are in a row. Later we subtract from $2^n$, the total number of sequences. This we can do by recursion. Let $x_n$ be the number of sequences of $n$ flips without TTT. For $n=0,1,2$, our answer is $1,2,4$, respectively.
If $n\geq 3$, note that the sequence ends with one of H, HT, HTT, since the last H must be one of the last 3 flips, or else the last 3 flips are all tails, which is not allowed.
Furthermore, appending H, HT, or HTT to a valid sequence produces another valid sequence. So, if it ends with H, there are $x_{n-1}$ ways to pick the first $n-1$ flips; if it ends with HT, there are $x_{n-2}$ possibilities for the first $n-2$ flips; and if it ends with HTT, we have $x_{n-3}$ choices.
Hence, $$\boxed{x_n = x_{n-1}+x_{n-2}+x_{n-3}}.$$ You can now use this to generate $x_n$ for any value of $n$, and take $2^n-x_n$ to compute the answer to your question.
Unfortunately, closed forms of this are weird, since in order to solve the recurrence we need the roots of the characteristic polynomial, which are quite disgusting in this case.
Comment: But (n-1)*2^(n-4) this formula works for n>3. Could you please explain this?
Comment: Unfortunately that is not true, although promising; for $n=7$, the answer is 47.


Retrieve argument in hook_menu() item url

Question: On my custom module, I defined the following hook_menu() function.
function motel_events_menu() {

    $items['ajax/pastevents/%'] = array(
        'title' => 'Get Past Events',
        'page callback' => '_motel_events_get_pastevents(%)',
        'access arguments' => array('access content'),
        'type' => MENU_CALLBACK,
    );

    return $items;
}

In this function, how can I retrieve the % argument that is passed in 'ajax/pastevents/%' so I can send it to my _motel_events_get_pastevents() function?
Answer: 'page callback' => '_motel_events_get_pastevents(%)',
should be
'page callback' => '_motel_events_get_pastevents',
then insert
 'page arguments' => array(2),

immediately after.
ajax is at position 0, pastevent is at position 1 and the wildcard % is at position 2. Counting for arguments starts at 0. The callback function will have a variable to pass in the wildcard value argument.
function _motel_events_get_pastevents($arg){

Some folks like to use the arg(n) form for arguments as its always present whether declared or not but I tend to use formal declarations.
Comment: Thank you very much for your reply, this is exactly what I've done.
I didn't say I changed `page callback' => '_motel_events_get_pastevents(%)` to `page callback' => '_motel_events_get_pastevents` in my comment though..
Thanks a lot!
Answer: You don't need to defined % in page callback you get url parameter within your function by calling arg(2).


Disable (JetBrains) code inspections for an entire project

Question: I'd like to disable code inspections for certain projects (like tests or quick-n-dirty proof-of-concept ones).
The section Suppressing code inspections in specific scope goes only up to the file level by suggesting this:

Inspection [name of inspection] | Disable once with comment | Disable all inspection in file - this option inserts a single comment - ReSharper disable All in the beginning of the file. This comment suppresses all inspections the file.

What would I have to do to go even further and do the same but for an entire project?
This means:

not a solution
not a file
not a method
not entirely
not roslyn-analyzers
but just a single project
Answer: I've never used it, but I think this is the closest option you can try:

It's also explained in Jetbrains' website.
Comment: The docs for that feature are [here](https://www.jetbrains.com/help/resharper/Code_Analysis__Configuring_Warnings.html#excluding-frozen-projects-from-indexing) - _"The ignored projects will not be indexed by ReSharper, meaning that no ReSharper features — e.g. code analysis, refactorings, code generation — will be available in those projects."_


Guitar practice

Question: I have noticed that even when I skip a day in my guitar practice my skills drastically go down. Furthermore, my friends are able to improve at guitar faster than I can and it seems that they don't easily lose practice even when they don't play it. I am confused. My guitar playing is always sloppy even when I practice well. How should I fix it? Please guide me on how to improve my skills and practice guitar.
I practice about 3 hours a day and I'm self taught. This has also become hurdle hence. I try to play smoothly yet I'm unable. It's always sloppy
Comment: Keep in mind that everyone has their own speed at learning new things. Dont loose your head because your friends may be faster... If you can provide more details maybe some people here can give you advice on how to improve your learning. How long do you play guitar? Do you take lessons from a teacher or is it self taught? How often and for how long do you practice? Where do you see your main difficulties etc...
Comment: You should get a guitar teacher. That will help make sure that you aren't practicing the wrong things and actually making you worse.
Answer: It's hard to point out whats the problem without seeing you playing. The best advise would be to contact a music teacher. They have the experience in teaching people and know what they are doing (and what you might be doing wrong). But I can give you some general advice on practicing:
Practice time:
Do you practice three hours straight? I think that is way to much. Make your practice sessions shorter. Maybe 20 or 30 minutes should be okay. If you want to practice more, make more sessions, but keep the time of each session about 30 mins and be sure to have enough time between the sessions. Even if practice can be boring some times, it should never be a hurdle to practice. You should have fun practicing.
What to practice:
Don't push yourself too much. Start with simple things and dont overwhlem yourself with complicated stuff. Try to find songs you like. Learning something you enjoy makes the process of learning joyfull.
Sloppy playing:
To play smoothly, you have to be relaxed. If you are trying too hard, you will end up playing sloppy again. Dont stress yourself. Make yourself comfortable when practicing. Dont get annoyed if are not able to play smoothly. It takes a lot of time.
You can try to slow down what you are playing. Like really slow. So slow, that you are able to play it smoothly. If you are able to play it slow but smootly, play it again and again. If I practice something, i have to play it correctly and smoothly like 3 times one after the other. Then I increase the tempo a little bit. Repeat until you have the original tempo.

Use a metronom.
Depending on what you are learning (like a whole song), you can try
to break it down in different parts which you can practice
regardless.
Try to find out what are the things that make you play sloppy,
extract them and practice them individually.
Try to count out loud while playing (1 and 2 and 3 and 4 and) and/or
tap your foot on the ground.

Something which helped me a lot was to seperate the rythm from the technical aspect (like which chords to finger and which pattern to play). By extracting the rythm and learn it by tapping it on my thigh, I was able to internalize it without the complexity of chord fingering etc. This is - at least from my little experience - the most common cause of sloppy playing.
Last but not least don't get annoyed if your progress is not as fast as the progress of your friends. Just dont give up and keep on playing. Sooner or later you will be able to catch up.
Answer: You say you practice 3 hours a day but you didn't for how many years.  That counts for a lot.  If you are self taught then there is no telling how many bad habits you've developed.  Time may not be the biggest issue.  When we practice we reinforce patterns and if, as you say, you are always sloppy and not smooth and you keep practicing in the hopes that it will clean up then you are really reinforcing "sloppy" as a standard.  If you had gone through a graded curriculum with an instructor, or self taught with high quality resources, you might be able to avoid this.
You mention playing guitar "faster".  Speed comes from practicing SLOWLY.  SLOW --> SPEED.  You need to understand that in order to play something fast it needs to be so deeply programmed into the muscle memory that it is automatic, as easy as breathing.  Until you have something memorized to that level attempting speed is useless.  Take a scale run, or a lick from a song, that you are trying to master.  Spend about a week playing it 100s of times a day so slow that you can't play it wrong, e.g. one note per second (that's right 1 per second).  Play with your hands and body relaxed but not totally limp (you still need good posture and correct hand position).  Get it to the point where you can hear it in your head when you're not playing, and can mimic the movements without the guitar.  Then after you've put in several 1000 reps start playing it in time with a metronome.  I should have asked or stated in the first place, use a metronome (are you?).
When it comes to pushing the speed of a piece that you have memorized there is a standard trick. Start at a speed where you can play it without any tension in your body (this is after the week or more spent on drilling it into your memory) and play it a few times at that speed.  Then go up 2bpm and repeat, then down 1bpm and repeat.  Continue on this path until you are 20bpm faster than you started provided that you don't start making mistakes or getting tense.  If you do then stop at whatever speed that happened and make a note of it.  Every day do this same exercise but starting at a higher speed.  If you're luck and you got 20bpm faster without problems you can start the next day at that new speed and push it another 20bpm.  This approach works very well for developing clean and accurate speed on any instrument.  It probably works for dancers too, working in complicated sequences.
The above practice advice assumes that you at least have a solid foundation in the basics.  That you have developed finger independence and have good hand posture, not squeezing too hard, not putting the thumb over the edge of the neck, etc.  It also assume that you have put in some time in developing basic technique.  If you have not developed these basic attributes then your approach may be self defeating.  In learning an instrument like the guitar it is important to divide up your practice time into a few categories.  These might be something like;
(1) Basic hand movement exercises.  This is not even music but the athleticism of guitar.  You can find 1000s of exercises that go back 200 years or so.  The exercises in all modern books are the same as Tarrega published for classical guitar in the 1800s.  Things like hammer on pull off chromatic exercises up the neck, the 1-2-3-4 exercise, "spider" exercises.  You will find exercises for the picking hand too, alternate picking, string skipping, finger picking etc.  Some of these are done on the open strings.  The point of them is to teach your fingers to move correctly and your pick to locate the correct strings without the added burden of trying to play a song.  The guitar is hard enough.  It takes time and patience to get skill on it.
(2) Basic music exercises.  These would be the basic rudiments of music (and probably the same for all instruments), such as the major scale, playing intervals in sequence and together, simple open string chord forms.  Most graded method books, e.g. Carcassi, Mel Bay, Levitt, provide a set of common exercises in each key.  Carcassi has the scale, the I-IV-V progression (or cadence), then a simple scale exercise and an arpeggio exercise, then 1 or 2 simple etudes that are hard enough to be a challenge for the beginner but not so hard that they kill you and not really worth making part of a repertoire.
(3) Simple and complex pieces.  This would be comprised of musical compositions that could be part of a musician's repertoire permanently.  Such pieces would require that the guitarist had at least developed some mastery of the first two categories.  A performance piece might takes several months to learn and get clean.
(4) Techniques and technical exercises.  This can be thought of as a more advanced version of the first and second categories.  The intent of the first category is to get the hand moving they way you want it to and in many cases one doesn't even listen to the guitar.  Classical guitar books will tell you to mute the strings.  You're only "feeling" your way around the instrument.  The technical exercises might involve complex musical patterns that are designed to give you exercise in a specific technique like, alternate picking, consecutive picking, sweeping, slurring, tapping.  These will also work your endurance.
There could be more categories and different guitarists may agree or disagree with my division.  In all of these cases it is important to use a metronome.
A lot of modern "self taught" guitarists never go through any of these training exercises and just go for category (3).  Like a total beginner wants to learn Eruption and thinks if they just try hard the technique will appear one day.  That usually does NOT happen.  Even the greatest guitarists and musicians in the world will tell you that after 40 years of training they begin each practice session with category (1) basic movement exercises.  These transform the body and everything is perishable so it needs to be reinforced forever.  I would not worry if you think that your friends play better than you with less practice.  They may have used their practice time more wisely, have more years experience, have more effects on their guitar, etc.  It may be that you are just frustrated and comparing yourself to other will not make that better.
I would not recommend practicing less, but practicing more intelligently.  Get a routine written on paper and follow it daily.  Pick a set of basic movements as a warm up, a set of scale exercises for practicing music, and a few pieces you want to learn and get better at.  Use the metronome and following 2 up 1 down method for speed.  Make sure you isolate specific movements that give you problems.  Perhaps the songs you play are cleaner than you think but one spot always comes out like crap.  It could be a string skip or a sweep that you're are not good at.  If so then get your hands on some basic movement exercises for that technique and drill them.  Then your song should get cleaner too.
Answer: A good teacher can look at how you play and give you a list of things you are doing wrong in your practice time. Chances are they will inform you that the time spent practicing has been rushed and perhaps unfocused or where the focus has been on quantity of music instead of quality of performance. Everybody practices differently, but there are some things you can learn to do that will help clean up the sloppiness and actually make it easier to play. You can pick and choose what suits you best, and the end result will be determined by those choices.
Answer: One simple but very effective thing you can do is to regularly record yourself while you practice.
Nothing fancy, for example you can use your phone to video-record yourself.
Later, play it back and observe... it may be unpleasant at first, but it will also be extremely useful to see both what you're doing right and what you're doing poorly. Having this sort of feedback will help you brain to learn its lessons, so to speak, much better and faster.
Answer: Here is a simple test and a small thumb rule
Given that you practice 3 hours a day, you have the commitment, so ruling out that issue and it is much more than I am able to give. So Kudos.  I am making the assumption that you are during your early years.
So there could be many other problems and this test will help - The test - Play an unknown tune to a trusted / family member or friend of yours if they are able to recognize it and hum to the tune. You have passed the test and you are good enough.
If not, find the issue, whether it is less correct notes, sound or issue in timing, and fix that. If you cannot find the issue, post a sample a lot of people will be happy to help. If nothing works find a good teacher.
Next then, the thumb rule. Whatever be the outcome of the first, The THUMB RULE is to stop comparing to others. Are you enjoying it? If yes, that's good. Rest all will follow. It does not matter if the friend is playing well. There are always tons of other people even on youtube who would make anybody feel shy.

Guitar practice


Help in proving $\lim \limits_{n \to \infty} (a_n \pm b_n) = a \pm b$

Question: This was left as a non-assessed exercise, and I am unsure of how to attack it:
If $a_n$ and $b_n$ are convergent sequences converging to $a$ and $b $ respectively. Then prove that:
$$\lim \limits_{n \to \infty} (a_n \pm b_n) = a \pm b$$
I am thinking I will have to call forth content in regards to $\epsilon - N$. But I have no $n$ to work with, so I have no idea where to take this. Any tips or advice is appreciated!
Note: I have seen Prove for all sequences $\{a_n\}$ and $\{b_n\}$, if $\lim a_n = a$ and $\lim b_n = b$, then $\lim a_n + b_n = a+b$ entirely in first-order logic, but I can't see this as being an efficient way of proving this, nor can I follow it entirely.
Answer: $\lim \limits_{n \to \infty} a_n=a$ means that $|a_n-a|\to 0$
similarly $\lim \limits_{n \to \infty} b_n=b$ means that $|b_n-b|\to 0$. Therefore
$|a_n\pm b_n-(a\pm b)|\leq |a_n-a|+|b_n-b|\to 0$
Of course, if you don't know that $\lim \limits_{n \to \infty}a_n\to a$ and $\lim \limits_{n \to \infty}b_n\to b$ the claim makes no sense.
Comment: Great answer. That makes perfect sense!
Answer: What you need to do is to consider $a_n$ and $b_n$ separately. You can get as close as you like to $a$ and $b$ respectively, so you can get within $\frac {\epsilon}2$ with $N_a, N_b$ respectively.
Now you take an $N$ at least as big as these, and you have $\frac {\epsilon}2$ simultaneously for both. Use the "triangle inequality" (inherent in the metric) to prove that you are within $\epsilon$ for the sum and difference.
Answer: I presume you have $a_{n} \to a$, $b_{n} \to b$ in your hypothesis.
By definition of limit you have:
$\forall \delta > 0 \quad \exists N \in \mathbb{N} \quad s.t. |a_{n} - a| \leq \delta \quad \forall n \geq N$
and of course you have the analogous for $b_{n}$. For clearness, call the first $N=N_{a,\delta}$ and the second $N_{b,\delta}$. Then you would like to fix $\varepsilon >0$ and write the property of limit for the sum for this fixed $\varepsilon$. Then take $N = max\left( N_{a,\frac{\varepsilon}{2}},N_{a,\frac{\varepsilon}{2}}\right)$, then you have
$|a_{n}+b_{n}-a-b| \leq |a_{n}-a| + |b_{n}-b|\leq \varepsilon \quad \forall n \geq N$.
Analogously for $a_{n}-b_{n}$.


400 Error: Bad request AutoDesk Forge Api

Question: I was following a GitHub sample project for AutoDesk Forge at https://github.com/Autodesk-Forge/viewer-nodejs-tutorial  and they said to implement Model Derivative uploader before implementing the viewer.  I tried this GitHub project- https://github.com/jaimerosales/modelderivative-nodejs-tutorial
As per their instructions I need to mention client id , secret key, and bucket name. And the running it, I am getting a 400 error.
<code>**** Getting bucket details :
nikhil7j9ug4pgfj7lcukyoht8salipyb7skfc7j9ug4pgfj7lcukyoht8salipyb7skfc
**** Getting all buckets ****
**** Uploading to

bucket:nikhil7j9ug4pgfj7lcukyoht8salipyb7skfc7j9ug4pgfj7lcukyoht8salipyb7skfc
File:./models/House.dwfx
**** Get all buckets response:
BUCKETS::::  extract-autodesk-io-20177j9ug4pgfj7lcukyoht8salipyb7skfc
BUCKETS::::  model2017-11-24-06-00-57-d41d8cd98f00b204e9800998ecf8427e
BUCKETS::::  model2017-11-24-06-01-25-d41d8cd98f00b204e9800998ecf8427e
BUCKETS::::  model2017-11-24-06-01-57-d41d8cd98f00b204e9800998ecf8427e
BUCKETS::::  model2017-11-24-06-02-10-d41d8cd98f00b204e9800998ecf8427e
BUCKETS::::  nikhil7j9ug4pgfj7lcukyoht8salipyb7skfc
BUCKETS::::
nikhil7j9ug4pgfj7lcukyoht8salipyb7skfc7j9ug4pgfj7lcukyoht8salipyb7skfc
BUCKETS::::  testname7j9ug4pgfj7lcukyoht8salipyb7skfc
error: sdaflksofk { statusCode: 400, statusMessage: 'Bad Request' }
{ statusCode: 400, statusMessage: 'Bad Request' }
</code>
Answer: Could you try to debug into the app to see if you can find out more about why you are running into this problem?
You could also try my sample app to create your own bucket and upload a file into it: OSS Manager
Source code: OSS Manager Source Code.
When you are viewing a file with this sample the URN will be shown in the Browser's Console Window
Comment: Hi, Thanks for your reply. I ran your live app and getting an error. https://pasteboard.co/GVAkCyP.png   .
Comment: When I debugged your code, I am getting same unauthorized error using ,y client id and secret!! Actually What is happening :(  ( log out : https://pasteboard.co/GVAxtkl.png )
Comment: Have you enabled "Model Derivative API" for your app? Click on your app in developer.autodesk.com and check what's listed in the top under "APIs"
Comment: I have only 2 Apis BIM360 and Data management API. I cant find others. Even there is no other api when I tried to create one!. Please find in the screenshot at https://pasteboard.co/GVGsjvv.png
Comment: Thank You!  I was running a basic account. I switched it to Trial and got all APIs.


Sharing IP black lists

Question: Are there any tools similar to fail2ban that would allow sharing of IP black lists?  What  I'm looking for is something similar to a DNS based blackhole list or real-time blackhole list as is used to stop the spread of spam.  However I'm not so much interested in SPAM but in attackers doing things like port scans, password guessing, etc.  My servers get attacked hundreds of times a day and attackers could be shut down so much quicker if there was a list of known (suspected?) attackers.
Answer: Such lists exist. Or pretty close. Example: https://www.spamhaus.org/sbl/
But my advice would be not to rely on those lists.
First of all install a tool like fail2ban or CSF+LFD on your servers - if you haven't already. Then offending IP addresses will be quickly blocked at firewall level.
When you have a machine exposed on the Internet being scanned is a fact of life.
The problem is that some of the scanning activity is legitimate, some of it is malicious. Not everyone has the same criteria about what constitutes malicious activity.
For example search engines will routinely scan the Internet for ports 80/443 in order to discover new hosts and websites to index etc. That is perfectly legitimate. Some hosts probe for FTP/SSH/telnet/Mysql/SQL Servers etc or scan all the TCP/IP ports. Many sysadmins won't like this kind of probing. But it can still be legitimate for statistical purposes.
It's always interesting to know the prevalence of FTP servers, or Gopher or whatever. Or the respective market share of web server software vendors. In order to aggregate this data, you have to scan.
That is also how some poorly-protected servers hosting databases or sensitive files are found and reported through responsible disclosure.
So my criteria would be to show some leniency when it comes to scanning, but throttle the requests.
On the other hand the brute-force attacks must be thwarted without mercy. By default CSF+LFD will ban an IP address for one hour after 5 failed SSH attempts I believe. It can also automatically blocks port scanning attempts.
You can tune the limits and also create your own rules.
In short, I would rather block the IP addresses actually attacking my servers, than base the decision process on a second-hand list that may not be accurate or relevant. There is no list that will cover all possible attackers.
Answer: Yes, there are such tools but in fact you can use fail2ban. See https://www.blocklist.de/en/. They have a config for fail2ban that also includes use of badips.com
https://docs.danami.com/juggernaut/user-guide/ip-block-lists has links to other lists that may allow user contribution.
I wouldn't use them because they can be easily abused.
Comment: Do these lists include some sort of mechanism to rank the reported abuse?  I'm not really concerned with robots, port scans, etc.  What I'm really interested in blocking as soon as possible are attempted break-ins.  I run fail2ban on the systems I can, but it seems far more efficient to cut off known offenders before they even get to one of my ports.  I'm not running large public servers and I'm also running a bunch of home automation stuff that I'd like to protect.  Ideally I'd want my router to just blackhole packets from known or even suspected abusers.  A few false positives are OK.
Answer: AbuseIPDB offers an API to check an IP address under several categories of abuse. It can integrate with CSF in both directions (query and report) and with fail2ban for reporting abuse only.
I wouldn't certainly use the IP addresses in this database to deny access from an IP, but it could be a hint to set lower fail2ban limits.
However as all other answers, this database can be easily overflown with fake reports from spammers whenever it becomes a problem.
Comment: Seems like all of these reporting systems need a reputation mechanism to prevent spamming of the lists.
Comment: Such a system is difficult to implement. You can not verify that the listed IP really attacked the reporter and the fact that you were not attacked by the listed IP doesn't mean it is clean. Spammers can report the same IPs which legitimate users do and add some of their own.

Sharing IP black lists


How can I prove that the first three decimal digits of a real number between 0 and 1 can be equal to the reciprocal of 2 to the power of that number?

Question: This is the challenge problem at the end of Chapter 1 of Solow's How to Read and Do Proofs.
The problem:

Find a counter-example to the following statement:  "If x is a
  positive real number between 0 and 1, then the first three decimal
  digits of x are not equal to the first three decimal digits of $2^{-x}$.

My attempt:

The hypothesis A is that x is a real number between 0 and 1.
The conclusion B is that the first three decimal digits of x are equal to the first three decimal digits of $2^{-x}$.
A possible key question is, "How can I show that the first three decimal digits of a real number between 0 and 1 is identical to first three decimal digits of the reciprocal of 2 to the power of that number?

I've tried to reformulate this as a problem with graphs. I graphed the horizontally flipped version of f(x) = $2^{x}$, then looked at the graph between 0 and 1.
But I'm not sure how to continue; it's possible I need to review the necessary background knowledge.
What topic in mathematics should I study to solidify my background knowledge to learn more relevant information for answering this question? If this isn't necessary, could I receive a hint on how to proceed?
Comment: Hint: what can you say about $x-y$ if $x$ and $y$ have identical first three decimal digits? How can you strenghten this property to get the inverse implication?
Comment: If $x$ is sufficiently close to the unique root of the equation $x=2^{-x}$, the first three (or more, if we so wish) decimals of $x$ and $2^{-x}$ will coincide.
Comment: Look [here](http://www.emathhelp.net/notes/algebra-1/linear-equations/definition-of-the-equation-roots-of-the-equation/), or at the answer of Christian Blatter.
Comment: @ProfessorVector I'm unsure how to use this info, possibly due to my misunderstanding of the definition of a unique root. But $f(x)$ = $2^{-x}$ does not approach 0 as $x$ either approaches 0 or 1 ($(f(x)$ approaches 1 and 0.5 respectively). (A [Yahoo Answer here](https://answers.yahoo.com/question/index?qid=20091203170017AAyteGT) states that it's a value that makes a function equal to 0.)
Comment: @ProfessorVector Thank you, I understand now! So if $x = 2^{-x}$, then $0 = x -  2^{-x}$. Now, let $f(x) = 0$. If $f(x) = 0$, then $x$ must equal $f(x)$. As $f(x)$ is a continuous function, $f(0) = -1$, and $f(1) = 1/2$, then there must be a value of $x$ for which $f(x) = 0$ between 0 and 1 by the Intermediate Value Theorem - as stated by Chrstian Blatter.
Answer: I started with $x=0.123$ and got $2^{-x}=0.918$
Then I considered $x=0.918$ and got $2^{-x}=0.529$
And went on applying the same function so I got the sequence
$0.123,\;0.918,\;0.529,\;0.693,\;0.618,\;0.651,\;0.636,\;0.643,\;0.640,\;0.641,\;0.641,\;0.641,\;\ldots$
Hope this is useful
The fixed point $x=0.641$ is the counterexample
Comment: Forgive me if this is a stupid question, but how do we know this recursion is convergent?
Comment: Yes it was indeed a stupid question. I understand now. Thanks.
Comment: @VladeKR As derivative $f'(x)=-2^{-x} \log 2$ satisfies $|f'(x)|<1$ on $[0,1]$ then the function satisfies the Lipschitz condition and has a fixed point such that $f(x)=x$
Answer: We are told to produce a counterexample to a somewhat  arbitrary statement. The circumstances suggest that there is such a counterexample. How to find it? Why should some $x\in\ ]0,1[\ $ have a similar decimal representation as $2^{-x}$? The natural reason is that in fact $x=2^{-x}$. Therefore let's prove that there is such an $x$. Unless some trouble of the kind $0.6299997$ turns up it then should be possible to name a clear cut (i.e., having finite decimal expansion) $x$ that is a counterexample to the alleged claim.
The function $f(x):=x-2^{-x}$ has values $f(0)=-1$, $f(1)={1\over2}$, hence at least one zero $\xi\in\ ]0,1[\ $. It turns out that $\xi\doteq0.641$. Entering $x:=0.641$ into a pocket calculator gives $2^{-x}\doteq0.641268$, establishing $x$ as a counterexample to the alleged claim.
Answer: Hint: you can construct an easy counter example using the fact that as $x \rightarrow 0$, $2^{-x} \rightarrow 1$.
Comment: Yes, sorry about that! I ignored the minus sign. But actually you can show that there exist a solution for $x = 2^{-x}$ between 0 and 1, since $f(x) = x - 2^{-x}$ is less than 0 at $x = 0$ and greater than 0 at $x=1$, intermediate value theorem suggest that there exists $c \in (0,1)$ such that $f(c) = 0$.
Comment: No, because as $x \to 0^+$, the first three digits of $x$ become $000$ while those of $2^{-x}$ become $999$.
Comment: I appreciate the attempt, but @Adayah is right - I've verified the comment by plugging in values.


How to get a Blowfish encryption key

Question: I have a bunch of messages in encrypted and decrypted form using Blowfish with the same key. Is there any way to get key out of those messages?
References to any algorithm or tool will be highly appreciated.
I have only basic knowledge of cryptography so please correct me if question is too broad and not specific enough.
Comment: Probably not -- but if you can do so, you'll be famous (at least among cryptographers) as the person who broke Blowfish using a known-plaintext attack.
Answer: No, cryptography uses mathematical / binary functions to create (or rather, transform) the original data into encrypted information. If it stored the key within the message, it would be rendered useless.
Here is a reference for how the Blowfish Algorithm works: Link.
Here is a reference on Symmetric-key cryptography, of which Blowfish is a part of (as a Symmetric Block Cipher): Link
Comment: Thanks for the info about Blowfish!
Answer: You do not have many chances of getting the key. The only attacks that I see are:

leverage on flaws in the implementation of the encryption. Blowfish is a block cipher, as such it encrypts only blocks of fixed size. In order to encrypt variable length files, you need a mode of operation and a padding scheme. In your post you do not tell which if these have been used for your files, but some of the few successful attacks against cryptography exploit wrong choice or implementations of these.
if the key was derived from a password you can try to guess the password. In addition to the above, you also need to know the algorithm that was used to derive the password. You can also find in the Internet several dictionaries, which are basically long lists of possible passwords. With some work you can write a program that checks several password per second. My experience says that if you are lucky and the password is in a dictionary you can probably find it in a few days.

For the sake of completeness, trying all the possible keys requires more than the life of the universe.
Comment: "Some of the few successful attacks against cryptography" - there's no shortage of successful attacks in the crypto sphere. :)
Comment: Note that Blowfish does have a set of 'weak keys' that are to be avoided but only if you don't use the full 16 rounds of encryption so this is a theoretical attack
Comment: Yes, errors in the implementation of Blowfish are unlikely. My answer applies to wrong implementations of modes of operations, like using multiple times the same IV.
Answer: If those messages were stored in the file and the file is encrypted, you can write a script to do dictionary-based attack on those files. Of course, I am assuming that those texts are stored in the file and file is encrypted.
Sample:
$echo "Super secret" >> secret.txt; openssl enc -aes-128-cbc -in secret.txt -out secret.enc -k password

you can reverse the process, by bruting it

How to get a Blowfish encryption key


Regression question in R- how to multiply an indicator by a continuous variable ( linearly dependent/ not interaction)

Question: I am trying to conduct a logistic modeling analysis. In this analysis, the data is as the following:
Y: binary (0,1)
And for the independent variables, they are as the following:
X1: DrugA: categorical variable. Did patients take Drug A ( yes or no)
X2: DrugA_Conc: continuous variable. the concentrations of Drug A which also includes zero.
X3: DrugB: categorical variable. Did patients take Drug B (yes or no).
Please note that in the input dataset, patients belong to one of 3 groups:

Did NOT take either Drug A nor Drug B (placebo)
Took Drug A but NOT Drug B (Drug A alone)
Took Drug A and Drug B (combination)

Please see the code below that generates a representation of  the independent variables dataset:
DrugA<-data.frame(DrugA=factor(1))
DrugA_Conc<-data.frame(DrugA_Conc=rep(seq(from = 0, to = 100, length.out = 3)))
DrugB<-data.frame(DrugB=factor(c(0,1)))
mergd<-merge(DrugA,DrugA_Conc)
mergd2<-merge(mergd,DrugB)
DrugA_2<-data.frame(DrugA=factor(0))
DrugA_Conc_2<-data.frame(DrugA_Conc=rep(0,length.out = 3))
DrugB_2<-data.frame(DrugB=factor(0))
mergd3<-merge(DrugA_2,DrugA_Conc_2)
mergd4<-merge(mergd3,DrugB_2)
mergd5<-rbind(mergd2,mergd4)

My question is the following: How can I code a logistic model in R to predict the effect of DrugA_Conc on Y with and without the presence of DrugB, and have the output predicted for the 4 possible scenarios ( the 3 listed above) +

Did NOT take Drug A but took Drug B (Drug B alone).

I tried the following codes in R but it did not work:
m1 <- glm(Y~ DrugB+DrugA*DrugA_Conc, data=all, family="binomial")
m2 <- glm(Y~ DrugB+I(DrugA*DrugA_Conc), data=all, family="binomial")

Please note that I am not trying to evaluate the interaction between DrugA and DrugA_Conc but rather trying to create predictions for 4 possible drug combination scenarios ( Drug A alone,  Both drugs, none of the drugs,Drug B alone) based on the dataset that has the first 3 scenarios.
Also, can the code be applied if I take the log of DrugA_Conc ( I mean log(DrugA_Conc))
In SAS, I found that a trick (multiplying DrugA*DrugA_Conc) can be applied to code the model like the following, which allows for predicting the 4 scenarios (Figure 1):
model Y(event='1') = DrugB DrugA*DrugA_Conc/

Best regards,
Comment: Is there a specific reason to use the indicator for drug A if the drug A concentration variable includes 0? I think it would be easy enough to recreate your scenarios with 2 variables, drugA_concentration and drugB. Then you're interaction could be 'DrugA_Conc*DrugB'
Comment: Question is closely related to (and a possible duplicate of) [this question](https://stats.stackexchange.com/questions/372257/).
Comment: Thank you @Ben-ReinstateMonica!!! your link is on point. Based on that, I tried to following models:
m1 <- glm(Y~ DrugB+DrugA+ DrugA:DrugA_Conc, data=all, family="binomial")
However, I got a non-significant estimate for DrugA parameter.
Next I dropped the DrugA term and reran the below model:
m2 <- glm(Y~ DrugB+ DrugA:DrugA_Conc, data=all, family="binomial")
which resulted in lower AIC, BIC, and improved significance of DrugA:DrugA_Conc parameter.
Would m1 be the model to use even if the parameter is not sig or should I rather use m2?
THX
Comment: See my answer below.
Answer: Your question concerns the use of nested variables in a regression model, which is discussed in general in this related question.  In your case you have indicator variables DrugA and DrugB with continuous nested variables DrugA_Conc and DrugB_Conc respectively.  Moreover, in your particular case, your drug concentration variables fully determine the initial indicators, to the indicator variables are functions of the concentration variables.
In such cases, because of the functional relationship between the variables, inclusion of both DrugA and DrugA_Conc will mean that you have linearly dependent explanatory variables.  Ordinarily, when dealing with nested variables, one ensures that the nested variable only enters the model through an interaction term with the required condition for it to be a meaningful variable.  Thus, to use the concentration of Drug A in your model, you would do this through the interaction term DrugA:DrugA_Conc.  However, in your case the initial baseline variable DrugA is fully determined by the nested variable DrugA_Conc, and so the interaction becomes redundant, and is equivalent to the baseline model term DrugA_Conc.
In view of this, I would suggest that your model only include DrugA_Conc (or equivalently DrugA:DrugA_Conc) and not DrugA.  If you include the latter then it will show up as contributing nothing extra to the model, since it is a function of a model term that is already included.
Comment: Thank you.
So this means that the model will be m2 <- glm(Y~ DrugA:DrugA_Conc, data=all, family="binomial")


Basic question on tossing coins

Question: Generally the sample space of two events

Tossing three coins at a time
Tossing one coin three times

are considered same . But I really don't know how can these two events be the same even in mathematics ?
I know this is a very silly question but I genuinely want to know about this .
Thanks in advance :)
Comment: I'm not sure if this will help you but you just need to consider all possible cases for three coins at a time if one side is named head and other one tails: {(h,h,h),(h,h,t),(h,t,t),(t,t,t),(t,h,h),(t,t,h),(h,t,h),(t,h,t)} and since tossing three coins separately dose not make any difference in the chance of head or tails would be the same so they have same sample space.
Comment: Can you explain why you think they should be different? We're assuming the coins are ideal: tossing a coin doesn't damage it or alter it in any way.
Answer: Assume the probability of heads or tails for the result of tossing any coin is 0.5. The events are independent.  There is no difference to the probability of obtaining 0, 1, 2 or 3 heads if three coins are tossed simultaneously or one coin three times.
Answer: Assuming independence,
The sample space of the experiment 1. is the following
$$\{(TTT),(CTT),(TCT),(TTC),(TCC),(CTC),(CCT),(CCC)\}$$
Here the outcomes $(ABC)$ indicate the outcomes of the three coins.
The sample space of the experiment 2. is the following
$$\{(TTT),(CTT),(TCT),(TTC),(TCC),(CTC),(CCT),(CCC)\}$$
Here the outcomes $(ABC)$ indicate the outcomes of the three tosses of the single coin.
Mathematically, if you want to calculate some probabilities in the two experiments, the results is evidently the same

Intuitive explanation.
Consider the Experiment 2 but tossing the coin repeatedly at the light speed...it is self evident that this overlaps with Experiment 1

Basic question on tossing coins


Can't connect to local MySQL server through socket /var/run/mysqld/mysqld.sock

Question: I have run into this error with a MySQL database installed on lucid lynx 10.04 that was working fine.  The last activity carried out before this problem manifested was exporting a database back up using phpmyadmin.  Now any attempt to connect results in this error whether from the command line, phpmyadmin tool or php script.
Most of the answers to similar posted questions involve finding where the mysqld.sock file is created and modifying the my.cnf file to point to its locaion.  However I cannot see where the mysqld.sock file is created.
I have verified that mysql is running as a service and following one forum advice, typing
ps -fea | grep mysqld results in the following:
rich      10796 8900 0 08:13 pts/1 00:00:00 grep --color=auto mysqld

mysqld is shown in red.  The rest of the advice given is to use the above output line to locate the file and modify the my.cnf file accordingly. However I do not know how to interpret the above information to this effect - being a noobie to ubuntu I am in the unhappy position of usually not understanding what the purpose of command lines (often suggested without an explanation) is and what their results should be so am reduced to randomly typing them in from various similar forum questions in the hope that eventually one will have the desired effect without breaking the whole thing in the process.
Could anyone advise a step by step trouble-shooting guide to this problem?  Where should the file be ceated in Ubuntu? The path in the /etc/mysql/my.cnf file is /var/run/mysqld/myslqd.sock.  I presume this must have been where the file was being created previously when it was working unless the my.cnf file has been changed by some sort of automatic update, which I wouldn't have thought so given a password is usually required in ubuntu before such changes are made.
Thanks
Richard
Comment: Judging from the output of `ps -fea | grep mysqld`, you _do not_ have MySQL running.
Answer: Try out the below steps .,
backup the my.cnf.,
change the socket location to different location .,.
restart the mysql with   #/etc/init.d/mysql restart
try to connect with new socket .
Comment: as far as I know, linux doesn't allow you to start mysql as that. It wants the `service mysqld [re]start` command.
Comment: Thanks to all who have responded so far.

Lanzz:  If I run service --status-list, mysql is shown with the symbol [?].  This corresponds to 'unknown'. I suppose I should be expecting [+].  If I run sudo initctl list, mysql is shown in the list of results with 'start/running'.  However there is no process number. Again I am not sure what I should be expecting.

tsabz: Sorry, not sure what you mean.

sandeep: Tried this but didn't work. The command for restart seemed to go off alright after a password prompt but there was no feedback.
T-Shirt Dude: this command line fails as unrecognised service.
Comment: Just to add have tried a command from ubuntu to check mysql status
sudo netstat -tap | grep mysql
This returns nothing, just goes to the next command prompt.
When using the sudo /etc/init.d/mysql restart command a message contained in the my.cnf file is shown which starts "Rather than invoking init scripts..." and ends "start: Job is already running: mysql"

The command 'service mysql restart' results in command unrecognised and the command 'sudo service mysql restart' hangs. Still none the wiser.
Comment: ....after a complete reboot due to the command 'sudo service mysql restart' hanging, normal service has resumed and everything is running again.  Had tried rebooting several times before to no effect so it seems the last command I tried shifted something.

Guess its just one of those things - two evenings of my life I will never get back and probably will never know why. Computers suck.


How to add paragragh information under hovered menu item?

Question:
Hi all,
I want to add descriptions under my menu items like above (Eg. Home (ADD my description) Products (etc)?
Do I have to create children under these menu items and enable these????
Any advice would be appreciated.
Drupal 7
Answer: I think you're going to need to do a little (code) preprocessing of the menu in your theme layer.
To achieve this, I would put my description text in the "Description" part of my links (which is actually the title attribute) and then add a theme_menu_link() function to template.php (https://api.drupal.org/api/drupal/includes%21menu.inc/function/theme_menu_link/7).
Most people overload this function to add attributes to the menu links (like classes or ID's) but you could also adjust it to pull out the text you have placed in the title attribute, then mark it up however you want (span, div, etc..) and then return it in the Menu Link Title.
I see that you have also tagged your question with Superfish so you may need to review the theme superfish function as well (http://api.drupalhelp.net/api/superfish/superfish.module/function/theme_superfish/7).


reliably and automatically determine connection path of physical position of HDD from /dev/sdX device file

Question: This is kind of a FAQ, but all the answers I found so far are not suited for full automation, which is what I need. So here it goes again.
In Linux:
Is there a reliable way to resolve the udev device name of a HDD (e.g. "/dev/sdg") into it's data path to identify the physical real-world cable the device is connected through (E.G. "Controller in PCIe-Slot2, SAS-Channel 0, Replicator Port 3"?
I run a server with a dozen SATA-disks in hot-swap backplanes. The disks are assembled into a software Raid6 using dmraid (Linux). For reasons beyond the scope of this question I want and need to run software raid, not hardware raid through a dedicated controller.
One of the shortcomings of software raid is that the failure LED on the drive bay doesn't come on when a drive fails in an array, because the enclosure has no way to poll mdadm for drive status. You have to find the position of the faulty drive manually.
I know you could just issue a dd if=/dev/sdg of=/dev/null and see which activity LED comes on, but I'm aiming for the pretty solution here.
To fix that I hacked together a little PCB that will talk to the backplane via i2c to switch on/off the failure LEDs of the bays and I have a little script that talks to this board via RS232.
mdadm can run a command when a failure event occurs, so I can tell mdadm to run my script and turn on the LED when a drive drops out of the array. The only problem is:
mdadm tells me "drive /dev/sdg1 has failed". But what I need is "Drive on Controller 1, Channel 2, Port 3 has failed", so I can identify which LED to switch on.
Does anyone know a reliable way to resolve a device name like /dev/sdg back to the path?
I know hdparm -I /dev/sdX will give me the serial number and vendor of the drive so I can manually identify the disk by looking at the label, but the point is to do this automatically. Reliably identifying the controller/port involved will suffice, since the wiring usually won't change when using backplanes and I know what controller port services what drive bay.
My first idea was to do ls -lah /dev/disk/by-path | grep /dev/sdX for the appropriate target name, however this proved unreliable, as not even half of the disks currently installed appear in that directory.
Just saying "your first controller will have sda-sdh, your second controller will have sdi-p" is also unreliable, since there's a race condition at boot and sometimes the one controller is initialized first, sometimes the other. Whichever is initialized first gets /dev/sda... Also things get complicated after a hot swap or if not all bays are populated.
lshw -short -c disk seems to generate an output similar to what I'm looking for, but I'm having trouble linking the path numbers shown there to the physical cables. Are these assignments stable? What is the exact pattern? (The numbering in the scsi paths is continuous even though not all my drive bays are populated, that leads me to think these scsi path nodes are dynamically assigned, and don't directly represent physical ports on the controller)
Does anyone know a linux command which takes a /dev/sdX as argument and reliably outputs the associated PCIe-slot and controller port?
Comment: I just found this bug report: [Debian Bug report logs - #687378](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=687378). It appears udev is supposed to create meaningful links for every drive that can be used for the purpose described above, but currently doesn't. So once this issue is resolved, grabbing the links from `/dev/disk/by-path` should be the way to go.
Comment: Poking about in sysfs might get you what you want.  Follow /sys/block/sda/device and it will point you to a sysfs node that should show the PCI ID of the card, a hostX identifying the sata port, and a scsi target number, which is always zero unless you have an expander, in which case, it should indicate the port number on the expander.
Comment: could you elaborate on that? All I found there so far was the scsi id (also shown by `lshw`), which unfortunately is not persistent. SAS-id is also present, but not related to the path...
Answer: As a previous comment mentioned, check out /sys/block/sdX.  If you cd to one of those directories, and then do ls -l, you'll see a symlink for device, which should point you to the path of the device.  For example, on my system I see device in /sys/block/sdz linked as follows:
device -> ../../devices/pci0000:00/0000:00:09.0/0000:08:00.0/host3/rport-3:0-5/target3:0:5/3:0:5:1/
This gives me the exact path to the device that backs the block device, from PCI path to host controller to LUN.  This may look a little different depending on the type of device you have - for example, mine is a fiber channel controller, hence the rport portion of it.