Some trigo identities

Question: I aacidently found the following:
$$\sin\frac{2\pi}{7}+\sin\frac{4\pi}{7}-\sin\frac{6\pi}{7}=\frac{\sqrt{7}}{2}$$
$$\sin\frac{2\pi}{11}-\sin\frac{4\pi}{11}+\sin\frac{6\pi}{11}+\sin\frac{8\pi}{11}+\sin\frac{10\pi}{11}=\frac{\sqrt{11}}{2}$$
$$\sin\frac{2\pi}{19}-\sin\frac{4\pi}{19}-\sin\frac{6\pi}{19}+\sin\frac{8\pi}{19}+\sin\frac{10\pi}{19}+\sin\frac{12\pi}{19}+\sin\frac{14\pi}{19}-\sin\frac{16\pi}{19}+\sin\frac{18\pi}{19}=\frac{\sqrt{19}}{2}$$
I wonder if there is a general rule such that, for prime number $p \cong3 \pmod 4$ and some suitable $f(k)=1$ or $f(k)=2$, the following identity holds?
$$\sum_{k=1}^{\frac{p-1}{2}}(-1)^{f(k)}\sin\frac{2k\pi}{p}=\frac{\sqrt{p}}{2}$$
Any idea?
Comment: I would conjecture $\sqrt{p} = 2 \sum_{k=1}^{(p-1)/2} \left(\frac{k}{p}\right) \sin \frac{2k\pi}{p}$.
Comment: @marlu's conjecture is spot on. You want to look up "(quadratic) Gauss sums) in a number theory book (Ireland-Rosen, for example).
Comment: And of course, $\sin \frac {2\pi}{3} = \frac {\sqrt{3}}{2}$.
Comment: Is there an analogue formula for cube roots of primes ? Maybe with cubic reciprocity ?
Answer: Very good observation. The proof is somewhat long, here I present a sketch. For full details, see Chapter 6 in Ireland-Rosen "A classical introduction to modern number theory".
Let $p$ be an odd prime, and set $\zeta=e^{2\pi i/p}$, so the powers of $\zeta$ are the $p$-th roots of unity. The relevance here lies in the fact that $\sin(2 \pi k/p)$ is the imaginary part of $\zeta^k$.
The Gauss sum $g_a$ is defined by
 $$ g_a =\sum_{k=1}^{p-1}\left(\frac{k}{p}\right)\zeta^{ak}, $$
where $\displaystyle \left(\frac{k}p\right)$ is the Legendre symbol: $0$ if $p\mid k$, $1$ if $k$ is a square modulo $p$, and $-1$ otherwise.
You can check that $g_a=0$ if $p\mid a$, and $g_a=\displaystyle\left(\frac{a}p\right)g_1$ otherwise. Anyway, you are interested in $g_1$ (or rather, its imaginary part).
One starts by verifying that $g_1^2=(-1)^{(p-1)/2}p$. This is not too bad. The standard way of checking this is to add $\sum_{k=1}^{p-1}g_kg_{-k}$ in two different ways.
If $p\equiv 3\pmod 4$, this means that $g_1=\pm i\sqrt p$. Ignoring the sign, the result follows, since $-1$ is not a square modulo $p$, so
 $$ \left(\frac kp\right)\sin\left(\frac{2\pi k}p\right)=\left(\frac {p-k}p\right)\sin\left(\frac{2\pi (p-k)}p\right) $$
for all $k$, and the sum of the first $(p-1)/2$ terms in $g_1$ equals the sum of the other half.
The difficulty lies in verifying the sign is $+1$ rather than $-1$. This actually gave Gauss a lot of work (he found $g_1^2$ in 1801, and $g_1$ in 1805). I do not know of any quick argument, but the proof in section 6.4 of the Ireland-Rosen book is rather elegant. Another book that presents a nice proof is Nathanson "Elementary methods in number theory". For what is worth, I think that the notation in Ireland-Rosen is somewhat easier to follow if you have no prior experience with Gauss sums. (If you have experience with contour integration, yet another proof is described in exercises in Berenstein-Gay "Complex variables: An introduction".)
Using the value of $g_1^2$, Gauss found a very clever (Fourier theoretic, we could say) proof of quadratic reciprocity. For completeness, let me add that for $p\equiv 1\pmod4$, we have $g_1=\sqrt p$.
Noam Elkies has posted on MO a very clever argument (that goes back to Schur) for evaluating $g_1$ by use of linear algebra.

Some trigo identities


How to create formulas to compute new variables in R considering names in a data frame and their column value

Question: Hi everybody I am trying to solve a little problem in R to compute new variables in R. The dput version of my data frame is:
structure(list(sexo = c(-22.84754, -30.95001, -37.36658, -45.64382,
-54.9466, -0.1732915), cli_edad = c(5.972972, 11.67697, 16.46362,
26.57938, 47.19307, 0.1037254), edad2 = c(-0.0637181, -0.1199798,
-0.1600652, -0.2424397, -0.4273092, -0.001068), veces_mora_ago12 = c(-100.6952,
-166.6598, -391.6087, -710.2349, -1098.773, -0.3525356), veces_mora_sep12 = c(20.06456,
162.5816, 388.1126, 738.8196, 1181.483, 0.2907068), veces_mora_oct12 = c(79.44273,
-15.99917, -13.33856, 9.459844, 103.7592, -0.0863719)), .Names = c("sexo",
"cli_edad", "edad2", "veces_mora_ago12", "veces_mora_sep12",
"veces_mora_oct12"), class = "data.frame", row.names = c(NA,
6L))

The rows of that data frame are the coefficients of different models. When I load a data frame in R I need to compute a new variable for each model. For example the loaded data frame DF will have the same names that z but in this case I have to compute 6 additional variables for DF this variables are defined for different formulas, for example this considering names of z and the first row of z:
DF$I1=-22.8475400*DF$sexo+5.9729720*DF$cli_edad-0.0637181*DF$edad2-100.6952000*DF$veces_mora_ago12+20.0645600*DF$veces_mora_sep12+79.4427300*DF$veces_mora_oct12

Like last formula I have to write 5 additional formulas for the combination between names and second row of z until names and sixth row of z.
I don't know if it is possible to create in R this formulas and when I load a data frame can apply these to compute the new variables. Thanks for your help.
Comment: It sounds like you do want matrix multiplication -- is `as.matrix(DF) %*% t(as.matrix(z))` not what you're looking for?  Can you give a sample `z` to work with?
Comment: The output of I1 you gave is `[1]  17410.94776  19549.83787  47112.85467  88294.47367 144127.32240 39.04883`.  What were you expecting as output?
Comment: Dear @JLLagrange it would be DF$I1 because I wat to apply that formula to a different data frame of `z` that has the same names that `z`
Comment: Can you say again what `z` is, and how it relates to `DF`?  If it is another data frame with the same number of columns as `DF`, you might think of doing this as vector algebra.
Comment: In your case, this would be `DF %*% as.matrix(c(-22.8, 5.97, -0.06, -100.69, 20.1, 79.4))`.  Then you could loop over data frames and multiply by that vector.  But maybe I don't understand what `z` is
Comment: Dear @JLLagrange `z` has in each row the different information of coefficients to compute a new value for a new data frame. In this case `DF` is the new data frame. The object is to get in a new column the sum product of each row of `z` with each row of a new data frame.
Answer: Your example:
> -22.8475400*DF$sexo+5.9729720*DF$cli_edad-0.0637181*DF$edad2-100.6952000*DF$veces_mora_ago12+20.0645600*DF$veces_mora_sep12+79.4427300*DF$veces_mora_oct12
[1]  17410.94776  19549.83787  47112.85467  88294.47367 144127.32240     39.04883

As @JJLagrange says I think you want this???
DM=as.matrix(DF)
DM%*% t(DM)
             1           2           3            4            5           6
1  17410.94776  19549.8379  47112.8547   88294.4737  144127.3224  39.0488287
2  19549.83787  55558.5082 129927.5614  240057.8018  375800.3450 113.9736682
3  47112.85467 129927.5614 305834.0191  566896.3661  890283.7105 260.2182340
4  88294.47367 240057.8018 566896.3661 1053167.3837 1658033.7139 475.0128054
5 144127.32240 375800.3450 890283.7105 1658033.7139 2619216.6537 736.2772169
6     39.04883    113.9737    260.2182     475.0128     736.2772   0.2570419


Pegar Elemento dinâmico no JSON

Question: tenho o seguinte json
{
    "error": false,
    "installments": {
        "visa": [{
            "quantity": 1,
            "installmentAmount": 500.00,
            "totalAmount": 500.00,
            "interestFree": true
        }, {
            "quantity": 2,
            "installmentAmount": 261.28,
            "totalAmount": 522.55,
            "interestFree": false
        }]
    }
}

Este objeto visa, pode ser alterado dependendo da chamado porem apenas este objeto, pode vir como bradesco, itau, etc.
Existe uma forma de acessar as informações deste objeto sem ser pelo nome explícito dele (installments.visa)?
De alguma forma jogar em uma variavel assim:
var cartao = installments.(truquemagico);

E acessar os dados assim:
console.log(cartao[0].totalAmount);

Answer: Há uma forma sim. Primeiro você pode descobrir qual é key do objeto em obj.installments usando for..in. Dessa forma, independente do que estiver ali (visa, master, etc) será salvo em uma variável. Então é possível acessar essa chave pelo colchetes [], ficando obj.installments[prop] - onde prop é o chave que foi descoberta.

var obj = {
    "error": false,
    "installments": {
        "visa": [{
            "quantity": 1,
            "installmentAmount": 500.00,
            "totalAmount": 500.00,
            "interestFree": true
        }, {
            "quantity": 2,
            "installmentAmount": 261.28,
            "totalAmount": 522.55,
            "interestFree": false
        }]
    }
}

// descobrir a key
var prop;
for (property in obj.installments){
  prop =  property;
}

console.log(obj.installments[prop][0].totalAmount);

Pegar Elemento dinâmico no JSON


Replace em dash with hyphen while reading excel files in Python using Pandas

Question: i am reading an excel file using panda data frames. Column names may contain hyphen or em dash. i would like to replace all em dash with hyphen so that i can read column values with hyphen. row ['abc-def'].
Answer: Hyphen or em dash, you can still read the columns (a string is a string!) But if you insist, here's how to fix the dashes:
<code>df.columns = [c.replace("–", "-") for c in df]
</code>


How to customize log system of python rq?

Question: Currently the logs of rq is something like this:-
15:15:03
15:15:03 *** Listening on ingest...
15:17:41 ingest: tasks.ingest_job(aws_key='SDLKASFJHJKAHAJ', aws_secret='LDFKASJKDFJKSAHKJHkjsadhfkjaKJDSAHK')

So, It logs the arguments of the job too, which is a security issue, as the argument contains aws secret and access key. Is there any way to fix this in python rq? such that it won't log the arguments, or any other way
Answer: You can setup your own worker script (described in the documentation) instead of running rq worker. This allows you to set the log_job_description flag of the Worker object.
For example:
#!/usr/bin/env python
import sys
from rq import Connection, Worker

# Preload libraries
import library_that_you_want_preloaded

# Provide queue names to listen to as arguments to this script,
# similar to rq worker
with Connection():
    qs = sys.argv[1:] or ['default']

    w = Worker(qs, log_job_description=False)
    w.work()

This will log each job as it's created, but won't display the function parameters in the log.
Answer: You can silence the logging:
rq worker ingest --quiet

I don't think there is a straightforward way for the user to modify the format of the logging.
Comment: I have the same problem, and neither this approach nor `--disable-job-desc-logging` helps, were you able to find a solution? thanks!
Answer: Just run worker with --log-format
https://python-rq.org/docs/workers/#worker-arguments


How to show that an m.d.s is not independent?

Question: I have to prove that this Martingale Difference: $x_t = u_t u_{t-1}$ where $u_t \sim^{iid} (0, \sigma^2)$ is not serially independent, but am failing to do such thing.
I also have to prove that it's conditionally homoscedastic.
What I have tried so far is to show that $E[x_t^2|x_{t-1}^2] \neq E[x_t^2]$, and therefore it would not be serially independent because of the second moment, but in my view that would be proving that the process is conditionally heteroscedastic, instead of homoscedastic.
What am I doing wrong? How would you proceed?
Thanks
Comment: Unfortunately, it is possible for the $x_t$ to be independent when the $u_t$ are not Normally distributed: let each $u_t$ have a Rademacher distribution, for instance. That may be why you are finding it difficult to prove this result: it's not generally true!  BTW, in my copy of Hamilton there is an Example 7.8 that looks just like your question, but it explicitly states the $u_t$ are Normal.
Comment: Does your notation mean only that the $u_t$ have zero means and variances of $\sigma^2,$ or is it perhaps intended to mean also that the $u_t$ have jointly Normal distributions?
Comment: @whuber it's not intended to mean a Normal distribution, this example came from Hamilton's Time Series Analysis where the exact distribution is not specified. In my view, to prove the lack of independency I could assume a Normal distribution as a counter-example, but I believe the conditional homoscedastic part should be shown more generally.
Comment: @whuber Thanks! But even if I assume they are Normally distributed, I can't seem prove serial dependency: with my strategy mentioned above, I have reached the result $E[x_{t}^2] = \sigma^4 \neq E[x_{t}^2 | x_{t-1}^2] = u_{t-1}^2 . \sigma^2$, and, although this would prove dependency, I think it would also contradict the conditional homoscedastic part...
Comment: Hints: since now you are assuming Normality, you know higher moments are finite.  This enables you to compute, say, $\operatorname{Cov}(x_t^2, x_{t-1}^2).$ If this is nonzero then $x_t^2$ and $x_{t-1}^2$ cannot be independent, whence $x_t$ and $x_{t-1}$ cannot be independent.  As far as conditional homoscedasticity goes, you may be overthinking that, because the demonstration requires no calculation at all.
Answer: Although comments to the question indicate the original problem assumes the $u_t$ have Normal distributions, it is of interest to determine the extent to which this distributional assumption is needed to prove the $x_t$ are not serially dependent.
Let's begin, then, with a clear statement of the problem.  It focuses on three variables $(u_{t-1}, u_t, u_{t+1}).$  Suppose they are identically and independently distributed (iid); and that this distribution has zero mean and a finite variance of $\sigma^2.$  (We drop any Normality assumption.)  Out of these variables we construct two others, $x_{t+1}=u_{t+1}u_t$ and $x_t = u_tu_{t-1},$ which are products of successive $u$'s.  Since $u_t$ is involved in the construction of both of the $x$'s, intuitively those $x$'s should not be independent.  But how to prove this?
This is trickier than it might appear, because in simulations with Normally distributed $u$'s, the scatterplot of $(x_t, x_{t+1})$ shows a complete lack of correlation:

Indeed, if we try to demonstrate lack of independence by showing there is nonzero covariance, we will fail (because the $u$'s are independent and have zero mean):
$$\begin{aligned}
\operatorname{Cov}(x_t, x_{t+1}) &= E[x_tx_{t+1}]-E[x_t]E[x_{t+1}]\\
&= E[u_{t-1}u_t\, u_tu_{t+1}] - E[u_{t-1}u_t]E[u_tu_{t+1}]\\
&=E[u_{t-1}]E[u_t^2]E[u_{t+1}] - E[u_{t-1}]E[u_t]\,E[u_t]E[u_{t+1}]\\
&= 0-0 = 0.
\end{aligned}$$
The simplest way I have found to demonstrate lack of independence is to consider the magnitudes of the $x$'s.  The logic will be this: if we can show that some function $f$ (in this case, the absolute value) when applied to both $x$'s, produces variables with nonzero covariance, then the $x$'s could not originally have been independent.  (A simple proof is at https://stats.stackexchange.com/questions/94872.)
So, let's carry out the preceding calculations using the absolute values.  We are permitted to do this because the finite variance of the $u$'s implies the expectations of their absolute values are finite, too (see Prove that $E(X^n)^{1/n}$ is non-decreasing for non-negative random variables).  Let this common expectation be $E[|u_t|]=\tau.$  Then
$$\begin{aligned}
\operatorname{Cov}(|x_t|, |x_{t+1}|) &= E[|x_t||x_{t+1}|]-E[|x_t|]E[|x_{t+1}|]\\
&=E\left[|u_{t-1}|\right]E\left[|u_t|^2\right]E\left[|u_{t+1}|\right] - E\left[|u_{t-1}|\right]E\left[|u_t|\right]\,E\left[|u_t|\right]E\left[|u_{t+1}|\right]\\
&= \tau \sigma^2 \tau - \tau^4\\
&= \tau^2(\sigma^2 - \tau^2).
\end{aligned}$$
Unless $\sigma^2=\tau^2$ or $\tau=0,$ this is nonzero and we are done.
Because squaring is a convex function, Jensen's Inequality implies $\tau^2\le\sigma^2$ with equality if and only if $u_t$ is almost surely constant.
The case $\tau=0$ occurs when $|u_t|$ is almost surely constant, whence $u_t$ itself is either $0$ or takes on the values $\pm\sigma$ with equal probability (a scaled Rademacher variable).  Both situations can be characterized as scaled Rademacher variables (with scaling factor $0$ in the first case).
That covers all the possibilities.  The result is,

When  $(u_{t-1}, u_t, u_{t+1})$ are iid with zero mean, $x_t = u_{t-1}u_t$ and $x_{t+1} = u_t u_{t+1}$ are independent if and only if the $u$'s each have a scaled Rademacher distribution.

(The "if" part can be shown with a direct calculation: tabulate the eight possible values of $(u_{t-1}, u_t, u_{t+1}),$ compute the corresponding values of $(x_t, x_{t+1}),$ and verify independence.)
To help the intuition, here is a scatterplot of the absolute values in the preceding figure (shown on fourth-root scales to remove the skewness from the marginal distributions; if you like, the function involved is $f(x) =\sqrt[4]{|x|}$).  The red line is the least-squares fit to these fourth roots.  The correlation is obviously nonzero.


plinq on large lists taking enormous time

Question: I have two in memory lists plays and consumers one having 15 mil objects and the other around 3 mil.
the following are a few of queries i'm firing..
consumersn=consumers.AsParallel()
                    .Where(w => plays.Any(x => x.consumerid == w.consumerid))
                    .ToList();

List<string> consumerids = plays.AsParallel()
                                .Where(w => w.playyear == group_period.year
                                         && w.playmonth == group_period.month
                                         && w.sixteentile == group_period.group)
                                .Select(c => c.consumerid)
                                .ToList();

int groupcount = plays.AsParallel()
                      .Where(w => w.playyear == period.playyear
                               && w.playmonth == period.playmonth
                               && w.sixteentile == group
                               && consumerids.Any(x => x == w.consumerid))
                      .Count();

I'm using 16 core machine with 32 GB RAM, inspite of this.. the first query took around 20 hours to run..
Am I doing something wrong..
All help is sincerely appreciated.
Thanks
Comment: Profiler is your friend here. But it looks like you're doing 15M * 3M operations in your very first query here.
Answer: The first LINQ query is very inefficient, parallelization can only help you so much.
Explanation: When you write consumers.Where(w => plays.Any(x => x.consumerid == w.consumerid)), it means that, for every object in consumer, you will potentially iterate over the whole plays list to find the affected consumers. So that is a maximum of 3 million consumers times 15 million plays = 45 trillion operations. Even across 16 cores, that is about 2.8 trillion operations per core.
So, the first step here would be to group all plays by their consumerIds, and to cache the result in an appropriate data structure:
var playsByConsumerIds = plays.ToLookup(x => x.consumerid, StringComparer.Ordinal);

Then, your first request becomes:
consumersn = consumers.Where(w => playsByConsumerIds.Contains(w.consumerid)).ToList();

This query should be much faster, even without any parallelization.
I cannot fix the following queries because I don't see exactly what you are doing exactly with group_period, but I would suggest using GroupBy or ToLookup to create all groups in a single pass.
Comment: Thanks.. I will read on ToLookup. As far as the other two queries are concerned.. they are happening inside a foreach loop and there is no groupby clause used there.. The only reason to get consumerids in second query is to be able to use it in third query. So will it be right to change .Select(c => c.consumerid)in second query to .ToLookup(x => x.consumerid, StringComparer.Ordinal); and consumerids.Any(x => x == w.consumerid) to consumerids.Contains(w.consumerid)
Comment: What I am suggesting is that you can probably use a `GroupBy` to avoid a loop for the second and third query.
Comment: Sorry, Unable to understand what you mean by using 'GroupBy' to avoid a loop.. could you please provide an example..The where clauses in the second and third query use values coming from different sources-group_period and (period and group) respectively..
Answer: The first query took 20 hours to run because plays.Any(x => x.consumerid == w.consumerid) needs to walk through the entire list of 15,000,000 plays each time the consumerid is not there.
You can speed this up by constructing a hash set of all consumer IDs in plays, like this:
var consumerIdsInPlays = new HashSet<string>(plays.Select(p => p.consumerid));

Now your first query can be rewritten for an O(1) lookup:
consumersn=consumers
    .AsParallel()
    .Where(w => consumerIdsInPlays.Contains(w.consumerid))
    .ToList();

Comment: @Arnab Second query does not search lists, so it wouldn't be faster. Third query can be made faster if you turn `consumerids` to a hash set, and use `Contains` in place of `Any`. You can also replace `Where(cond).Count()` with `Count(cond)`.
Comment: can I use Hashset for the second query and will it be faster?? Also should I change consumerids.Any(x => x == w.consumerid) to consumerids.Contains( w.consumerid) in the third query. will that be faster?? Tx


Suitable performance metric for an unbalanced multi-class classification problem?

Question: I have an unbalanced multi-class classification problem with the following class distributions:
Class 0: 17.1%
Class 1: 63.2%
Class 2: 19.7%

I am using scikit-learn's Support Vector Classifier with 'balanced' class weights to classify samples into one of the three classes. However, I don't know what would be the most suitable performance metric to evaluate the result? So far I have been using F1-micro score, but not sure if this is the best option for multi-class imbalance problems?
Answer: There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, DP, etc.
Disclaimer:
If you use python, PyCM module can help you to find and calculate these metrics.
Here is a simple code to get the recommended parameters from this module:
>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

>>> score = cm.Kappa


problem importing csv file to pycharm using pandas

Question: I am trying to import file to PyCharm using pandas here is my code:
 import pandas as pd
 data=pd.read_csv(r'C:\Users\agns1\Downloads\data_work_final.csv')

sadly I'm getting this error:
 File "pandas\_libs\parsers.pyx", line 542, in pandas._libs.parsers.TextReader.__cinit__
 File "pandas\_libs\parsers.pyx", line 642, in pandas._libs.parsers.TextReader._get_header
 File "pandas\_libs\parsers.pyx", line 843, in pandas._libs.parsers.TextReader._tokenize_rows
 File "pandas\_libs\parsers.pyx", line 1917, in pandas._libs.parsers.raise_parser_error
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 4: invalid continuation
 byte

any thoughts on how can I fix this ?
Comment: Does this answer your question? [python: UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 0: invalid start byte](https://stackoverflow.com/questions/23772144/python-unicodedecodeerror-utf8-codec-cant-decode-byte-0xc0-in-position-0-i)
Answer: You need to check the file encoding:
with open(r'C:\Users\agns1\Downloads\data_work_final.csv', 'rb') as rawdata:
    result = chardet.detect(rawdata.read(10000))

print(result)

You'll get something like:
{'encoding': <'the actual encoding'>, 'confidence': xxx, 'language': xxxx}

Then do:
data=pd.read_csv(r'C:\Users\agns1\Downloads\data_work_final.csv', encoding='<'the actual encoding'>')


How to notify from B activity to A that list data has been changed?

Question: I have a Activity ItemList.java. When user click on item, it starts new activity, let's say ItemDetailActivity.java.
What I want to do is, when the user presses a button on ItemDetailActivity.java, it should tell ItemList.java to refresh list item. I know that myListAdapter.notifyDataSetChanged() will refresh the list. But how do ItemList.java will know to when to change data.
I don't want to use a static method or variables. I know how to use handlers, but don't know how to pass handler to other activities.
What I am thinking is to create my own Event / Listener and associate with the list, so when event fires, then change list. Would a Broadcast receiver help me?
Comment: Pass the reference of your ItemList to your ItemDetailActivity and call some refresh function?
Comment: Use a [`ContentProvider`](http://developer.android.com/guide/topics/providers/content-providers.html). It's some work, but it pays off.
Answer: You could try startActivityForResult(intent_to_activity) and in the same itemlist class, override the method :onActivityResult(). In this method do the notify data set change and it should reflect.
Comment: this is typically what you should do in most basic cases
Answer: The best way would be to use the observer pattern (refer to http://en.wikipedia.org/wiki/Observer_pattern), but you could make a public static method in ItemList.java containing a call to myListAdapter.notifyDataSetChanged().
Answer: You really shouldn't assume that a reference to your old activity will be kept alive by Android. If this is all based on a database, then you really don't need to do anything for your ItemList to update. If its not, you can simply call myListAdapter.notifyDataSetChanged() in the onResume() of your ItemList activity.
Comment: That's not fear sir :( I don't want to do this every time.
Comment: Well, if you are absolutely sure the only time you will ever modify that data is in the ItemDetailView, then you could do what Urban suggested or set a static boolean flag somewhere. Then you just check that flag to see if you need to update your ItemList. Finally, its not at all uncommon practice to update your list in onResume. Its entirely possible that your activity is paused for days, weeks, or even months.


Simple ordinal question

Question: Sorry I asked this question as a comment on other thread. I thought I would better ask it as a separate question since I have seen this notation more than few times now and I am not quite certain what it means (I would just like to be "sure" of whether it corresponds with what I describe below or not .... which I had thought of separately before). Since I have had this confusion for some months now after seeing this notation, I would appreciate an answer and be pretty happy to have it cleared.
First the definition .... for any (countable) ordinal $\alpha$ define $PS[\alpha]$ to be the minimal among the following: "smallest ordinal that can't be computed using an (ordinary) program which has access to some arbitrary well-ordering of $\mathbb{N}$ with order-type $\alpha$."
What I mean is that if you take some specific bijection between $\alpha$ and $\mathbb{N}$ (denote the bijection as $f$).Then if you denote $PS_f[\alpha]$ as the smallest ordinal that can't be computed using an (ordinary) program that has access to well-order relation corresponding to the bijection, we define:
$PS[\alpha]=$minimal value $PS_f[\alpha]$ among all possible bijections $f$
Now define:
$\beta_0=\omega_{CK}$
$\beta_{\alpha+1}=PS[\beta_\alpha]$
When $\alpha$ is a limit, $\beta_\alpha$ is defined as supremum of all smallers $\beta_x$'s (where $x<\beta$).
I hope this wasn't too long-winded.Now my question simply is how does $\beta_n$ correspond to notation $\omega^{CK}_n$. I mean obviously $\beta_0=\omega^{CK}_1$, what about beyond this?
Comment: Bumped to correct the somewhat misleading phrase "some arbitrary well-ordering of $\alpha$ (in terms of $\mathbb{N}$)."
Answer: Let me first rephrase your question a bit more clearly (if this isn't what you're asking, let me know). It sounds like you are asking about Muchnik (weak) reducibility of ordinals: if $\alpha,\beta$ are countable ordinals, we say $\alpha\le_w\beta$ if any copy of $\beta$ computes a copy of $\alpha$, where by "copy" we mean a relation on the naturals which is isomorphic to $\beta$ (resp. $\alpha$).
It's a theorem of Sacks that the following are equivalent, for a countable ordinal $\theta$:

$\theta$ is admissible.
For some real $r$, $\theta$ is the least ordinal with no $r$-computable copy.

This immediately shows that $\beta_n=\omega_{n+1}^{CK}$ for finite $n$. At transfinite levels, though, interesting things happen: your $\beta_\omega$ is just $\sup_{n\in\omega}(\omega_n^{CK})$, but this ordinal is not admissible (being an admissible limit of admissibles is stronger than just being admissible) -  this is because the map sending $\eta$ to the $\eta$th admissible is $\Sigma_1$, and in particular any admissible limit of admissibles $\alpha$ satisfies $\alpha=\omega_\alpha^{CK}$. So in fact we get the opposite pattern: $$\beta_\omega<\beta_{\omega+1}=\omega_\omega^{CK}<\beta_{\omega+2}=\omega_{\omega+1}^{CK}<...$$ Basically, what we're seeing here is that the "obvious" limit stage definition actually plays badly with the concepts at hand.
EDIT: Addressing the comments below:

The definition of $\omega_\alpha^{CK}$ that I'm using is "the $\alpha$th admissible" - that is, the unique ordinal $\gamma$ such that $L_\gamma\models KP$ and $\{\beta<\gamma: L_\beta\models KP\}$ has ordertype $\alpha$. Note that $L_\omega\models KP$, so $\omega_0^{CK}=\omega$ and $\omega_1^{CK}$ is what it should be under this definition.
The "higher off-by-one" pattern above goes always thus, except at admissible limits of admissibles (also called "recursively inaccessibles"). When $\theta$ is an admissible limit of admissibles, we get $\beta_\theta=\omega_\theta^{CK}$.

Let me end by mentioning a couple hopefully interesting facts. In the interest of full disclosure, this is part of my work - see respectively this paper (which was partly motivated by this work of Hamkins and Li) and this paper.

Muchnik reducibility is nonuniform - this is really what "weak" is referring to. We could also ask about Medvedev (strong) reducibility, $\le_s$: for countable ordinals $\alpha,\beta$ we write $\alpha\le_s\beta$ iff there is some single Turing functional $\Phi_e$ such that whenever $B$ is a copy of $\beta$, $\Phi_e^B$ is a copy of $\alpha$. This is extremely different; for example, we always have $\alpha\le\beta\implies\alpha\le_w\beta$, so Muchnik reducibility linearly (pre)orders the countable ordinals, but there is a club of countable ordinals which are pairwise $\le_s$-incomparable! Similarly, in general the set of ordinals Medvedev reducible to a given one is not closed downwards.
Muchnik (and Medvedev) reducibility makes sense in a broader context: it's really about countable structures, not just ordinals per se, and as such is part of computable structure theory. And even the countability restriction can be dropped, by looking at computations in generic extensions. While on the face of things this looks like a horrible intrusion of set theory, Shoenfield's absoluteness theorem then implies that we actually get something not too terrible; in particular, for all countable structures $\mathcal{A},\mathcal{B}$, the following are equivalent:

There is a generic extension of the universe in which $\mathcal{A}\le_w\mathcal{B}$.
In every generic extension of the universe in which $\mathcal{A},\mathcal{B}$ are each countable, we have $\mathcal{A}\le_w\mathcal{B}$.

This lets us continue Sacks' theorem into the uncountable: an ordinal $\alpha$ is admissible iff there is some real $r$ in some generic extension of the universe such that $\alpha$ is the least ordinal with no $r$-computable copy. And in fact the Medvedev version of this picture - which, by Shoenfield again, is similarly well-behaved - is a necessary (so far as I know) tool in proving the result mentioned in the previous bulletpoint.
Comment: Yes, I think you have understood rightly what I was asking.  In the terminology of the question, I was defining $PS[\alpha]$ to be the smallest ordinal such
that $PS[\alpha] \nleq_w \alpha$. Just a couple of more questions though.
(1) What exactly is the precise (and general) definition of $\omega^{CK}_\alpha$ then (sorry I just haven't seen it anywhere)? And is there a generally used phrase such as $\alpha$-th "this" for $\omega^{CK}_\alpha$?
(2) Secondly, you mentioned that $\beta_\omega<\beta_{\omega+1}=\omega_\omega^{CK}<\beta_{\omega+2}=\omega_{\omega+1}^{CK}<...$
 (continued)
Comment: Does that carry over for higher values? I mean is the following true or not(??): $\beta_{\omega \cdot 2+1}=\omega_{\omega \cdot 2}^{CK}<\beta_{\omega \cdot 2+2}=\omega_{\omega \cdot 2+1}^{CK}<...$
And does that carry over to limit values after that?
Comment: Also, regarding the second part of your answer, obviously some of this is too advanced for me. But I have seen Prof. Joel's paper (I mean just seen it, not read/understood it). Anyway, if the notion mention there is same as one here (https://math.stackexchange.com/questions/108348/can-we-implement-omegack-1-using-omegack-11-as-an-oracle) then I think I do understand the definition at least (since I do understand the construction in linked answer). Anyway, thanks for the answer. It was quite useful.
Comment: @SSequence See my edits.
Comment: One last question (in case you want to answer). In the definition of $\beta_1$ in terms of $\beta_0=\omega_{CK}$, $\beta_1$ was defined as smallest ordinal such that $\beta_1 \nleq_w \beta_0 $. However, if we take an actual well-order of $\omega_{CK}$ in terms of $\mathbb{N}$ (say in most obvious way by adjusting indexes of smallest programs that calculate ordinals less than $\omega_{CK}$), it seems plausible that using that specific well-order an (ordinary) program wouldn't be able to calculate $\beta_1$. I mean I wouldn't bet on it but it seems plausible. If it is true, good ref. for this?
Comment: @SSequence I'm not quite sure what you're asking. "$\beta_1\not\le_w\beta_0$" means *exactly* that there is some copy of $\beta_0$ which does not compute any copy of $\beta_1$. Or are you asking whether a *canonical* copy functions as such an example? (If so, yes: [Kleene's $\mathcal{O}$](https://en.wikipedia.org/wiki/Kleene%27s_O) is in some sense the canonical copy of $\omega_1^{CK}$, and does not compute any copy of $\omega_2^{CK}$.)
Comment: Okay  .... the last sentence in your last comment was more or less what I was asking about (and was saying could be plausibly true). Good references for this specific fact (the more the better I guess)? Anyway, thanks for help.
Comment: @SSequence For which fact - that $\omega_1^{CK}\not\ge_w\omega_2^{CK}$, or that the copy gotten from Kleene's $\mathcal{O}$ witnesses that? (The former is just an instance of Sacks' theorem.)
Comment: I meant the latter one
Comment: @SSequence You can prove this as follows. Fix a copy $A$ of $\omega_1^{CK}$ which doesn't compute any copy of $\omega_2^{CK}$. Kleene's $\mathcal{O}$ - and hence the canonical copy $C$ of $\omega_1^{CK}$ gotten from $\mathcal{O}$ - is hyperarithmetic in $A$. By Spector's theorem, any ordinal of which $C$ computes a copy has a copy computable from $A$. Since $A$ doesn't compute any copy of $\omega_2^{CK}$, neither does $C$. Basically, the canonical copy $C$ is "hyperarithmetically minimal."
Comment: It seems what you have written seems to address a part of my question.
But my question was probably pretty dumb (sorry my knowledge is very rudimentary). Let me state it very precisely (you can answer if your patience hasn't run out). Part of my question was also that if you define a function such as $f: \mathbb{N}^2 \rightarrow \mathbb{N}$ (also I am guessing this counts as a "copy" of $\omega_{CK}$ .... or very close to it at least) in the following manner: (a) If either of $x \notin \mathcal{O}$ or $y \notin \mathcal{O}$ is true then $f(x,y)=3$.
Comment: If both $x,y \in \mathcal{O}$ then we define (using the bar notation in wiki link) (a) $f(x,y)=1$ if  $\left | x  \right | < \left | y  \right |$ (b) $f(x,y)=0$ if  $\left | x  \right | > \left | y  \right |$ (c) $f(x,y)=2$ if  $\left | x  \right | = \left | y  \right |$. So I was asking that why a program with access to function $f$ can't compute a copy of $\omega^{CK}_2=\beta_1$. Anyway, I will have to do some studying up here obviously (for this question and related ones).
Comment: @SSequence Yes, that's exactly what I answered: the reason is that that canonical copy $f$ is "hyperarithmetically minimal" in the sense that it is hyperarithmetic in any *other* copy of $\omega_1^{CK}$. This means that if $f$ computed a copy $C$ of $\omega_2^{CK}$ then $C$ would be hyperarithmetic relative to *any* copy of $\omega_1^{CK}$, hence by Spector's theorem any copy of $\omega_1^{CK}$ would compute a copy of $\omega_2^{CK}$ - that is, $\omega_1^{CK}\ge_w\omega_2^{CK}$, and this contradicts Sacks' theorem.
Comment: OK thanks, that's great. Your answer was/is a bit advanced for me (and some terminology such as "canonical copy" I was very unsure what you meant). That's why I was trying to make sure you interpreted my question right (which you did). In due time I will understand this I suppose (at least I hope so) .... so this will be quite useful at some point. Anyway, that concludes this discussion.

Simple ordinal question


Two player pool, probability of winning

Question: I have what seems like a simple question, but it's been a while since I've done any P/S. So i come to SE for help!

Two player pool/billiards: P1 has probability p of sinking a ball on any shot and has N balls remaining, while P2 has prob q and M balls remaining.
Question: What is the probability of the first player winning?

I can type out my reasoning (and will in an edit - will post before i reason it out though), but my answer has come down to:
$$\sum_{j=0}^{M-1} [p^N q^j \sum_{i=0}^\infty [(1-p)^i (1-q)^i]]$$
Is this correct or close?
Reasoning:
not really theoretical reasoning, but extrapolating the simple cases outwards:
(hit = h, miss = m, with probability p = w/p)
1 ball each: possible victory paths -
P1 wins w/p,
P1 m w/ (1-p), P2 m w/ (1-q), P1 wins w/ p
... etc - $p \sum_{i=0}^\infty (1-p)^i (1-q)^i$
2 and above balls each:
At some point, all the P1 hits must occur - $p^N$
All possible amounts of P2 hits must be accounted for - $\sum_{j=0}^{M-1} q^j$
Every miss variation is accounted for - * <-- This is where i think I am wrong. Is it actually a double sum in and of itself? IE $\sum_{i=0}^\infty \sum_{k=0}^\infty (1-p)^i (1-q)^k$ ?
EDIT: Some wolfram alpha shows me that $\sum_{i=0}^\infty (1-p)^i = \frac 1 p$, so I guess my final final equation can be simplified to
$$\sum_{j=0}^{M-1} \frac{p^N q^j}{pq} $$ ??
etc.
Comment: The problem neglects my usual way of winning: the opponent scratching on the 8 ball.
Comment: Formula seems wrong as $N=0$, $p$ is anything, does not always lead to a probability of 1.
Comment: hmm. good point
Comment: well, lmao that final thing can't be right, that's > 1 for one scenario hahahaha oops
Comment: hahaha you must usually play against me then
Answer: Let $P(k)$ denote the probability that $P_1$ sinks all $N$ balls by $k^{th}$ turn (i.e., with exactly $k-1$ misses). The final shot has to be a hit. Before that there were $N-1$ hits and $k-1$ misses, which can be arranged in any order. So
$$
P(k) = \binom{N+k-2}{N-1}p^{N}(1-p)^{k-1}
$$
Similarly if $Q(k)$ denotes the probability that $P_2$ sinks all $M$ balls by $k^{th}$ turn, then
$$
Q(k) = \binom{M+k-2}{M-1}q^{M}(1-q)^{k-1}
$$
Now, for $P_1$ to win, the number of turns he takes to finish must not be more than that of $P_2$. So, required probability is
$$
\sum_{i\leq j} P(i)Q(j)
$$
This might not give a quick idea about the probability value, but compuationally, I don't think it is difficult as computing $P(k)$ and $Q(k)$ can be done very fast by recursion and even though it is an infinite sum, it converges quickly. I am not sure if this is the best we can get though.
Comment: It looks like $P(k)$ and $Q(k)$ can be greater than $1$. Also, wouldn't sinking $N$ balls in $k$ turns mean you missed $k-N$ times, not $k-1$?
Comment: @compadee Thanks for pointing that out. $p$ and $1-p$ terms should actually be in numerator. Now, we can show that $P(k)$ and $Q(k)$ are always $\leq 1$. By a turn, I mean a series of hits without any miss ($P_2$ gets his turn only when $P_1$ misses).
Comment: This is excellent, thank you! It also looks pretty much like the simplest form you can take it too. Nice work

Two player pool, probability of winning


Send platform message to Android method from background flutter isolate

Question: I want to have a dart background service running forever (isolates) which will communicate with a server through websockets. And I have an API for Android which gather information to send to the server.
How can I invoke that Android methods which use callbacks and everything from the background isolate?
EDIT
So far in dart I created an Isolate to periodically invoke the poolSong method in background even if the user is using another app or has the screen turned off.
But that gives me the error below...on the github issues they say that I can't send a platform message from a different Isolate unless it's the main. But if I do it from the main isolate, when the user exit the app that isolate will terminate too.
MainDart
class _MyHomePageState extends State<MyHomePage> {
    final GlobalKey<ScaffoldState> scaffoldKey = new GlobalKey<ScaffoldState>();
    static const platform = const MethodChannel('mainService');

    static _poolSong(SendPort sendPort) async {
        const oneSec = const Duration(seconds:1);
        new Timer.periodic(oneSec, (Timer t) => platform.invokeMethod('poolSong'));
    }

    @override
    void initState() {
        super.initState();
        ReceivePort receivePort = ReceivePort();
        Isolate.spawn(_poolSong, receivePort.sendPort);
    }
        ·
        ·
        ·

MainActivityJava
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    GeneratedPluginRegistrant.registerWith(this);

    new MethodChannel(getFlutterView(), CHANNEL).setMethodCallHandler(
        (call, result) -> {
            if (call.method.equals("startService"))
                startService();
            if (call.method.equals("poolSong"))
                poolSong();
        }
    );
}

ERROR
E/flutter (25412): [ERROR:flutter/runtime/dart_isolate.cc(717)] Isolate (765499726) 'main.dart:_poolSong()' exited with an error
E/flutter (25412): [ERROR:flutter/shell/common/shell.cc(186)] Dart Error: Unhandled exception:
E/flutter (25412): error: native function 'Window_sendPlatformMessage' (4 arguments) cannot be found
E/flutter (25412): #0   Window.sendPlatformMessage (dart:ui/window.dart:811:9)
E/flutter (25412): #1   BinaryMessages._sendPlatformMessage (package:flutter/src/services/platform_messages.dart:39:15)
E/flutter (25412): #2   BinaryMessages.send (package:flutter/src/services/platform_messages.dart:87:12)
E/flutter (25412): #3   MethodChannel.invokeMethod (package:flutter/src/services/platform_channel.dart:286:49)
E/flutter (25412): <asynchronous suspension>
E/flutter (25412): #4   _MyHomePageState._poolSong.<anonymous closure> (package:musictie/main.dart:37:54)
E/flutter (25412): #5   _Timer._runTimers (dart:isolate/runtime/libtimer_impl.dart:382:19)
E/flutter (25412): #6   _Timer._handleMessage (dart:isolate/runtime/libtimer_impl.dart:416:5)
E/flutter (25412): #7   _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12)

Comment: please provide some code (what you have done until now) to get better response
Comment: @MohamadArmoon Done. Thanks for the replay. I read the article 'Executing Dart in the Background with Flutter Plugins and Geofencing' and there they say that for background execution I must use isolates.
Answer: This error happens when trying to run dart code when the app is in the garbage (app closed), Until now the best way for dealing with background service is to do what you need in native code, and dont try to run anything from dart when app closed.
Answer: Since Flutter 3.7 plugins and platform channels can be used from any isolate, not only from the root one.
References:

https://medium.com/flutter/introducing-background-isolate-channels-7a299609cad8
https://github.com/flutter/flutter/issues/13937
Comment: for me it does not work


How do I go about solving this problem?

Question: A card from a pack of 52 cards is lost. From the remaining cards of the pack, two cards are drawn and are found to be both diamonds. Find the probability of the lost card being a diamond?
could you explain what is the relation between the lost card and the cards we draw. I mean how could you go about solving this ridiculous question but there exists a definite solution. If I am wrong please tell me what am I missing.
Comment: please up vote if you think this is a nice problem?
Answer: The lost card influence the probability of drawing certain patterns of cards from the remaining deck.
Let $L$ be the event that the lost card is a diamond.
Let $D$ be the event that two cards drawn from the remaining deck are diamonds.
We are interested in $Pr(L|D)$
From Bayes rule, we have $$Pr(L|D)=\frac{Pr(D|L)Pr(L)}{Pr(D)}$$
Also note that $$Pr(D)=Pr(D|L)Pr(L)+Pr(D|L^c)Pr(L^c)$$
These are the ingredients needed, are you able to compute it?
Answer: If the lost card is a diamond, you are drawing from a pack with $12$ diamonds and $39$ other cards.  What is the chance you get two diamonds from that pack?  If the lost card is not a diamond, you are drawing from a pack with $13$ diamonds and $38$ other cards.  What is the chance you draw two diamonds from that pack?  It is higher, which shows how the suit of the lost card matters.
Really what you are asked is to compare the chance of drawing three diamonds with the chance of drawing one non-diamond and two diamonds from a standard pack.

How do I go about solving this problem?


Trouble working with a one-dimensional Java array with unspecified number of elements

Question: I have a one-dimensional Java array:
    //Array with the thread of each client
    private Client_Procesor_Thread Client_Thread_List[];

When I want to use it, I do this:
    Client_Thread_List[Next_Client] = new Client_Procesor_Thread();

You may notice that, before creating the thread, I have to create the array like this:
    Client_Thread_List = new Client_Procesor_Thread[15];

However, I can't create the array, because I don't know how many threads I will have. This is a server, and I don't know how many clients I will have.
I know in other languages like Visual Basic.NET I can create an array of 15 elements for example, and after this, resize it to a larger one without losing the stored elements if I need to.
¿How would I proceed in Java?
Thanks for all,
Btc
Comment: you should use ArrayList
Comment: use a dynamic list, such as [ArrayList](http://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html)
Comment: Your problem is with threads to begin with -- what if you expressed your problem with this angle?
Answer: If you need a kind of *growing array, use an ArrayList
Comment: This seems like an answer rather than a comment.
Comment: This should be a comment please!!
Comment: @RDC: I know it's a very short answer, but I think t answers the question.
Comment: Oops.. it was my mistake,@MrSmith42 Thank you for help +1
Answer: private List<Client_Thread> client_Thread_List = new LinkedList<Client_Thread>();

see http://docs.oracle.com/javase/6/docs/api/java/util/LinkedList.html
example http://www.cs.washington.edu/education/courses/cse341/06au/java/LinkedListExample.java
Comment: I did this just trying while debuging, because i didn't see your answer. Thanks anyway!


How to get Panelizer to only affect a specific node

Question: I have Panelizer installed on my Drupal 7 site. I would like to use Panelizer for only specific nodes and not have it apply on all my nodes (as it currently is). Reading the documentation it seems it is possible to do this but I can't get it to work.
In Admin->Structure->Pages I have the node_view enabled, and I see an entry for my particular node there as well (page-videos_view with the correct url path). If I disable the node_view, then Panelizer is effectively disabled as it doesn't apply any panels even though the view I want is still active.
I assume I am missing a setting somewhere but don't know what it is. Can anyone help out?
Answer: I have figured out my error. You need to go to Admin->Configuration->Content Authoring->Panelizer and make sure you don't have the Provide Default Panel option selected for the content type you are using Panelizer on.


Variable-length form submissions

Question: I'd like to create a form_for statement which allows the user to select elements one-by-one from a drop down menu (or an auto-complete field). As the user selects each item, they should be displayed in list format at the top of the screen. When the user has finished selecting elements, the form can be submitted and the list of selected elements is sent to the receiving controller action.
Can someone tell me how to implement this?
Answer: Maybe Chosen is what you need.
Take a look at:
http://harvesthq.github.com/chosen/
Comment: That looks like an interesting approach. However, the user wouldn't be selecting from a pre-defined set of elements. I realized that my original question is misleading because is said "auto-complete," implying that there is a set of elements to choose from. However, my current goal is for the user to type in a pair of strings for each element to be added to the list.
Comment: Yes! Thanks so much Laurens. This is much closer to what I need.

Do you have any suggestions for how I can use this in a way that allows the user to easily submit a set of name-email pairs?
Comment: Currently I don't have the time to do a writeup for this.

This should get you on the right track:
http://railscasts.com/episodes/75-complex-forms-part-3
Also check out the other rails casts about complex forms, hope this helps.
Comment: http://stackoverflow.com/questions/4812003/adding-fields-dynamically-in-a-nested-model-form-in-rails-3 has a good answer as well.
Comment: It says I need a reputation of 15 before I can vote up answers. Sorry.
Comment: You don't have to vote up, you can mark the arrow left from the answer to 'accept' the answer.

Variable-length form submissions


Importing large flat-file data sources with Drupal 7 with Views 3 integration

Question: My goal is to produce a fast, reliable and automated method for accessing read-only data contained in several very large flat-file data sources (CSVs, Fixed Width and XML docs) using Drupal 7 that can be queried against using the Views 3 module.  I would prefer to use already available modules, but building a custom module is also an option.
To help rule out modules and methods not suited for the task, here are the stats on the files I'm working with:

Yearly Import: 8,500,000 line CSV file. (Purged and reloaded yearly. Has primary key.)
Weekly Import: 350,000 line fixed width file. (Purged and reloaded weekly. No primary key.)
Hourly Import: 3,400 line CSV file. (Would like to update and sync as often as possible, but no more than every 20 min. Has primary key)
Daily Import: 200 item XML file. (Purged and reloaded daily. Has primary key)

Converting between the three formats is not an issue and can be done if it will improve import performance or allow better tools to be made available. (AWK for Fixed Width to CSV, etc.) Retrieval and conversion automation is easy via cron and sh scripts, but still need to automate the Drupal 7 integration. The use of custom tables is also possible as long as vews can reference the data using relationships.
What would be the best practice to accomplish this type of data integration with Drupal 7?  Also, am I leaving out any important details regarding the data or what I'm trying to accomplish?

Here are a few projects I'm currently looking at to find a solution.  I would like to expand on this to help others decide which route to take when working with larger data imports.
Importing data into nodes:

Feeds (Currently Alpha for D7)

Feeds will import the data reliably.  Speed is reasonable for the smaller data sources but is too slow for the 300k+ tables.
Automation available using cron and Job Scheduler (Currently Alpha for D7).

Migrate

Not having an index or unique key available in the source data is making this difficult to use.  It is faster than feeds, but still slow to import the very large tables.
Automation is available via drush and cron.
Custom Tables Instead of Nodes

Data Module (Currently Alpha for D7)

The Data module looks very promising, but is very buggy for D7 at the moment. The automation and import speed requirements would be easily met using data, but reliability is lacking.  The views integration (link is for D6) looks very promising.

Table Wizard (Not available for D7)

Added this for reference.  There is no D7 candidate at this point, but could serve as a starting point for a custom module.

Views Schema (Abandoned, D6 only)

Added this for reference.  This seems to have been absorbed by Table Wizard in Drupal 6.  Again, added only for reference.

Importer (RC2 for D7)

Seems to require Table Wizard (D6 only) for Views integration.  Added for reference, but doesn't meet Views requirement.

@MPD - Added "Custom Tables" as a possible solution, and expanded modules.  Thank you for this addition.
Answer: My gut tell me that this plan will make your servers catch on fire...
Seriously, if you are churning that much data, then I think you need to keep the data in an external datasource and then integrate it with Drupal.
My initial thought would to use two databases for the external data, so that you can do the weekly import w/o disturbing things too much.  In other words, get database A up and running, and then import into B.  When the import is done, make B the live source.  Then wipe and import into A.
I have done a lot of integration of external datasources into Drupal, and it really isn't that hard.  I gave an overview in Transition plan for PHP5 abomination to Drupal.  That was for Drupal 6, but the same thing basically applies to Drupal 7.   Essentially, you simulate what CCK / Fields API does with your own interface.
Not having a UUID for the weekly database really throws a wrench in the works, though.  That part requires a lot of though, more that can be provided in a Q/A forum like this.
If you really do want to go down the import route, I would bail on Feeds and Migrate and write your own import script.  Basically, you do the initial bookstrap process from index.php, query your datasource, make your nodes, and then save them.  Programmatically making nodes is easy.
The best way to start out with this is to make a node with the UI, then print_r it, and replicate the object with code in your import script.  Taxonomy, files, and noderefs, are hard parts, but you just need to get familiar with these portions of the API to build up these object properties.  Once you have a valid node object, then you can just do a node_save().  Make sure you set a very large limit with set_time_limit() so your script runs.
EDIT BELOW TO ADDRESS CLARIFICATION/EXPANSION:
Personally, we stopped using the contrib module based approaches for data imports a while ago.  They do work mostly well, but we just ended up spending way too much time fighting them and decided the cost/benefit was too low.
If you really need the data in Drupal proper, then my opinion about a custom import script hasn't changed.   One of the modules you reference could be used as a starting point for how to build the node objects, then just loop through your data build nodes and save them.  If you have a PK, you can easily add in logic to search the database and node_load(), modify, and save.  An import script is really only a few hours work if you know the Drupal API.
If views integration is a key (and it sound like it is based on the edit) and you want to do the external tables approach, then your best option is do do a custom module and implement hook_views_data to get your data into views.  More than likely, you will custom module anyway to support your datasource, so adding this hook shouldn't be that much more work.  The TW and Data modules should have some example to get you going.
Personally, though, I have never found the views integration with external data to be really worthwhile.  In the cases where I have considered it, the data was just too "different" to work well with a views based approach.  I just end up using the method I described in the "abomination" link above.
Comment: You've brought up three excellent points, and I'm going to adjust my question accordingly. Mass importing and exporting would be nice but when importing hundreds of thousands, or possibly millions of nodes at this point seems at best, unrealistic. Custom tables could also be very usable if they can be integrated with views.  Thank you for your response @MPD.
Answer: I think a node based (or even entity based) approach will burn out your server with millions of node. Besides, looking at your hourly import, that means your'll make a node_save() at least once a second. That's too much for Drupal and cause a performance problem.
The reason behind that is for those content, you won't need any hook mechanism, you won't need pathauto (but you can manually create alias, it's much cheaper than pathauto), you won't need fields... Write a simple "INSERT" query is 100x faster than node_save() or entity_save().
1/ IMHO the best option is a custom table and a custom module for your data import, then write Views handlers for Drupal integration.
2/ The database cache is invalidated during the hourly import. If it takes too much time, you can think about a replication. In the easiest form, create two identical tables, use the first one, import to the second, switch your Drupal configuration to use the second table, sync the 2nd table to the 1st (then optionally switch back to the first). Another solution is in your custom import script, prepare and group the INSERT/UPDATE queries, then only send it at the end  in one transaction to reduce the database write time.


can one have multiple html files in a single template directory in django?

Question: I have a templates directory with two other directories, home, and diagnosis each directory having its own respective HTML file. So I want to know if that's the best way to do it or am I doing it the wrong way?
Link to access the directory structure
Answer: Your question's answer is easily get from this site SimpleIsBetterThanComplex
This site is provide easy way to learn for beginner's ....
Comment: yes this is really good for beginner, and i was glad to appreciate your answer
Answer: I've read Two Scoops of Django Best Practices For Django book and it really provides the best practices to use Django, in the templates chapter,  there are two best styles for laying out templates.

2 - Tier Template Architecture like:
templates/
    base.html
    dashboard.html # extends base.html
    profiles/
        profile_details.html
        profile_form.html

which is the style you are using

3 - Tier Template Architecture like:
templates/
    base.html
    dashboard.html # extends base.html
    profiles/
        base_profile.html # extends base.html
        profile_details.html # extends base_profile.html
        profile_form.html # extends base_profile.html

The author said the 2-tier architecture is best for sites with a consistent overall layout from app to app.
And the 3-tier architecture is best for websites where each section requires a distinctive layout.
I highly recommend this book for the best practices in using Djang.
https://www.amazon.com/Two-Scoops-Django-Best-Practices/dp/0981467342


Sum of Fourier Series, Sine function to Euler

Question: This might potentially be one of the dumbest question, but I'm gonna ask anyway.
view image pls
How do I get $\pi/2$?
I know for a fact $e^{jθ} = cos\theta  + j sin \theta$
but in this case $e^{j3\omega_0t}$... how do I find $\theta$?
Btw period is given as $T_0 = 2\pi/\omega_0$
I do not know what to put on the title because I am new to this topic, so feel free to offer your suggestions
Answer: Notice, when $\text{z}\in\mathbb{C}$ and $\Re\left[\text{z}\right]=0$ and $\Im\left[\text{z}\right]>0$:
$$\arg\left(\text{z}\right)=\frac{\pi}{2}+2\pi\text{k}$$
Where $\text{k}\in\mathbb{Z}$ and I worked with radians.
Comment: OMGGGG I KNEW IT WAS EASY BUT I DIDNT KNOW IT WAS THIS EASY!!!!!! Thank you so much!!!!!
Comment: @JoeyPeh Hahaha you're welcome, mate :) I'm glad that I could help.


Excel and Capital IQ Plug-in

Question: Is there a way to refresh Capital IQ Plug-in when programmatically open Excel file. Any explanation will be much appreciated.
Answer: A little bit late, but maybe this can help.
In the Capital IQ plugin, go to settings --> Data --> Automatically retrieve data
Make sure that this box is checked.
This will automatically refresh all data in that particular worksheet.
Something to note, ensure all Excel instances are closed (with Task Manager) before opening your file - Capital IQ plugin tends to "live" in the first instance and will throw an error when you try to refresh.

Excel and Capital IQ Plug-in


How to run JetBrains WebStorm refactoring 'change ES6 import' for the whole project

Question: After refactoring a project, some files were moved to another npm package and after that source files changed to re-export them from a new place (to minimize migration effort).
After some time I did notice useful refactoring named 'Change ES6 import' which allows to change import from re-exported type to original:

It is possible to run it in a batch for all files in the project? Is there another refactoring/way to do that?
Comment: I do not think it is possible, Fix All is only supported for inspections not intentions. https://intellij-support.jetbrains.com/hc/en-us/community/posts/206886125/comments/360000724419
Answer: Try this:

Open project tree. Usually it's Alt + 1 key combination.
Select required folder and open context menu (mouse right btn click).
Select Reformat code option (Ctrl + Alt + L) or Optimize imports (Ctrl + Alt + O).
Comment: unfortunatelly, this method cannot run required refactoring
Comment: Maybe, you should write your own script or use `sed` utility to replace text in files


PHP MySQL query statement problems

Question: I've been trying to fix this PHP MySQL query all afternoon and am losing my mind! the following query works!:
$sql = "SELECT o.order_id, CONCAT(o.firstname, ' ', o.lastname) AS customer, o.shipping_code, o.total, o.channel, o.currency_code, o.currency_value, o.date_added, o.date_modified, c.image,
        cg.name AS cgroup, CONCAT(o.payment_address_1, '
', o.payment_city, ', ', o.payment_zone, ' ', o.payment_postcode) AS address, o.telephone AS phone, o.email AS email,
        (SELECT os.name FROM " . DB_PREFIX . "order_status os WHERE os.order_status_id = o.order_status_id AND os.language_id = '" . (int)$this->config->get('config_language_id') .
        "') AS order_status FROM `" . DB_PREFIX . "order` o LEFT JOIN `" . DB_PREFIX . "customer` c ON(o.customer_id=c.customer_id) LEFT JOIN `" . DB_PREFIX . "customer_group_description`
        cg ON(c.customer_group_id=cg.customer_group_id)";

but when I try to add the following to it it breaks.
(SELECT action FROM " . DB_PREFIX . "donate_history h LEFT JOIN " . DB_prefix . "order od ON (h.customer_id = od.customer_id)),

I dont if Im writing it right. I've rewritten it, removed parts, tried to make as simple as possible, but I can't fix it. This is what I've been trying :
$sql = "SELECT o.order_id, CONCAT(o.firstname, ' ', o.lastname) AS customer, o.shipping_code, o.total, o.channel, o.currency_code, o.currency_value, o.date_added, o.date_modified, c.image,
        cg.name AS cgroup, CONCAT(o.payment_address_1, '
', o.payment_city, ', ', o.payment_zone, ' ', o.payment_postcode) AS address, o.telephone AS phone, o.email AS email,
        (SELECT action FROM " . DB_PREFIX . "donate_history h LEFT JOIN " . DB_prefix . "order od ON (h.customer_id = od.customer_id)),
        (SELECT os.name FROM " . DB_PREFIX . "order_status os WHERE os.order_status_id = o.order_status_id AND os.language_id = '" . (int)$this->config->get('config_language_id') .
        "') AS order_status FROM `" . DB_PREFIX . "order` o LEFT JOIN `" . DB_PREFIX . "customer` c ON(o.customer_id=c.customer_id) LEFT JOIN `" . DB_PREFIX . "customer_group_description`
        cg ON(c.customer_group_id=cg.customer_group_id)";

Comment: what is the error ?
Comment: Two observations: calling a table 'order' is asking for trouble; what if there's a payment_address_2?
Comment: You don't have any alias for the column. After the closing parentheses write something like `as action` see the others to see what I mean.
Comment: what's the status of *this* question? you've been comments and an answer, yet nothing from you.
Comment: @Strawberry *"calling a table 'order' is asking for trouble"* - Right you are and is unknown as to what's going on with the question.
Comment: Adam; if the question's a stale mate, it should be deleted. You realize that `order` is a MySQL reserved word, I hope you know that that needs special attention as your query *will* error out, as posted.
Answer: Looks like your new subquery you tried adding didn't have an alias.
SELECT o.order_id, CONCAT(o.firstname, ' ', o.lastname) AS customer, o.shipping_code, o.total, o.channel, o.currency_code, o.currency_value,
   o.date_added, o.date_modified, c.image, cg.name AS cgroup, CONCAT(o.payment_address_1, '
', o.payment_city, ', ', o.payment_zone, ' ', o.payment_postcode) AS address,
   o.telephone AS phone, o.email AS email,
    (SELECT action
         FROM " . DB_PREFIX . "donate_history h
         LEFT JOIN " . DB_prefix . "order od
             ON (h.customer_id = od.customer_id)) AS donate_history,
    (SELECT os.name
        FROM " . DB_PREFIX . "order_status os
             WHERE os.order_status_id = o.order_status_id AND os.language_id = '" . (int)$this->config->get('config_language_id') .
    "') AS order_status
    FROM `" . DB_PREFIX . "order` o
        LEFT JOIN `" . DB_PREFIX . "customer` c
             ON(o.customer_id=c.customer_id)
        LEFT JOIN `" . DB_PREFIX . "customer_group_description` cg
             ON(c.customer_group_id=cg.customer_group_id)

PHP MySQL query statement problems


Subir para o BitBucket um branch já criado

Question: Tenho um projeto no Git Local e quero subir para o bitbucket, porem ao seguir as instruções:
Get started with command line

Step 1: Switch to your repository's directory  cd /path/to/your/repo
Step 2: Connect your existing repository to Bitbucket git remote add
  origin ssh://firstname.lastname@example.com/nutriex_project/checkoutnovo.git git
  push -u origin master

Ocorre o seguinte erro:

$ git push -u origin master Warning: Permanently added the RSA host
  key for IP address '188.8.131.52' to the list of known hosts.
  Permission denied (publickey). fatal: Could not read from remote
  repository.
Please make sure you have the correct access rights and the repository
  exists.
Comment: Segue esse tutorial da locaweb, tive o mesmo problema https://wiki.locaweb.com.br/ptbr/Configurando_Github_e_Bitbucket_na_Hospedagem_Linux 1º Definindo Chaves SSH
2º Configurando no Bitbucket
Answer: Você precisa pegar a chave pública do seu computador, que geralmente está em
<code>/home/<user>/.ssh/id_rsa.pub
</code>
e adicioná-la nas suas configurações do bitbucket.
Na sua conta do Bitbucket, vá em
<code>Bitbucket Settings > SSH Keys > Add Key
</code>
e, na janela que se abrir, você precisa adicionar o conteúdo do arquivo id_rsa.pub
Answer:
Please make sure you have the correct access rights and the repository exists.

Já verificou se voce tem as pemissoes corretas?
Você precisa de adicionar sua chave ssh no bitbucket, Segue o link de um tutorial deles explicando:
https://confluence.atlassian.com/bitbucket/set-up-ssh-for-git-728138079.html


How to run ./configure from a different folder?

Question: I'm writing an ansible playbook which is supposed to compile from source some tool and install it.
While checking my playbook, I've tried running the relevant commands manually to check their output.
So, In my ansible play book I've configured this:
- name: Configure kafkacat
      command: . {{ install_dest }}/configure arg1
        args:
          enable_json: "--enable-json"
      sudo: yes

But before running the playbook, I ran manually:
. /tmp/kafkacat/configure --enable-json which is supposed to be equivalent to running ./configure in the current directory, but the command fails and I get an error.
So how can it be done?
Thanks in advance
Answer: In a lot of cases, depending on the software configure relies on having includes and headers opened from the current working directory.
ansible does support a "chdir" argument. Maybe that is the solution that will work for you?
Answer: No, I don't think they are equivalent.
The dot without following space replaces the path to your current directory, whereas a dot preceding a space makes commands of a script run in your current shell.
There is a good explanation here :
https://askubuntu.com/q/232932


PubSub in App Engine Standard Java 8: NullPointerException at ApiProxyImpl$CurrentRequestThreadFactory.newThread(ApiProxyImpl.java:1267)

Question: In a new App Engine standard Java 8 environment I'm seeing the following error when trying to publish to a PubSub topic:
io.grpc.internal.ChannelExecutor drain: Runnable threw exception in ChannelExecutor (ChannelExecutor.java:89)
java.lang.NullPointerException
    at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory.newThread(ApiProxyImpl.java:1267)
    at com.google.common.util.concurrent.ThreadFactoryBuilder$1.newThread(ThreadFactoryBuilder.java:162)
    at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:612)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925)
    at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1587)
    at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:336)
    at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:555)
    at io.grpc.internal.ManagedChannelImpl.rescheduleIdleTimer(ManagedChannelImpl.java:334)
    at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:299)
    at io.grpc.internal.ManagedChannelImpl$4$1.run(ManagedChannelImpl.java:357)
    at io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:87)
    at io.grpc.internal.ManagedChannelImpl$4.get(ManagedChannelImpl.java:359)
    at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:218)
    at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
    at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:62)
    at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:276)
    at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
    at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:186)
    at com.google.pubsub.v1.PublisherGrpc$PublisherFutureStub.publish(PublisherGrpc.java:460)
    at com.google.cloud.pubsub.v1.Publisher.publishOutstandingBatch(Publisher.java:329)
    at com.google.cloud.pubsub.v1.Publisher.publishAllOutstanding(Publisher.java:304)
    at com.google.cloud.pubsub.v1.Publisher.access$500(Publisher.java:79)
    at com.google.cloud.pubsub.v1.Publisher$5.run(Publisher.java:283)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:295)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Related issue is here with a bit more information: https://github.com/GoogleCloudPlatform/google-cloud-java/issues/2275
This is a Gradle project with the following dependencies:
dependencies {
  providedCompile 'javax.servlet:javax.servlet-api:3.1.0'
  providedCompile 'com.google.appengine:appengine:+'
  testCompile 'org.eclipse.jetty:jetty-server:9.+'
  testCompile 'org.eclipse.jetty:jetty-servlet:9.+'
  compile group: 'com.google.guava', name: 'guava', version: '19.+'
  compile group: 'com.google.inject', name: 'guice', version: '4.+'
  compile 'com.google.appengine:appengine-api-1.0-sdk:1.+'
  compile 'com.google.cloud:google-cloud-storage:1.2+'
  compile group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '0.20+'
  compile ('com.google.cloud:google-cloud-datastore:1.2+')
  compile('com.google.apis:google-api-services-cloudkms:v1-rev15-1.22.0') {
     exclude(group: 'com.google.guava', module: 'guava-jdk5')
  }
  compile(group: 'com.google.api-client', name: 'google-api-client', version: '1.+') {
     exclude(group: 'com.google.guava', module: 'guava-jdk5')
  }
  compile(group: 'com.google.api-client', name: 'google-api-client-appengine', version: '1.+') {
     exclude(group: 'com.google.guava', module: 'guava-jdk5')
  }
}

I'm deploying by passing the war file to appcfg.sh rather than using the App Engine Gradle plugin.
PubSub works fine in my Java 8 flexible environment with older PubSub libraries (0.11 series), but since switching to the Java 8 standard environment with newer PubSub libraries (0.20 series) I hit this error every time.
The only other reference to this error I've seen is in this Google Groups discussion: https://groups.google.com/d/msg/google-appengine/2UTg4Eqskyk/PbqYEK3GAAAJ
UPDATE 1:
After trying the truly scary work-around suggested in that Google Groups thread above (https:// gist.github.com/cmaan/7752e3c4fd0b1ba90a745cb6db232206) the NPE has stopped happening.  However, there is a new problem now:
Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED
    at io.grpc.Status.asRuntimeException(Status.java:543) ~[grpc-core-1.4.0.jar:1.4.0]
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:442) ~[grpc-stub-1.4.0.jar:1.4.0]
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) ~[grpc-core-1.4.0.jar:1.4.0]
    at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76) ~[grpc-core-1.4.0.jar:1.4.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:512) ~[grpc-core-1.4.0.jar:1.4.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:429) ~[grpc-core-1.4.0.jar:1.4.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:544) ~[grpc-core-1.4.0.jar:1.4.0]
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52) ~[grpc-core-1.4.0.jar:1.4.0]
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:117) ~[grpc-core-1.4.0.jar:1.4.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_112-google-v7]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_112-google-v7]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_112-google-v7]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:295) ~[na:1.8.0_112-google-v7]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_112-google-v7]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_112-google-v7]
    ... 1 common frames omitted
Caused by: java.io.IOException: Could not get the access token.
    at com.google.auth.oauth2.AppEngineCredentials.refreshAccessToken(AppEngineCredentials.java:136) ~[google-auth-library-oauth2-http-0.7.0.jar:na]
    at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:149) ~[google-auth-library-oauth2-http-0.7.0.jar:na]
    at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:135) ~[google-auth-library-oauth2-http-0.7.0.jar:na]
    at io.grpc.auth.GoogleAuthLibraryCallCredentials$1.run(GoogleAuthLibraryCallCredentials.java:110) ~[grpc-auth-1.4.0.jar:1.4.0]
    ... 7 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112-google-v7]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112-google-v7]
    at com.google.auth.oauth2.AppEngineCredentials.refreshAccessToken(AppEngineCredentials.java:131) ~[google-auth-library-oauth2-http-0.7.0.jar:na]
    ... 10 common frames omitted
Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call memcache.Get in a thread that is neither the original request thread nor a thread created by ThreadManager
    at com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:844) ~[runtime-shared.jar:na]
    at com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:183) ~[runtime-shared.jar:na]
    at com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:180) ~[runtime-shared.jar:na]
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:93) ~[appengine-api.jar:na]
    at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28) ~[appengine-api.jar:na]
    at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java:51) ~[appengine-api.jar:na]
    at com.google.appengine.api.appidentity.AppIdentityServiceImpl.getAccessToken(AppIdentityServiceImpl.java:300) ~[appengine-api.jar:na]
    ... 14 common frames omitted

Answer: I was able to find a work-around to be able to publish to a PubSub topic in the Java 8 standard environment until a permanent fix is released: https://github.com/GoogleCloudPlatform/google-cloud-java/issues/2275
It's necessary to first obtain credentials from within an App Engine request thread.  Then, pass those credentials to a new Publisher that is created inside a non-request thread, such as one created by a Java ExecutorService.  This combination works but is not very efficient.


Diameter of coins with a given height and source volume?

Question: We need to make $1.15\times10^5$ coins with a thickness of $2.0\times10^{-3}m(2mm)$ from a $0.12m^3$ block of silver. I'm trying to calculate what diameter the coins will end up being.
Did I set up the equation correctly where I'd get my answer from solving for D? I tried to draw it out to help understand what needs to be done. $$0.12m^3 = (1.15\times10^5) \times ((2.0\times10^{-3}m)\times \frac{\pi D}{2})$$
My solution:
$$0.12m^3 = (1.15\times10^5) \times ((2.0\times10^{-3}m)\times \pi r^2)$$
  $$\frac{0.12}{(1.15\times10^5)} = (2.0\times10^{-3})\times \pi r^2$$
  $$\pi r^2 = \frac{(0.12)}{(2.0\times10^{-3})(1.15\times10^5)}$$
  $$r^2 = \frac{(0.12)}{\pi(2.0\times10^{-3})(1.15\times10^5)}$$
  $$ 2r = 2\sqrt{\frac{(0.12)}{\pi(2.0\times10^{-3})(1.15\times10^5)}} = 0.025774m$$
  $$ D = 2.5774cm $$
Comment: One way to find the error is to check the units.  The left side has dimensions of volume ($m^3$) while the right has dimensions of $m^2$ because $D$ has dimensions of $m$.
Answer: The term $\dfrac{\pi D}{2}$ needs to be changed to  $\dfrac{\pi D^2}{4}$.
This is because the area of a circle of radius $r$ is $\pi r^2$, so the area of a circle of diameter $D$ is $\dfrac{\pi D^2}{4}$.  Thus if the coin has thickness $t$, it has volume $\dfrac{\pi D^2}{4}t$. If we want to make $n$ coins, and the amount of silver available is $V$, then
$$V=\frac{\pi D^2}{4}tn.$$
Comment: Oh that makes sense I was wondering if I did that conversion correctly. Thanks.
Answer: Ok so you know that $\pi r^2(2*10^{-3})*1.18*10^5=0.12$
So $r^2=\frac {0.12}{2\pi10^{-3}*1.18*10^5}$ and
$\huge2r=2\sqrt {\frac{0.12}{2\pi10^{-31.18*10^5}}} \approx 0.0254443$
The problem in what you did is that $r=d/2$ so $r^2=d^2/4$ as André ponted out.


How to create a bulk grant view form for the node access module?

Question: I'm completely new to Drupal and trying to learn it, but it's pretty hard.
The Node Access module is great and I want to make functionalty for it for bulk grant for several users. The idea is to have a form grid with several users in columns and nodes and grants in rows. It will look like that:

What I've come to so far:
<code>function table_test_form($form, &$form_state, $node) {
  $form_values = &$form_state['values'];
  if (!$form_values) {
    $form_values = array();
// Get node titles for the first column
        $results = db_query("SELECT DISTINCT title, node.nid
      FROM node_access
      LEFT JOIN node ON node_access.nid = node.nid
      WHERE node.nid < :nid
      ORDER BY node.nid", array(
        ':nid' => 5,
      ));
    foreach ($results as $noderow) {
      // make an array for every row
      $form_values['noderowid'][$noderow->nid] = array(
        'name' => $noderow->title,
      );
      $current_nid = $noderow->nid;
      //Get users and grants for the current node
      $results_grants = db_query("SELECT uid, name, grant_view
      FROM node_access
      LEFT JOIN users ON uid = gid
      WHERE nid = :nid AND uid < :uid
      ORDER BY uid ", array(
        ':nid' => $current_nid,
        ':uid' => 5,
      ));

//      Tried this but didn't work
//      if ($results_grants) {
//        while ($row = $results_grants->fetchAssoc()) {
//          $items_grants[] = array(
//                                    $row['name'] => $row['grant_view']
//          );
//         }
//      }
      $items_grants = $results_grants->fetchAllKeyed(1,2);
//    add the grants for every user to the node row
      $form_values['noderowid'][$noderow->nid][] = $items_grants;
    }
  }

  if (!isset($form_values['noderowid'])) {
    $form_values['noderowid'] = array();
  }

  $noderows = $form_values['noderowid'];

  $form['nid'] = array(
    '#type' => 'hidden',
    '#value' => $node->nid,
  );

  //  the table.
  if (is_array($noderows)) {
    $form['noderowid'] = array('#tree' => TRUE);

    foreach ($noderows as $key => $field) {
        if(!isset($form['noderowid'][$key])){
        $form['noderowid'][$key] = array();
    }
    $form['noderowid'][$key]['name'] = array(
        '#type' => 'hidden',
        '#value' => $field['name'],
      );
    $arr = $form['noderowid'][$key];
      //add checkboxes
            foreach ($arr as $nodetitle => $f){
               if($nodetitle!='name'){
                    $form['noderowid'][$key][$nodetitle] = array(
                      '#type' => 'checkbox',
                      '#default_value' => $f[$nodetitle],
                    );
               }
            }
    }
  }

  $form['submit'] = array(
    '#type' => 'submit',
    '#value' => t('Save Grants'),
  );
  return $form;
}

function theme_table_test_form($variables) {
  $output = '';
  $form = $variables['form'];
  $rows = array();
//get the names of the users and put them in the header
  $uid = 5;
  $sql = "SELECT name FROM users WHERE uid < :uid  ORDER BY uid ASC";
  $result = db_query($sql, array(':uid' => $uid));

  if ($result) {
    while ($row = $result->fetchAssoc()) {
      $items[] = array(
      'data' => t($row['name'])
      );
    }
  }
  // make table.
  unset($rows);
  $noderows = element_children($form['noderowid']);
  if (count($noderows) > 0) {
    $header = $items;
    $rows = array();

    foreach ($noderows as $key) {
      $row = array();

   // first column
      $row[] = $form['noderowid'][$key]['name']['#value'];
   // other columns
      foreach(array_slice($header, 1) as $usr){
              if(!isset($form['noderowid'][$key][$usr])){
                $form['noderowid'][$key][$usr] = array();
         }
          $row[] = drupal_render($form['noderowid'][$key][$usr]);
      }

      $rows[] = $row;
    }
    $output .= theme('table', array('header' => $header, 'rows' => $rows));
  }

  $output .= drupal_render_children($form);

  return $output;
}
</code>
It renders the users and nodes names, but not the checkboxes.
I guess it's a pretty bad code and there are too many queries to the database, but I want to just make it work for now.
Any suggestions?
Answer: Assuming this is for D7, this is a typical case to handle using the Rules module, combined with Views Bulk Operations, with zero custom coding. So if you are "completely new to Drupal and trying to learn it" (as in your question), you may want to try to learn to walk before trying to fly. Read on for further details.
Advice: the Node Access module is unsupported (repeat: unsupported). You really should consider using the Content Access module instead.
What you actually need to do is similar to what is shown in the video about Using VBO to load list of objects into Rules, starting from about 5:30, up to 11:30. Here is a summary of what is shown in this video (quote from the linked page):

You need a view that lists the entities you want to work on in Rules. Note that you don't need a display of the view – and it might even be better not to have one if you don't want the view to be displayed somewhere.
However, you need one bulk operations field. The type of bulk operations field determines what entity type will be sent to Rules. Note that you don't need any actions enabled for the field – it is enough that it is present.
You need a rule that, as an action, loads entities from a VBO. (This is available under the "Views bulk operations" group.) All view displays with at least one VBO field will be selectable.
The action provides a list of entities, that can be used just like other lists in Rules. Combine with loops, actions and Rules Scheduler to make awesomeness happen.

For your case, for the Rules Action you can use the integration of the Content Access module with the Rules module to create a Rules Action that updates the access of the node, similar to the Rules Action I included in my sample rule included in my answer to the question "How to revoke edit access to the node's author after publishing a node?".
Comment: thank you for your comprehensive answer. I will try the Content access/rules/VBO solution, although I preferred custom code as it seems to me simpler and more flexible. Still I want to understand what is the problem with the form I posted and why the checkboxes don't appear. I see you really know drupal, do you have an idea what I've done wrong?
Comment: @Toni It looks like you may have created an extra account.  If you wish to merge them, use the "Contact Us" link at the bottom of the page.


map.end() error "iterator not dereferenceable" c++

Question: //Works
cout << "map[0] value is " << doubleStatsMap.begin()->first<< endl;
//gives error
cout << "map[last value is " << doubleStatsMap.end()->first << endl;

Im simply trying to get the value of the last element of my map. It works correctly with
 "map.begin->first"
but is giving "map/set iterator not dereferencable" for the
 "map.end()->first".
 It cant be empty as the map has a beginning thus it has an end. Everything I've read says this should work. Any suggestions greatly appreciated!
Comment: *"Everything I've read says this should work."* You should either get better reading material or read the material more carefully.
Comment: Docs: http://en.cppreference.com/w/cpp/container/map/end
Comment: Iterators form a half-open range... Please read the documentation on iterators again.
Comment: You're thinking of `back()`, which returns a reference to the last element.
Comment: `end()` is an iterator to *beyond last element* - you cannot dereference it; it does not point to a valid element.
Answer: Trying to get anything from the end iterator causes undefined behavior.
To get the last item, you can use std::map::rbegin().
// cout << "map[last value is " << doubleStatsMap.end()->first << endl;
cout << "map[last value is " << doubleStatsMap.rbegin()->first << endl;

Comment: THANK YOU. I don't care if I get downvoted for asking this. I saw examples of end and its been driving me crazy.
Comment: so what is the point of having a "first" and "second" function attached to end(). further, what is the point of even having a [poorly named] end function anyway as maps cannot be referenced like arrays by index anyway?
Comment: @Mason, good questions. The answers to them are long and not appropriate in a SO comment.
Comment: @Mason `end` is not "poorly named" - read up on half-open ranges. Also, read this: https://stackoverflow.com/questions/13066884/what-is-half-open-range-and-off-the-end-value (especially the accepted answer).


How can I exclude configuration from import/export?

Question: I thought this was a simple use case of the new Configuration management system, but I haven't had any luck finding out how to solve this:
Problem
I want to store configuration in git and use drush to export configuration during development and then on deployment import the configuration. Pretty similar to doing a features revert in Drupal 7. My problem is that I don't want to store access codes in git for various integrations. This results in these configurations being deleted on
$ drush cim -y
Where I looked
I was hoping there would be a simple list/configuration for configs that should be excluded on import/export. It seems there was at some point but it must have been removed again, since it's available at present Drupal 8 release.
I looked into how config changes are being made comparing the active and sync storange to see if there as a place where I could remove changes, this didn't seem to be the case.
I looked at how drush does the configuration import as it has some configuration excludes of it's own, but it didn't look like this was extensible.
I looked at ConfigEvents, but these all seem to happen after an import, so doesn't look like this can be used.
Am I missing something, or is it not possible to simply exclude configs from import/export?
Answer: You can use the "Config Ignore" module: https://www.drupal.org/project/config_ignore

Ever experienced that your site's configuration was overridden by the configuration on the filesystem, when doing a drush cim?
Not anymore!
This module is a tool to let you keep the configuration you want, in place.
Comment: config_ignore is probably the most stable and simple solution currently available, and this solution probably best addresses the OP's desire to have "a simple list/configuration for configs that should be excluded on import/export"
Comment: i totally recommend it
Answer: You can use a combination of config_ignore and config_split for this.
Config ignore allows you to ignore a subset of configuration entities during import (also prevents deletes since the 2.x release). Unfortunately it doesn't prevent excluding config during export.
To exclude configuration entities during export, you can use config_split, create a new config_split entity, and leave the folder empty. This prevents the configuration from being exported to the file system; instead, it exports it in the database.
I wrote Exclude config from configuration management in Drupal 8 about this topic.
Answer: It's not possible to have excludes specifically, but there is something.
Just like $conf in Drupal 7, there's $config in settings.php that you can use to override anything in configuration locally. The format is $config['name.of.config']['nested']['key'].
Note that whatever is stored in configuration is still in git, so you either need to keep none or your testing access codes in git. Also, the UI will show what's actually stored in config and will currently not provide any indication that it is overridden. There are pending issues to improve that.
I understand that this has limitations but right now, it's not possible to keep something out of the exported configuration. As far as I know.
Comment: So basically, you have to set / update configuration in your settings.php file instead of using the interface.
Comment: Yes, this is nothing new. Nothing has changed in the workflow in this context in d8.
Answer: Since Drupal 8.8 modules can now be excluded from configuration synchronization by adding $settings['config_exclude_modules']to your settings.php file.

$settings['config_exclude_modules'] = ['devel', 'stage_file_proxy'];

See the change record Modules can be excluded from the configuration synchronization.
Comment: This seems a bit too aggressive. It removes a bunch of config files instead of just one.
Answer: In order to split configurations you can use https://www.drupal.org/project/config_split.

Enter config_split that provides a Drupal console command for
  importing and exporting filtered configuration. Drush integration is
  likely to follow soon (after all the filter is inspired by the
  --skip-modules filter of drush).

You can split exports into different dirs which you can then git ignore.
There was a very nice presentation in drupal con dublin 2016 by the people in charge of the CMI initiative which I urge you to check out no matter what.
Answer: Drupal 8
Please note that this module is deprecated and will not receive a Drupal 9 release.

You can use Drush CMI tools. Then you have a drush cexy command and a config-ignore.yml to provide a list you can adjust. We needed it once to have block instances excluded while at the same time the block bases needed to get processed.

We want to export all configuration, but we want to exclude certain patterns.
This is where the --ignore-list option of drush cexy comes in.
In our project we have a ./drush folder, so we stick a file in their
called config-ignore.yml with contents as follows.
ignore:
  - field.field.contact_message.*
  - field.storage.contact_message.*
  - contact.form.*
  - core.entity_form_display.contact_message*
  - core.entity_form_display.contact_form*
  - core.entity_view_display.contact_message*
  - core.entity_view_display.contact_form*
  - system.site
  - workbench_email.workbench_email_template.*

So now we run drush cexy like so
drush cexy --destination=/path/to/config-export --ignore-list=/path/to/drush/config-ignore.yml

So what this does is export the active configuration, and then apply
the ignore list to remove unwanted configuration.
So now when you run git status you should only see changes you want to
commit.

Source: https://www.previousnext.com.au/blog/introducing-drush-cmi-tools

Installation
cd ~/.drush
wget https://raw.githubusercontent.com/previousnext/drush_cmi_tools/8.x-1.x/drush_cmi_tools.drush.inc
drush cc drush

Source: https://github.com/previousnext/drush_cmi_tools
Answer: I just tested @berdir in #1 answer and it run perfectly. Only I add a little note: you have to put whole config in this var, complete. Wihtout it, $config var not run correctly.
Something like this:
 $config['language.negotiation'] = array(
  'session' => array(
    'parameter' => 'language',
  ),
  'url' => array(
    'source' => 'domain',
    'prefixes' => array(
      'es' => '',
      'pt-br' => '',
    ),
    'domains' => array(
      'es' => 'YourLocalDomain',
      'pt-br' => 'Anotherlocaldomain',
    ),
  ),
  'selected_langcode' => 'site_default',
  'langcode' => 'es',
);

Documentation:
https://www.drupal.org/node/1928898
Note from above documentation:
"Note that values overrided via $config within settings.php will not be viewable from the Drupal administration interface."
Comment: You should not have to do this. It should merge in any values you define, but maybe it doesn't work as expected with some structures.
Comment: I can confirm that you can do something like `$config['module.settings']['some']['value'] = 'foo';`
Comment: Oh, I couldn't make to run it correctly. I will try to debug it again and if not run correctly maybe I can open a bug in d.org.

Thanks!
Answer: Using config split (recommanded)
The config split module was made specifically for this need.
Config split is integrated with drush.
Using Drush only
Drush it's also supposed to be able to do that by using the --skip-modules flag.
You can add the following lines in a drupal/drushrc.php in the webroot of your project to do it automatically.
$command_specific['config-export']['skip-modules'] = array('devel');
$command_specific['config-import']['skip-modules'] = array('devel');

See http://www.drush.org/en/master/config-exporting/#ignoring-development-modules
Sadly there is a bug with this functionality: https://github.com/drush-ops/drush/issues/1820. So for the moment you also need to add those config files in your .gitignore so exported config files don't get committed. Their is an ongoing to perhaps abandon this (buggy) functionality from drush in favor of config split.
Comment: Ignoring the config files in git doesn't exactly work. Configuration not exported will get deleted. For example, custom panels pages will be deleted upon importing config.
Comment: @user157272 even if you import using drush with ['config-import']['skip-modules']?
Comment: drushrc.php can even be placed outside webroot. For example one level higher, which is useful when you work with Drupal in a composer setup: https://github.com/drupal-composer/drupal-project/.
Comment: This didn't work for me. I added modules, yet they were still added to the enable/disable modules config.
Comment: I updated the answer with the current best practice (config split)


Jenkins disable build now

Question: I need to disable "Build now" option on Jenkins, I have a job building a branch and deploy SNAPSHOTs to a server. I need a production Job, that generates only stable versions, using maven release, but for this job I think the best approach is disable "Build Now" option and use only Maven Release. So Can I do this?
Answer: Use Authorization strategy plugin to disallow running the job. If configured right - no one will see "Build Now" button. Another options is Job Restrictions plugin, but it looks a bit clumsy or simply not that good for this particular task.

Jenkins disable build now