With two related variables, eg, religion and religiosity, how do I transform them into one variable for regression?

Question: Say I have the nominal variables of religion (0=Athiest; 1=Christian; 2=Jewish; 3=Muslim; 4=Other)
And then a scale variable of religiosity from 1-10
If I want just one scale variable, so that I can plug the value into a regression modeling something DV that is either on a scale of 1-10, or is binary, is the solution to this to create a dummy variable for religion and then compute all new variables of
'Atheist-ness' = Religion_Athiest*Religiosity
'Christian-ness' = Religion_Christian*Religiosity
'Jewish-ness' = Religion_Jewish*Religiosity
'Muslim-ness' = Religion_Muslim*Religiosity
'OtherReligious-ness'  = Religion_Other*Religiosity
?
Because then my regression model will look like:
Constant + (factor* value) + (factor* value) + (factor* value) + (Atheist-ness* value) + (Christian-ness* value) + (Jewish-ness* value) + (Muslim-ness* value) + (OtherReligious-ness* value)
All of these religions are discrete variables, so will it not matter, because any new individual plugged in will have 0 for all values related to religion, other than the one that they apply to?
My first question is: Is there a more correct and efficient way to do this? I am using SPSS. This seems like the intuitive way to do it but I am not sure if there is another way I am missing.
My second question is: Can someone explain to me in layman's terms what I've done and more importantly why I've done it?
Comment: @DaveHarris you could explain this in a nicer tone and actually provide some constructive feedback. As far as categorical variables go, it is very common to represent them as zero-or-one dummy codes, so you very much can multiply them together. If what they are trying to do doesn't make sense, explain in an answer and in a way that makes people feel as if they want to actually return to this forum for help in the future.
Comment: @DaveHarris The OP specifically mentioned dummy-coding the categorical variable, which would not treat Muslim as literally three times as much as Christianity. Please follow the guidelines for not being rude and moreover very much unhelpful to the OP.
Comment: @DaveHarris I am with MarkWhite: your tone is reading hostile. Your comment "I am trying to be helpful. I am not trying to be rude. I am being matter of fact. There is no emotional content involved on my side. It is how you are reading it." suggests you might benefit from [today's xkcd](https://xkcd.com/1984/). You tone is coming through in your imputations of meanings the OP has not written.
Answer: First, if you could add a few things in an edit to your question, I think we could help more:

What is the goal of your analysis?
What are you trying to predict from this model?
How was "religiosity" measured?

I will assume that "religiosity" means how dedicated they are to whatever their religious belief is: a 5 means they are moderately committed, a 10 means they are very committed, etc.
Since someone can only select one religion, what dummy-coding the religious affiliation and then multiplying each by religiosity does is make everyone have a score of zero for everything and then a score that is equal to their religiosity score for whatever religion they subscribe to (since their score for that dummy-coded variable is 1).
So, if I was an atheist who selected 8 on "religiosity" (here, I'm assuming it is how committed one is to their religion), my dummy-coded variable for atheism would be "1", so the scores for me resulting from multiplication would be:

Athiestness: 8
Christianness:0
Jewishness: 0
Muslimness: 0
Otherness: 0

On their own, these scores might not be helpful. So in the model you specified with only these multiplicative terms present, I would have a hard time finding that model useful or very interpretable.
However, if you include the dummy codes, the religiosity score, and all of the multiplicative effects together in a model, you have an interaction you are testing. What this model could do is help answer the question: "Does religiosity relate to my DV for different religious groups?"
For example, let's say you had the hypothesis that religiosity is related to capitalistic beliefs, but only for Christians. You might make this hypothesis due to research on Protestant work ethic, for example.
A model that looks like:
capitalistic_beliefs ~ all_religion_dummies + religiosity + all_religion_dummies * religiosity

Could answer that question. (Where all_religion_dummies refers to all of the 0-or-1 variables for if they are Atheist, Christian, Jewish, etc.)
If you were to get a significant interaction, you could them probe it with simple slopes analyses to see if the slope for religiousness is significant only for Christians.
Again, it is difficult to tell if what you are doing makes sense or is helpful to your research goal without knowing the dependent variable. I'd be happy to take a look at the question again if you edit it with more detail or need to know anything further.

To demonstrate, I simulated data where there was a relationship between religiosity and capitalistic beliefs for Christians, but not for other religions. I simulated these data in R using the following code, then ran a model and tested for the interaction:
set.seed(1839) # set seed for replicability
n <- 5000 # set number of participants
religiosity <- rnorm(n) # generate religiosity as normally distributed variable
# make it so that capitalistic beliefs (cb) is only related to religiosity
# for christians
cb_other <- rnorm(1000) # just predicted by error
cb_muslim <- rnorm(1000) # just predicted by error
cb_jewish <- rnorm(1000) # just predicted by error
cb_christian <- religiosity[3001:4000] + rnorm(1000) # by religiosity and error
cb_atheist <- rnorm(1000) # just predicted by error
# make into data set
dat <- data.frame(
  religiosity = religiosity,
  capitalistic_beliefs = c(
    cb_other, cb_muslim, cb_jewish, cb_christian, cb_atheist
  ),
  religious_affiliation = factor(c(
    rep("other", 1000), rep("muslim", 1000), rep("jewish", 1000),
    rep("christian", 1000), rep("atheist", 1000)
  ))
)

# run regression with interactions you speak of—the interaction terms are where
# religiosity is multipled by their affiliation
# I do not specify dummy-codes explicitly, because R does it on its own
# without me having to do it! one of the reasons I would suggest using R
# instead of SPSS!
# compare model without interactions (just plus sign) to those with interactions
# (including multiplication)
model_without_ints <- lm(capitalistic_beliefs ~ religiosity + religious_affiliation, dat)
model_with_ints <- lm(capitalistic_beliefs ~ religiosity + religious_affiliation +
                        religiosity * religious_affiliation, dat)
anova(model_without_ints, model_with_ints) # interaction is significant!

That last call gives us this result, showing the interaction was significant:
Model 1: capitalistic_beliefs ~ religiosity + religious_affiliation
Model 2: capitalistic_beliefs ~ religiosity + religious_affiliation +
    religiosity * religious_affiliation
  Res.Df    RSS Df Sum of Sq      F    Pr(>F)
1   4994 5711.1
2   4990 4949.2  4    761.93 192.05 < 2.2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Then we can graph it!
library(ggplot2)
ggplot(dat, aes(x = religiosity, y = capitalistic_beliefs, color = religious_affiliation)) +
  geom_smooth(method = "lm", se = FALSE)

Which looks like:

In short: Those scores of dummy-codes multiplied by religiosity don't mean much alone. But when you include them as interaction terms in a regression context, they can test interesting moderation hypotheses.
Note: I know the R code above is somewhat inefficient, but I wanted to make it so that one could follow along if they were only familiar with SPSS, per the OP mentioned in their post.
Comment: @SmallChess I updated my answer for clarity. It is 8 * 1, because since I identify as an atheist, my dummy-coded variable would be 1 for atheist and 0 for everything else, leading to those scores.
Comment: I read it several times but not really sure. Why 8 times 0 is 8?? The answer is unclear to me.
Comment: Thank you for your answer, Mark, this made a lot of sense to me up until about halfway through where my lack of technical language knowledge started making it difficult, but all of the first part made sense. My data is literally just a list of characteristics (height, eye colour, religion, religiosity, political affiliation, political interest, level of education etc) about a group of individuals, there are some DVs such as average perceived attractiveness, career satisfaction, level of charity giving, fitness etc.
Comment: It's just a dataset with no real research question in mind, but I'm using it as practice data and religion and religiousness seemed like a good opportunity to look at a categorical and scale measure that are related, and how they can help model an equation. So, can I model how generous someone is likely to be? Or how attractive they report themselves to be found by people they date?
Comment: @Sophie I updated my answer including a sample data set, analyses, and visualization that I just simulated. You could look at, for instance, of religiosity is related to generosity *differently for people across different religious groups.* I would suggest looking into the literature on "moderation" and "moderators" to get a feel for what those interaction terms could tell you.
Comment: For the sake of the example, religiousity could be measured using a validated measure of religious orientation, standardised to a scale of 1-10.
Comment: Thanks Mark, this is beginning to make sense to me but I will have to read up on interactions more to try and understand in full what you mean and then try and update what I'm still unclear about.
Comment: @SmallChess I also now included a simulated dataset that should get this point across. With that `dat` object in your R session, you can run `model.matrix(~ -1 + religious_affiliation, dat) * dat$religiosity` to get those scores I lay out in my post.
Comment: @Sophie The easiest place to begin is when there are just two categories. That way there is only one dummy-code variable of 0 or 1. What you are doing is what is sometimes called a "multicategorical" interaction, or at least that is what Hayes—who wrote the famous PROCESS macro for SPSS—calls them, which are much harder to understand, in my opinion.
Answer: You cannot transform them together.  There is an illusion that they are similar because they both involve religion.  They are very different.  Consider the following set of categorical variables, $\{cow, apple, house\}.$  We will denote them $\{1,2,3\}$.  Basically, if there is a relationship, it is hidden, although one might exist.  They may all be on the same parcel of land.  Now consider a scaling variable, their mass.  Let us assume we have various weights of cows, apples, and houses and we rank each 1,2,3$\dots$10.  Now assume that you pick up one cow of rank 3 mass and one house of rank 1 mass.  $1\times{3}=3\times{1}$ but you cannot really say what that three means.  These are separate dimensions.
Although it is true that you can sometimes multiply dummy variables, this is not one of those cases.  Multiplication implies that there is an interaction between the dimensions.
There are also serious problems with your categories.  They have to be mutually exclusive.  Your categories are not.  Buddhism doesn't have a god, although Buddhists can optionally have gods.  So a Buddhist could be an atheist or an other.  Some relgious groups, such as Mormon's are not considered Christian by other religious groups.  If this is self-report then you can get a misclassification.  Mormons and members of the Oriental Orthodox Church do not accept the council of Nicaea's statement of faith, which to most Christian groups implies they cannot be Christian.  Finally, you can be Jewish and Christian at the same time or Jewish and Muslim or Jewish and atheist and Jewish and other.
Categorical variables must be mutually exclusive.  People who use religion in research usually go around the categorical labels if there is some risk of misclassification unless self-classification is of interest in and of itself.  For example, Barrack Obama self-classifies as black but there was a significant issue at the beginning of his campaign as to whether others considered him black or white.  If you are studying classification, then the labels matter, but if the correctness of the label matters then you ask questions that get you to the label.
To give an example, the Orthodox do not accept the filioque in the Nicene Creed.  If you were trying to identify Christians you could not ask if they accept the creed as the Orthodox read it without the filioque while Catholics and Protestants read it with the filioque.  Simply asking if you believe the Nicene creed would lump the Orthodox in with Protestants and Catholics, which the Orthodox would tell you is an error as a strictly partisan Orthodox would tell you there are Christians and heretics and the Orthodox are the Christians.
You need to figure out what your actual dependent variable is and figure out what your research question is and your available variables.  After that, you can work out your form and collect your data.
There is also a wording issue.  Religiosity does not apply to atheists as atheists have no belief in the divine, by definition.  They are a non-group.  To use a famous phrase, atheists are a religion as non-stamp collecting is a hobby.  You would be better off including atheists as "others" unless religiosity matters to you.  If it does, then atheists will be a confounding variable and you need a way to separate them out.  You may seek some set of behaviors that parallel religiosity such as passion for things people believe in.
If you have two dimensions then you are stuck with two dimensions.
EDIT
So let's assume you can construct mutually exclusive groups.  Then you need to be very careful on what your religiosity and your religiosity times group dummy would mean.
If you are assuming that there is a cluster of behaviors called religiosity that is common to all humans, then you are going to have be very careful by adding an interaction term.
Consider a different variable, such as walking speed.  A religious group shouldn't impact walking speed unless one of the "other" groups happens to engage in some behavior that is known to impact walking speed.  Imagine a religion built around track and field events, then you can see it could be the case.
Religiosity is going to be very difficult to measure because if you include groups like atheists, then if one of your scale questions is "prays every day," then you are going to get zeros in that question.
That is a valid interaction effect and the multiplication would mean something.  But it begs the question of "did you not know that before you asked the research question?"
The multiplication component will imply that there is a unique relationship between the group and religiosity that is not common to other humans.  There is a reason I find this concerning.
The F-test checks to see if all slopes are equal to zero.  Normally this isn't an issue.  Now let us imagine one of your variables was the velocity of a falling bowling ball.  We know due to gravity that the object will have a positive velocity.  That one slope cannot have a zero, so the F-test will automatically show up as "significant."
By including atheists in a religiosity question, you may get a case where no matter what you ask, all F-tests will show up as "significant."
It is a psychometrically difficult question because it may be that there are atheists who would score high on religiosity if religiosity were instead a measure of passion or devotion to something believed in or amount of time spent working on it.
I would look at some scale such as the Work Environment Scale which is an unbelievably careful instrument to measure the work environment.  You are also going to have to be very careful with language use.  I can remember a young woman who converted from being a Baptist to being a Catholic.  Her sole reason for converting, and I did get to know her pretty well, was that Catholics looked up at the cross when they prayed and Baptists looked down.  She felt one might be right while the other one wrong.  This led her to investigate and she chose Catholicism.
This very small difference altered her life and her definitions of right and wrong.  You have to be super careful with language and religion.  Very small things can have deep or profound meanings to some that you do not intend.
If one is realistic rather than theoretical regarding a psychometric instrument, an interaction effect would likely really imply that members of a group are understanding the wording of your questions differently from the people in other groups, rather than that there is an actual behavioral, emotional or deep cognitive difference.
Comment: The data is example data and I am just trying to get to grips with how to analyse data before I do any data collection, so for the purposes of the example, religion is mutually exclusive for simplicity's sake. Instead of asking these two questions I could have asked 'how often do you pray/go to a religious service/engage in religious community etc' for each of the religion groups and only allowed an answer to one with all other answers defaulting to 0, which would have given me a measure of 'how Muslim-ish' a Muslim which is directly comparable to 'how Jewish-ish a Jew is'
Comment: but as I only have religion and religiousness as variables, I'm trying to see if I can get scale data for Jewish-ness, Muslim-ness etc in a model that predicts a DV by making a new binary IV for each religion and combining this with religiousity to end up with measures of how Muslim, Christian etc people are.
Comment: @Sophie This is an interesting measurement conundrum because probably someone's religion is mutually exclusive for most people. (If you want strictly religious beliefs, I would avoid asking a question that could get people answering for secular/cultural/ethnic reasons). I do not think you necessarily need a continuous variable for "how Muslim a Muslim is." You could just ask them their religion. All of this depends on your research question, though, as I discussed in my answer.
Comment: I think religiosity is an important measure, though, because within my data, I have a Christian who is very devout, and a Christian who believes in Jesus and goes to Church once a year, for example. I might have an Atheist who doesn't really care much about their beliefs, and one likes to engage in this belief a lot. At the end of the day I am not too worried about the measures, more about how to deal with related variables.
Comment: @Sophie can you create multiple scales of "religiosity?"  For example, can you separate out belief from behavior?  Am I allowed to attend church every day but have no belief?  Can I have fanatical belief but don't attend services?


Use header variable in NGINX to forward traffic

Question: I am using NGINX (nginx Docker image) as a reverse proxy and want to use the $host variable in stream/server context to forward the traffic to specific host which is defined in the Host header parameter of the incoming request. The configuration is the following:
events {
}

stream {
  log_format log_stream '$remote_addr [$time_local] $protocol'
  '$status $bytes_sent $bytes_received $session_time';

  access_log /var/log/nginx/access.log log_stream;
  error_log  /var/log/nginx/error.log;

  server {
    resolver 220.127.116.11 ipv6=off;
    listen       127.0.0.1:18443;
    proxy_pass   $host:443;
  }
}

There is an error during starting:
[emerg] 1#1: unknown "host" variable

According to the nginx documentation this variable should be populated. Do you know how can i use it inside the server directive to forward the traffic? Can i use other header parameter with specified hostname/address to forward the traffic?
Comment: What host name? There is no such variable in the stream context. You are just proxying a raw TCP stream. What host header?
Comment: What about this [$hostname](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#var_hostname) variable from the ngx_stream_core_module
Comment: That is the hostname of the system on which nginx is running (or in the case of Docker, the random container name).
Answer: Your intention seems to be to pass through TLS connection via nginx stream module. If you want to target different destinations depending on the SNI field of TLS header, then you need to use the following configuration:
map $ssl_preread_server_name $destination {
    host1.example.com backend1;
    host2.example.com backend2;
    default backend3;
}

stream {
    upstream backend1 {
        server 192.168.100.1:443;
    }

    upstream backend2 {
        server 192.168.100.2:443;
    }

    upstream backend3 {
        server 192.168.100.3:443;
    }

    server {
        listen 127.0.0.1:443;
        proxy_pass $destination;
    }
}


Probability of what exactly was calculated by Chevalier de Méré?

Question: Chevalier de Méré knew that probability of rolling six with a six-sided die is $\frac 16$. He reasoned that his chance to roll six with 4 tries is $\frac 46$, which is not the case. It is $1 - (\frac 56)^4$.
Which event's probability was calculated then? That is, which event has probability of $\frac 46$ in an experiment of throwing a six-sided die four times?
Comment: He did not think that the probability of rolling at least one $6$ was $4/6$.  He knew it was a slightly better than $50$-$50$ proposition, and asked about a double $6$ in $24$ rolls of two dice. He wondered whether the chances were the same.  (The mean number of $6$ in $4$ rolls is $4/6$; the mean number of double $6$ in $24$ is $24/36$, which is the same as $4/6$. There was some confusion at the time about the relationship between probability and expectation.)
Answer: There are several events that have a probability of $\frac46$ in that experiment. For example:

Rolling a number smaller or equal to $4$ on the first throw
Rolling a number larger or equal to $3$ on the second throw
Rolling a number between $2$ and $5$ (inclusive) on the third throw

And on and on and on.
Comment: @WinVineeth No, he isn't. You are *assuming* he is. And sorry, but in an experiment where you roll a die $4$ times, the event "I rolled a number smaller than $4$ on the first throw" **is** an event.
Comment: He is asking an event which is relevant to the previous experiment..
Comment: I know that you have given several events... If he was not asking for a relevant event, why will he quote the first sentence then?
Comment: @5xum great, thanks! Can you come up with an example of an event that has probability of 4/6 and includes several throws (all four throws in the best case)?
Comment: @Baldee: An event in which the 1st throw is a number between $1-4$, the 2nd throw is a number between $1-6$, the 3rd throw is a number between $1-6$, and the 4th throw is a number between $1-6$ (which is exactly the description given in the first bullet above).


N1QL and Node on Couchbase

Question: I am trying to use N1QL with Node.js SDK of Couchbase. Here is my code:
<code> var config = require('../../config/init');
        var N1qlQuery = require('couchbase').N1qlQuery;
        var query = N1qlQuery.fromString("SELECT * FROM users");
        var db = couchbaseService();
        console.log('start');
        db.query(query, function(err, result) {
            if (err) {
                console.log('inside err');
                console.log(err);
                return;
            }
            console.log('No error in NIQL');
            console.log(result.object);
        });
</code>
Here is the error:
<code> TypeError: Cannot read property 'length' of undefined
    at Bucket._query (/vagrant_data/xxxxx/node_modules/couchbase/lib/bucket.js:331:52)
    at Bucket.query (/vagrant_data/xxxxx/node_modules/couchbase/lib/bucket.js:607:17)
    at module.exports.show (/vagrant_data/xxxxx/api/controllers/udk/UserController.js:33:12)
    at routeTargetFnWrapper (/vagrant_data/xxxxx/node_modules/sails/lib/router/bind.js:178:5)
    at callbacks (/vagrant_data/xxxx/node_modules/sails/node_modules/express/lib/router/index.js:164:37)
    at param (/vagrant_data/xxxxx/node_modules/sails/node_modules/express/lib/router/index.js:138:11)
    at param (/vagrant_data/dataminer/node_modules/sails/node_modules/express/lib/router/index.js:135:11)
    at pass (/vagrant_data/xxxxxx/node_modules/sails/node_modules/express/lib/router/index.js:145:5)
    at nextRoute (/vagrant_data/xxxxx/node_modules/sails/node_modules/express/lib/router/index.js:100:7)
    at callbacks (/vagrant_data/xxxxx/node_modules/sails/node_modules/express/lib/router/index.js:167:11) [TypeError: Cannot read property 'length' of undefined]
</code>
Please note,problem is not about connecting to bucker or server. I used .get() and .insert() and they are working fine.something missing in N1QL library. Anybody can Help?
Comment: Have you tried using latest Developer Preview3 http://www.couchbase.com/communities/n1ql
Answer: You have to enable the n1ql on the given bucket:
<code>bucket.enableN1ql(serverUrl+":8093")
</code>
More can be found here:
http://docs.couchbase.com/sdk-api/couchbase-node-client-2.0.0/Bucket.html#enableN1ql

N1QL and Node on Couchbase


Loss function and Regression in Real World Data

Question: For Regression problem and loss function. It has a y(x) and t(target) variables.
$$L(t_y(x)) = (y(x)-t)^2$$
in real world. what is the target variable represent?
If you collect the data. How do you expect some target value?
I have a trouble to under stand what is target variable in real world.( How you expect some target value from unknown dataset?) Maybe i misunderstand about something in regression problem. Can anyone give some examples with find the expectation loss?
Answer: You math notation is less clear and can be improved. For example, what is the difference between $t_y(x)$ and $y(x)$? $y$ is a function or response variable?
I will start to introduce the notation in classical way from beginning.
In regression setting, we are predicting a continuous variable. For example, we have a person's weight, and we want to predict a person's height based on he/she's weight. Then the height is the target variable.
Suppose you collected 100 persons' data, and the data is in pairs. For each person, you will have a height and weight value. I am making up first 3 data points here. Let's assume weight is in kg and height is in cm.
1st data point: (60kg, 170cm)
2nd data point: (65kg, 175cm)
3rd data point: (80kg, 180cm)
...
100th data point: (70kg, 181cm)

Now, let's use $x$ to represent your "feature" or "independent variable", and use $y$ to represent "target" or "dependent variable". In addition, we use subscript $i$ to represent $i^{th}$ data point. In such setting $x_1=60$, $y_1=170$, $\cdots$.
Our goal is to predict height "accurately", but how to measure "accuracy"? We can use squared loss, where we use $\hat y_i$ to represent predicted value on $i^{th}$ data. Suppose we have 100 data points in total, we want minimize this Loss for all data. So we are summarizing from $1$ to $100$.
$$\sum_{i=1}^{100}(y_i-\hat y_i)^2$$
Now, one step further, where does $\hat y_i$ comes from? It comes from a function of your "independent variable" $x_i$. So, $\hat y=f(x_i)$, then the equation becomes
$$\sum_{i=1}^{100}(y_i-f(x_i))^2$$
Which is squared loss on your whole data set for regression setting.


Allow user to use entity reference to nodes they own?

Question: I have a Doodad content type that has a couple of Entity Reference fields, for example, parentDoodad. Those fields point to other Doodad nodes. When I'm logged in as an Admin, I'm able to use the auto-complete field to set any doodad in the parentDoodad field.
However, if I switch to a different user and create Doodad1 and Doodad 2 (so that I have multiple nodes owned by this user), when I try to use the entity reference field, it won't show me any of my other Doodads.
If I give my non-admin-user the "Bypass content access control" permission, then I'm able to see and choose those other Doodads in the parentDoodad field.
I want users to be able to choose any of their own Doodad nodes. Is it just a matter of configuring certain permissions? How do I let UserA be able to choose from UserA's nodes in that entity reference field?
Answer: You can do it this way.

Create a view to list the nodes of particular type.
1a. Add a contextual filter of Content: Author Uid and provide default values as USER ID from logged in user
1b. Add a Entity Reference display to the view.
Now go to the Entity Reference field you have created in your content type and edit it.
2a. Under Entity selection select the Mode to Views: Filter by an entity reference view then you will get the view you have created above and select it.
2b. Save the changes.

This way the user will see only the node they created in the reference field. Read More
There is a second way also, you can implement hook_query_entityreference_alter() and alter the query. Read More
Answer: Wolverine's solution will break if an admin is editing the node. The field will then show only nodes the current user (admin) has created.
A better solution is to use "User ID from URL" and check "Also look for a node and use the node author"; that way when content managers are reviewing nodes, the entity reference field will reflect the original author's nodes, rather than the managers' own.
Answer: if I select {"User ID from URL" and "Also look for a node and use the node author"}: This solution will break if an user is adding a new node. The field will not show any nodes


Models with reserved keywords

Question: I want to create a model called 'File', but it is a reserved model name is rails. I can't think of anything else sane to call the model, so I was wondering if there  is a standard way of dealing with this issue, for example adding a prefix or suffix (_File, FileItem, etc)?
Answer: This problem is addressed with modules:

Modules are a way of grouping together methods, classes, and
  constants. Modules give you two major benefits:

Modules provide a namespace and prevent name clashes.
Modules implement the mixin facility.

[...]
Modules define a namespace, a sandbox in which your methods and
  constants can play without having to worry about being stepped on by
  other methods and constants.

In your case:
module MyRailsApp
  class File
  ...
  end
end

whereby your File class is used as MyRailsApp::File. This is the typical solution in Ruby, in Ruby on Rails this might be handled differently, please see the following references for an in depth discussion:

Handling namespace models (classes) in namespace
ActiveRecord: Can haz namespaces?
Namespaced models and controllers
Namespaced models
A simple alternative to namespaced models
Comment: How do you put a model in a module in rails?
Comment: @CameronMartin Not different than you put a class in a module.
Comment: You can create models in modules by doing `rails generate model modulename/modelname` right? Is it better to put all your models in a module, or choose a non-conflicting name?
Comment: @CameronMartin That is a good question, after all this topic has its own discussion ongoing. I've added additional references which discuss this problem, since it seems there is no clear answer.

Models with reserved keywords


Using hook_form_alter to enter placeholder in mailchimp signup fom

Question: For the signup form I'm using the Mailchimp Signup module. The idea is to hide the label, which I did using the unset(); function, and put an a placeholder in the input field where people enter their emailaddress.
Here's the code:
function bw_custom_form_alter(&$form, &$form_state, $form_id){

  if ($form_id == 'mailchimp_signup_subscribe_block_mysite_group_form') {
    //dpm($form);
    unset( $form['mergevars']['EMAIL']['#title']);
    $form['mergevars']['EMAIL']['#placeholder'] = t('Your emailaddress');
    dpm($form['mergevars']['EMAIL']['#placeholder']);
    }
}

dpm($form['mergevars']['EMAIL']['#placeholder']); returns the string Your emailaddress nicely in the green drupal message frame. Yet, the actual placeholder isn't there in my input element. Looking at the attributes of the particular inputfield in question, the attribute isn't added either:
<input id="edit-mergevars-email" class="form-text required" type="text" maxlength="128" size="25" value="" name="mergevars[EMAIL]">

Answer: You have to use this:
$form['mergevars']['EMAIL']['#attributes'] = array(
 'placeholder'=> t('Your placeholder'),
);

Answer: If you want all field titles as placeholders:
$key_blacklist = array('#prefix','#suffix','#tree');

foreach($form['mergevars'] as $key => $var) {
  if ( ! in_array($key, $key_blacklist)) {
    $form['mergevars'][$key]['#attributes'] = array( 'placeholder'=> $form['mergevars'][$key]['#title']);
  }
}


Why didn't Oklahoma attempt an extra point after their fumble recovery touchdown at the end of regulation time in 12/7/13 NCAA game vs OK ST?

Question: Why didn't Oklahoma attempt an extra point after their fumble recovery touchdown on the last play of regulation time in its 12/7/13 NCAA game vs Oklahoma State? Is there a NCAA rule answering this question?
Answer: NCAA Football Rule 8-2-3-a, covering the "try" (either extra point or two-point conversion), states:

a. The ball shall be put in play by the team that scored a six-point touchdown.
  If a touchdown is scored during a down in which time in the fourth period
  expires, the try shall not be attempted unless the point(s) would affect the
  outcome of the game.

If the score had been 34-32 or 33-32 in favor of Oklahoma State or tied 33-33 after the touchdown, then Oklahoma would have attempted the try. But since Oklahoma already had the lead, it was not necessary.
My interpretation of "unless the point(s) would affect the outcome of the game" leads me to believe that the try is only attempted if it would help the scoring team; i.e. a team that scores to take a one-point lead as time expires would not have to attempt the try, even though the defense could in theory score two points on a turnover or blocked kick.
Note that the NFL's rules differ on this point; in the NFL, the try is always attempted except after touchdowns in sudden-death overtime.


JFrame large grid map

Question: I'm trying to make a large grid map of 160x120 JButtons with JFrame but it's too much to fit on the window.
How can I overcome this?
public class DisplayTable extends JFrame {
    public static void main() {
        JFrame frame = new JFrame("puzzle layout");
        //frame.setResizable(true);
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.setSize(3200, 800);

        JPanel panel = new JPanel();
        panel.setComponentOrientation(ComponentOrientation.LEFT_TO_RIGHT);
        panel.setBorder(new EmptyBorder(0,0,0,0));
        panel.setLayout(new GridLayout(12,16));

        for(int i=0; i<120; i++) {
            for(int j=0; j<160; j++) {
                 JButton temp = new JButton("1");
                 panel.add(temp);
            }
        }
        frame.add(panel);
        frame.setVisible(true);
    }
}

Answer: Use a JScrollPane to wrap the panel ... also, you should use pack instead of setSize, but only after you've added all your components to the screen
See How to use scroll panes for more details

JFrame large grid map


django named url not picking variables

Question: I have the following url pattern -
url(r'^detail/(?P<var>[\w]+)/(?P<var_slug>[\w-])/$', 'player_detail', name='player_detail'),

In my view, I have the following -
model_dict = {"player":PlayerProfile, "event":PlayerEvent, "need":PlayerNeed}
def player_list(request, var=None, var_slug=None):
    '''
    displays the list of vars
    '''
    objs = model_dict.get(var).objects.filter(slug=var_slug).order_by('-creation_time')[:20]
    template_name = "list_"+str(var)+"s.html"
    return render(request, template_name, {"objs":objs})

In my templates I finally do the following -
details of Player

The error I get is following -
Reverse for 'player_detail' with arguments '()' and keyword arguments '{u'var': u'baseball', u'slug': u'obj.slug'}' not found.

What am I missing?
also, is it a good way to pick models dynamically depending on the variable in the parameter, and generating a template name on the fly?
Answer: slug='obj.slug' should be slug=obj.slug
Your regex doesn't match a .
Also you probably want[\w-]+ just like your other regex.. one or more [\w or -]

django named url not picking variables


Atiyah-Macdonald Proposition 2.4

Question:
I don't understand what the above equation means. My interpretation is that given any element $x$ of $M$, the equation $\phi (x)^{n}+a_{1}\phi(x)^{n-1}+...+a_n=0$ holds. Am I correct? I don't think so, because modules aren't necessarily closed under multiplication.
Comment: Note that the definition of modules does not include multiplication and $\phi(x) \in M$. So the only possible interpretation is given by lhf.
Answer: Note that $\phi^{n} = \phi \circ \phi \cdots \circ \phi$, $n$ times. So, $\phi^{n} (x)=\phi(\phi(\cdots\phi(x)))$, $n$ times.
Then
$$
\phi^{n}+a_{1}\phi^{n-1}+\cdots+a_1\phi+a_n =0
$$
means
$$
\phi^{n} (x)+a_{1}\phi^{n-1} (x)+\cdots+a_1\phi(x)+a_n x=0
$$
for all $x \in M$. Just note that the $a_i$ do not depend on $x$.
Comment: So, in particular, OP's "interpretation" is not correct.

Atiyah-Macdonald Proposition 2.4


Cogs discord.py

Question: I am trying to move my discord.py code into cogs. I keep on getting the same error Exception has occurred: ModuleNotFoundError No module named 'cogs'.
bot.py
from discord.ext import commands
bot = commands.Bot(command_prefix='!')

bot.load_extension('cogs.maincog')

bot.run('token')

maincog.py
from discord.ext import commands

class MainCog(commands.Cog):
    def __init__(self, bot):
        self.bot = bot

    @commands.command()
    async def test(self, ctx):
        await ctx.send("test")

def setup(bot):
    bot.add_cog(MainCog(bot))

Folder:

Thanks!
Comment: Because both of the files are in the same directory, u can just use: bot.load_extension('maincog')
Answer: Because maincog is in the same directory as bot, cogs. doesn't need to be included.
This is because python will look for a subdirectory called cogs which is not needed in your case.
New Bot.py code:
from discord.ext import commands
bot = commands.Bot(command_prefix='!')

bot.load_extension('maincog')

bot.run('token')

Answer: The bot.py don't need to be included on the folder, so put it out and this should work, but there are some improvements:
bot.py (Out of the folder)
import os
from discord.ext import commands

client = commands.Bot(command_prefix='!')
# If you don't want the default help command and want to make by yourself, you can use this:
client.remove_command('help')

# With this event, always when your bot is online, it will be printed on the terminal:
@client.event
async def on_ready():
    print('Bot Online!')

# This causes all files inside the folder with the ending .py to be loaded, without having to load one by one:
for filename in os.listdir('cogs'):
    if filename.endswith('.py'):
        client.load_extension(f'cogs.{filename[:-3]}')

client.run('token')

OBS: I writed "client" but you can replace it for "bot" if you want to. It's just the way I prefer.
OBS2: If you keep this with "client", replace the "bot" on maincog.py by "client", just like this:

maincog.py (Inside the folder)
from discord.ext import commands

class MainCog(commands.Cog):
    def __init__(self, client):
        self.author = None
        self.client = client

    @commands.command()
    async def test(self, ctx):
        await ctx.send('test')

def setup(client):
    client.add_cog(MainCog(client))

Cogs discord.py


How do I run a Java program from the command line on Windows?

Question: I'm trying to execute a Java program from the command line in Windows.
Here is my code:
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

public class CopyFile {
    public static void main(String[] args) {
        InputStream inStream = null;
        OutputStream outStream = null;

        try {
            File afile = new File("input.txt");
            File bfile = new File("inputCopy.txt");

            inStream = new FileInputStream(afile);
            outStream = new FileOutputStream(bfile);

            byte[] buffer = new byte[1024];

            int length;

            // copy the file content in bytes
            while ((length = inStream.read(buffer)) > 0) {
                outStream.write(buffer, 0, length);
            }

            inStream.close();
            outStream.close();

            System.out.println("File is copied successful!");

        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

I'm not sure how to execute the program - any help?
Is this possible on Windows?
Why is it different than another environment (I thought JVM was write once, run anywhere)?
Comment: `javac CopyFile.java` and then `java CopyFile`
Comment: @Elizabeth Turner first you have to make sure that you have installed JRE (Java Runtime Env) and that it's accessible form every folder (the path to Java/Javac is included in the PATH env variable). Then run the commands I wrote above from the same folder in which CopyFile.java is located.
Comment: I'm at the command line now, do I need to save my txt files in the same folder as the program for them to be invoked?
Comment: What package does `CopyFile` reside
Comment: See [java](http://docs.oracle.com/javase/7/docs/technotes/tools/windows/java.html), [PATH and CLASSPATH](http://docs.oracle.com/javase/tutorial/essential/environment/paths.html) and this question [Running Java Program From Command Line](http://stackoverflow.com/q/9879679/16487).
Comment: @alfasin lets assume that JRE isn't installed, how would i go about installing it
Comment: @ARJUN, your link is no longer valid. It looks like it go to a spam place.
Comment: Check for JEP 330 for more information about running Java programs as if they was shell or Python scripts.
Comment: @elizabeth-turner, no you don't need to have the text file in the same place as the java program.  But you should probably use an argument to the `main` method to tell where it is, and then add to the command line the path to the file.  As for instance `dir`, `ls`, `copy` and `cp` works.
Comment: Check this link [Running a Java program from command Prompt](https://msccomputerscience4all.blogspot.com/2013/02/running-java-program-from-command-prompt.html)
Comment: As of Java 11 you can run directly from a single `.java` file. So `java CopyFile.java` should work.
Answer: Source: javaindos.

Let's say your file is in C:\mywork\
Run Command Prompt
C:\> cd \mywork

This makes C:\mywork the current directory.
C:\mywork> dir

This displays the directory contents.  You should see
  filenamehere.java among the files.
C:\mywork> set path=%path%;C:\Program Files\Java\jdk1.5.0_09\bin

This tells the system where to find JDK programs.
C:\mywork> javac filenamehere.java

This runs javac.exe, the compiler.  You should see nothing but the
  next system prompt...
C:\mywork> dir

javac has created the filenamehere.class file.  You should see
  filenamehere.java and filenamehere.class among the files.
C:\mywork> java filenamehere

This runs the Java interpreter.  You should then see your program
  output.
If the system cannot find javac, check the set path command.  If javac
  runs but you get errors, check your Java text.  If the program
  compiles but you get an exception, check the spelling and
  capitalization in the file name and the class name and the java
  HelloWorld command.  Java is case-sensitive!
Comment: The issue im having is when i try to run the compiler I get an error "could not find or load main class CopyFile
Comment: If you have the folder containing CopyFile.class set as the current directory, simply try "java -cp . CopyFile" Otherwise "java -cp FULLPATH CopyFile"
Comment: What's missing: if the class was declared to be in a certain package, then you have to put the `.class` file in the appropriate directory structure. Current directory is the default classpath, so you don't have to set `-cp .` manually.
Comment: For sure, you should be using a packages: package mypackage;     then go to your projectname/ .    Run java mypackage.Main
Comment: @ElizabethTurner, I had the same error, and it was because I had opened the grandparent folder and I used a project with local packages at the start of the code. The result: I had multiple projects in the working directory. If you use **package [projectfoldername];** in the first lines of your code files, you must use **java [projectfoldername]\xyz.java** "outside the projectfolder", and with that folder structure of the package you also run the class afterwards. See e.g. https://stackoverflow.com/questions/9234488/running-java-class-from-terminal
Comment: I was having a similar issue. As sugggested by @ArturCzajka I just needed to add -cp . but **it needs to be before the name of the class** So like this: java -cp . Test
Comment: How would I create a batch file that compiles&runs the code, yet if it was compile before, just launch it?
Comment: What happens when you run Java filenamehere. I'm guessing the java compiler DOESN'T compile anything and instead the .class file is being passed to the JVM?@NicholasKadaeux
Answer: To complete the answer :

The Java File
TheJavaFile.java

Compile the Java File to a *.class file
javac TheJavaFile.java

This will create a TheJavaFile.class file

Execution of the Java File
java TheJavaFile

Creation of an executable *.jar file

You've got two options here -

With an external manifest file :

Create the manifest file say - MANIFEST.mf
The MANIFEST file is nothing but an explicit entry of the Main Class
jar -cvfm TheJavaFile.jar MANIFEST.mf TheJavaFile.class

Executable by Entry Point:

jar -cvfe TheJavaFile.jar <MainClass> TheJavaFile.class

To run the Jar File
java -jar TheJavaFile.jar

Comment: can you please show the contents of MANIFEST.mf, because when I try to "Executable by Entry Point" it says the main class can't be found
Comment: @Pini Cheyni If your class is called the MainClass then your manifest file will have an entry for the Main-Class : MainClass
Comment: @PiniCheyni, The 2 ways to create executable jars are explained well here.
https://docs.oracle.com/javase/tutorial/deployment/jar/appman.html
Comment: on point 3, when you call `java TheJavaFile` thats equivalent to `java TheJavaFile.class`.. so youre not executing the java file like you stated. youre executing the class file. java files are sources which are not executable
Answer: Since Java 11, java command line tool has been able to run a single-file source-code directly. e.g.
java HelloWorld.java

This was an enhancement with JEP 330: https://openjdk.java.net/jeps/330
For the details of the usage and the limitations, see the manual of your Java implementation such as one provided by Oracle: https://docs.oracle.com/en/java/javase/11/tools/java.html
Answer: Complile a Java file to generate a class:
javac filename.java

Execute the generated class:
java filename

Comment: Notice that  your "filename" is actually a name of a class. The java.exe will search every *.jar file for a class of this name.
Comment: @Doomjunky Could you explain it, please? Is that bad or good? Should it be named differently?
Answer: In case your Java class is in some package. Suppose your Java class named ABC.java is present in com.hello.programs, then you need to run it with the package name.
Compile it in the usual way:
C:\SimpleJavaProject\src\com\hello\programs > javac ABC.java

But to run it, you need to give the package name and then your java class name:
C:\SimpleJavaProject\src > java com.hello.programs.ABC

Comment: works for java 8
Answer: Assuming the file is called "CopyFile.java", do the following:
javac CopyFile.java
java -cp . CopyFile

The first line compiles the source code into executable byte code. The second line executes it, first adding the current directory to the class path (just in case).
Comment: mycomputer:bin user$ java -cp . Main.class
Error: Could not find or load main class Main.class
Answer: It is easy. If you have saved your file as A.text first thing you should do is save it as A.java. Now it is a Java file.
Now you need to open cmd and set path to you A.java file before compile it. you can refer this for that.
Then you can compile your file using command

javac A.java

Then run it using

java A

So that is how you compile and run a java program in cmd.
You can also go through these material that is Java in depth lessons. Lot of things you need to understand in Java is covered there for beginners.
Answer: You can compile any java source using javac in command line ; eg, javac CopyFile.java.
To run : java CopyFile.
You can also compile all java files using javac *.java as long as they're in the same directory
If you're having an issue resulting with "could not find or load main class" you may not have
jre in your path. Have a look at this question:
Could not find or load main class
Answer: On Windows 7 I had to do the following:
quick way

Install JDK http://www.oracle.com/technetwork/java/javase/downloads
in windows, browse into "C:\Program Files\Java\jdk1.8.0_91\bin" (or wherever the latest version of JDK is installed), hold down shift and right click on a blank area within the window and do "open command window here" and this will give you a command line and access to all the BIN tools. "javac" is not by default in the windows system PATH environment variable.
Follow comments above about how to compile the file ("javac MyFile.java" then "java MyFile") https://stackoverflow.com/a/33149828/194872

long way

Install JDK http://www.oracle.com/technetwork/java/javase/downloads/index.html
After installing, in edits the Windows PATH environment variable and adds the following to the path C:\ProgramData\Oracle\Java\javapath. Within this folder are symbolic links to a handful of java executables but "javac" is NOT one of them so when trying to run "javac" from Windows command line it throws an error.
I edited the path: Control Panel -> System -> Advanced tab -> "Environment Variables..." button -> scroll down to "Path", highlight and edit -> replaced the "C:\ProgramData\Oracle\Java\javapath" with a direct path to the java BIN folder "C:\Program Files\Java\jdk1.8.0_91\bin".
This likely breaks when you upgrade your JDK installation but you have access to all the command line tools now.
Follow comments above about how to compile the file ("javac MyFile.java" then "java MyFile") https://stackoverflow.com/a/33149828/194872
Answer:
STEP 1: FIRST OPEN THE COMMAND PROMPT WHERE YOUR FILE IS LOCATED. (right click while pressing shift)
STEP 2: THEN USE THE FOLLOWING COMMANDS TO EXECUTE.
        (lets say the file and class name to be executed is named as Student.java)The example program is in the picture background.
     javac Student.java
     java Student

Answer: As of Java 9, the JDK includes jshell, a Java REPL.
Assuming the JDK 9+ bin directory is correctly added to your path, you will be able to simply:

Run jshell File.java — File.java being your file of course.
A prompt will open, allowing you to call the main method: jshell> File.main(null).
To close the prompt and end the JVM session, use /exit

Full documentation for JShell can be found here.
Answer: Now (with JDK 9 onwards), you can just use java  to get that executed.
In order to execute "Hello.java" containing the main, one can use:
java Hello.java
You do not need to compile using separately using javac anymore.
Comment: This feature has been added in JDK 11.
Answer: You can actually run Java program as you would shell or python scripts without manually compile the Java file, as described in
JEP 330. That is available since JDK 11.
If you create a file testing, and put the following you should be able to run it as command testing.  You  need to make it executable in Linux and Mac OSX with chmod +x testing.
    #!/usr/bin/env java --source 11
    public class Test {
        public static void main(String [] args) {
            System.out.println("Hello world!");
            System.exit(0);
        }
    }

You are not allowed to use the file extension .java in the previous example.
    $ chmod +x testing
    $ ./testing
    Hello world!
    $

But you can still execute if it is was name Test.java without the shebang "#!" prefix like this:
    public class Test {
        public static void main(String [] args) {
            System.out.println("Hello again!");
            System.exit(0);
        }
    }

Then execute the file Test.java with following command:
    $ java Test.java
    Hello again!
    $

So this works as long as you have a new enough Java compiler in the path, check with java -version.  More information in this blog.


How to fetch 8 million records of Google Cloud using bigquery API in Node.js?

Question: I am querying the Google cloud Data using Bigquery.
When i am running the query it return about 8 millions of row.
But it throws error :

Response too large to return

How i can get all 8 million records,can anybody help.
Comment: One way can be split the request, million by million, or something like that.
Comment: yes,i am spliting the records but after that it again through error like Response too large to return
Comment: Make sure that you indeed want to see all 8 million rows. It's kinda absurd you want all, and most probably you put the problem wrong, and you want to aggregate the results, and paginate them.
Comment: You have a low rate. Important on SO, you have to mark accepted answers by using the tick on the left of the posted answer, below the voting. This will increase your rate. See how this works by visinting this link: http://meta.stackoverflow.com/questions/5234/how-does-accepting-an-answer-work#5235
Comment: dont worry ,i will mark accepted once i get the result
Comment: you have a long history, all your past questions are with unaccepted answers
Answer: 1. What is the maximum size of Big Query Response?
As it's mentioned on Quota-policy queries maximum response size: 128 MB compressed (unlimited when returning large query results)
2. How do we select all the records in Query Request not in 'Export Method'?
If you plan to run a query that might return larger results, you can set allowLargeResults to true in your job configuration.
Queries that return large results will take longer to execute, even if the result set is small, and are subject to additional limitations:

You must specify a destination table.
You can't specify a top-level ORDER BY, TOP or LIMIT clause. Doing so negates the benefit of using allowLargeResults, because the query output can no longer be computed in parallel.
Window functions can return large query results only if used in conjunction with a PARTITION BY clause.

Read more about how to paginate to get the results here and also read from the BigQuery Analytics book, the pages that start with page 200, where it is explained how Jobs::getQueryResults is working together with the maxResults parameter and int's blocking mode.
Update:
Query Result Size Limitations - Sometimes, it is hard to know what 128 MB of compressed
data means.
When you run a normal query in BigQuery, the response size is limited to 128 MB
of compressed data. Sometimes, it is hard to know what 128 MB of compressed
data means. Does it get compressed 2x? 10x? The results are compressed within
their respective columns, which means the compression ratio tends to be very
good. For example, if you have one column that is the name of a country, there
will likely be only a few different values. When you have only a few distinct
values, this means that there isn't a lot of unique information, and the column
will generally compress well. If you return encrypted blobs of data, they will
likely not compress well because they will be mostly random. (This is explained on the book linked above on page 220)
Answer: try this,
Under the query window, there is an button 'Show Options', click that and then you will see some options,

select or create a new destination table;
click the 'Allow Large Results'

run your query, and see whether it works.
Comment: It will create a new table, but you can delete it later. Or you can set up a table that for this purpose, and overwrite it every time, then it kind of temporary table.  It is just my short path to solve this issue.


Can one find 4024 positive integers such that the number "2013" cannot divide the sum of any 2013 of them

Question: As stated on the title,

Is it possible to find $4024$ positive integers such that the sum of any $2013$ of them is not divisible by $2013$?

I used to assumed they have to be distinct. Being inspired by the answer below, I am thinking maybe we have to select those $4024$ numbers by two types: the first group of $2012$ integers being those that are divisible by $2013$, and the rest $2012$ integers are those not divisible by $2013$. So that if you pick any $2013$ of them you will end up having to pick at least $1$ of those $2012$ not-divisible-by-$2013$ integers. Does that work out?
If the numbers don't have to be distinct then the following suggestion seems to work well, but I am not sure whether they have to be distinct.  I will update the question if there are any changes.  Thank you guys.
Comment: If you just pick them so that *none* of them are divisible by 2013, then certainly no subset of them of size 2013 will have any numbers divisible by 2013.
Comment: Did you mean "the *sum* of any 2013 of them is not divisible by 2013" ?
Comment: oh yeah sorry that's what i meant, now the typo is corrected
Answer: Half of them 1, half of them 2013.
Comment: I definitely think this is working well when we are allowed to choose an integer more than 1 time.
But just wondering if the condition that all the 4024 numbers have to be distinct, is my way of thinking above still correct? thank you.
Comment: If you want them to be distinct, you can take the numbers $1+2013k$ and $2013k$ for $k=1,2,\dots,2012$.


Ramdisk like ZFS behavior

Question: Is it possible to configure ZFS to avoid ramdisk usage?
In next scenario:

create big file (100GB for example) or a lot of small files (there is much more free RAM than total file(s) size)
delete this file

it will be almost no disk IO. Disk IO should happen when there is no free RAM anymore or when cache flushed explicitly (before reboot/shutdown) or data in cache old enough (hours)- it is ok if cache will be flushed from time to time, but not too often (once in an hour is ok). Performance is more important than data integrity.
I could use ramdisk for that, but it will be much more convenient if it possible to configure zfs like that.
Comment: I'm confused by your question. Are you trying to avoid disk writes, or are you trying to flush to disk immediately?
Comment: Avoid. What the point to write a lot of data on slow disk if files will be deleted relatively soon after creation. And because most of data wrtten in docker, lxc & kvm machines I need proper fs backing them. Many reasons: I'm not happy with overlay graphdriver in docker and aufs does not work on debian (proxmox 5) out of the box. Manualy created and managed ramdisk will work, but I'm trying to avoid extra effort and maintaing when containers/vm will need more memory.
Answer: You're better off using a RAMdisk, maintaining it will take a bit of extra effort, but fighting ZFS as much as you'd need to to get it to do this will probably take more.
This is basically totally counter to the way that ZFS wants to work. ZFS will do everything it can to keep your data safe, and then after that it will make your data fast.
If you really want to do this on ZFS it would be really risky and I have no idea if it would actually work. You'd need to create a completely separate Zpool to handle this, and it would probably break completely very quickly after you implemented it. You'd basically need to configure ZFS to hold on to massive amounts of dirty data (sitting only in cache) and you'd be fighting ZFS' desire to keep that data reliable.
You'd need to mess around with setting an insanely high zfs_dirty_data_max limit and set zfs_delay_min_dirty_percent to 100 for the entire pool; throw a tonne of RAM at it; heavily tune the I/O queues for all write operations and the amount of writes you allow it to do at a particular time.
All of that basically would break ZFS' desire to make sure that your data stays safe no matter what happens. In addition you'd probably get your performance totally clobbered whenever you reached those limits and ZFS tried to commit 100GB of buffered data to disk at once, rather than having it make many smaller 100MB writes.
If you really want to try to make it work, take a look at the tunable parameters on ZoL (which I'm assuming you're using rather than Oracle/Solaris ZFS):
http://fibrevillage.com/storage/171-zfs-on-linux-performance-tuning
Comment: Testing thin lvm + zram (had to add types = [ "zram", 252 ] into /etc/lvm/lvm.conf). Looks good so far. In proxmox I could create temporary vms and lxc containers on lvm+zram and permanent vms/lxc on zfs+hdd. So I have added flexibility with that approach. I also should be able to run 2 docker daemons: one for lvm+zram and another for zfs+hdd - but I only read about it and have not tried yet. Unfortunately I could play with that at work only in background and I only shopping for high mem machine for home.
Comment: zfs+zram or zfs+brd also should work fine and even easier to maintain for me (as I not very familiar with lvm), but I think it less efficient, as zfs is too sophisticated for ramddrive - maybe I'll test such variant too.
Comment: That's the issue - I don't want to delay data writes to disk - I want to avoid most of them (100GB written to fs, but only 100MB written to disk). With ramdrive it will be something like zfs send/receive by cron and before shutdown. I guess easy solution does not exists, but I was hoping that zfs with huge amount of settings have some uknown to me options which will do the magic
Comment: I have a feeling that I'll end up with lvm on zram and configure to use proxmox & docker to use these lvms directly
Comment: I'm interested in seeing how you solve this, post an update if you remember to once the solution is in-place. @ISanych

Ramdisk like ZFS behavior


How does the one Spring Boot microservice know how it can send a POST to another service?

Question: In this tutorial (Spring Cloud Microservices), the code from this repository is used to illustrate the example:
https://github.com/saturnism/spring-cloud-gcp-guestbook/tree/master/1-bootstrap
In the repo, two Maven projects are found using Spring Boot (guestbook-frontend and guestbook-service). Both are run as a service. Via the frontend service, messages can be posted to the backend service. My question is, where is it configured such that the frontend knows where the POSTs go to? Where is the connection between the two?
Answer: Check @FeignClient annotation in
<code>https://github.com/saturnism/spring-cloud-gcp-guestbook/blob/master/1-bootstrap/guestbook-frontend/src/main/java/com/example/frontend/GuestbookMessagesClient.java
</code>
Answer: In order to explain this question in detail, I will highlight the core patterns of microservices

Core Development Patterns
Routing Patterns
Client Resiliency Pattern
Security Pattern
Logging and Tracing Pattern
Build and Deployment Pattern

Above mentioned patterns are used to deal with problems that come when we move onto microservices. One of the problems which your question is highlighting is that "We can never be sure which machine will be running our microservice" One naive approach to resolve this problem is that we maintain some kind of map with all the details of machines and its services. And in order to establish communication between services we can use that map. But this approach is naive and it has its own problems.
SO to deal with this we will have to make use of Routing Patterns. In Routing Patterns there is a concept of Service Discovery There are different ways to do Service Discovery that is provided by different platforms. For instance: Netflix provides Eureka and Feign Client which later on was added in Spring Cloud, HashiCorp provides Service Discovery with Consul, and AWS ECS also has its service discovery mechanism.
Having said that I can conclude that there are different ways by which you can call another service from one microservice of spring-boot. But since you are more concerned about spring cloud, In spring Cloud you would need Eureka Server and Feign Client. Adding them to your project is fairly simple.
you can follow this tutorial to use Eureka and Feign Client https://dzone.com/articles/microservices-communication-feign-as-rest-client
Comment: You highlighted " where IPS are configured such that the frontend knows where the POSTs go to? Where is the connection between the two" ---->

Since you will be using Eureka as Service Discovery, whenever your services go up it will register itself with Eureka. And since eureka has all the details of all of the services. whenever you try to contact to your service using feign client it will get its details from Eureka and forward your request to your destination sevice


Select a cell in tkinter.treeview and get the cell data

Question: I would like to use a mouse pointer to select a cell in a table created by tkinter.treeview() and print out the cell's value. How can I do this?
Below is a sample code to create a tkinter.treeview table. In this case, I would like to use my mouse pointer to click on one of the values say "Airport". However, when I click that cell, the entire row (i.e. the widget item)  is instead selected. How can I avoid this so that I can get what I want?
import tkinter as tk
import tkinter.ttk as ttk

class App(tk.Frame):
    def __init__(self, parent, *args, **kwargs):
        ttk.Frame.__init__(self, parent, *args, **kwargs)
        self.tree = ttk.Treeview(parent, columns=("size", "modified"))
        self.tree["columns"] = ("date", "time", "loc")

        self.tree.column("date", width=80)
        self.tree.column("time", width=80)
        self.tree.column("loc", width=100)

        self.tree.heading("date", text="Date")
        self.tree.heading("time", text="Time")
        self.tree.heading("loc", text="Loc")
        self.tree.bind('<ButtonRelease-1>', self.selectItem)

        self.tree.insert("","end",text = "Name",values = ("Date","Time","Loc"))
        self.tree.insert("","end",text = "John",values = ("2017-02-05","11:30:23","Airport"))
        self.tree.insert("","end",text = "Betty",values = ("2014-06-25","18:00:00","Orchard Road"))

        self.tree.grid()

    def selectItem(self, event):
        curItem = self.tree.focus()
        print (self.tree.item(curItem))

if __name__ == "__main__":
    window = tk.Tk()
    app = App(window)
    window.mainloop()

Comment: The code you posted shows the values from all the columns of the selected item, and you now claim to have figured out which column the user clicked on. Put those two together.
Comment: use the x,y from the event object. Those coordinates are relative to the widget. `pointerxy` returns the coordinates relative to the screen.
Comment: You won't be able to highlight just a single cell, but you can get the value of the cell that was clicked on. Is that what you want? Have you looked through the documentation for the Treeview widget to learn about methods that help you learn what was clicked on?
Comment: Yes that is what I want. Is `.identify_element(x, y)` the method?
Comment: How do I get the x and y of the mouse pointer when not using a canvas?
Comment: The event object that is passed to the callback has a `x` and `y` attribute.
Comment: I figure that `x,y = self.tree.winfo_pointerxy()` could give the coordinates of the mouse pointer relative to widget's root window. However, I found that  `.identify_element(x,y)` does not give the value of the clicked cell. Can you point me to the method I should be using?
Comment: I have changed to `x,y = event.x, event.y`, thanks. Still the have not found the right method to get the value of the clicked cell.
Answer: Thanks to Bryan Oakley's guidance, I have revised the selectItem callback to:
def selectItem(self, event):
    curItem = self.tree.item(self.tree.focus())
    col = self.tree.identify_column(event.x)
    print ('curItem = ', curItem)
    print ('col = ', col)

    if col == '#0':
        cell_value = curItem['text']
    elif col == '#1':
        cell_value = curItem['values'][0]
    elif col == '#2':
        cell_value = curItem['values'][1]
    elif col == '#3':
        cell_value = curItem['values'][2]
    print ('cell_value = ', cell_value)

It gave me what I wanted. Hope this script will help any other asking the same question.


Eigenvalues and eigenvector

Question: I have found this one but do not got any clue how to solve it.

Let be $A$ a symmetrical matrix $3X3$ with real numbers. Prove that if $\lambda_1$,$\lambda_2$ are different eigenvalues with their own eigenvector $v_1\neq 0$ and $v_2\neq 0$. Then $v_1$ is orthogonal to $v_2$ from that show that $v_1$ and $v_2$ are linear independences.

To me its pretty obvious the second part but I can not prove that $v_1$,$v_2$ are orthogonal
Answer: I think you probably want $A$ with real coefficients here.
Suppose $Av_1=\lambda_1v_1$ and $Av_2=\lambda_2v_2$ with $\lambda_1 \neq \lambda_2$ and $A$ is symmetric $A^T=A$.
Then
$$\lambda_1 v_1\cdot v_2= (Av_1)\cdot v_2 = (Av_1)^Tv_2=v_1A^Tv_2=v_1\cdot (Av_2) = \lambda_2 v_1\cdot v_2$$
Thus $\lambda_1 v_1\cdot v_2 = \lambda_2 v_1\cdot v_2$ if $v_1 \cdot v_2 \neq 0$ then you have $\lambda_1=\lambda_2$ a contradiction
Comment: A contradiction is when you have two properties that are both true but must have opposite truth values. Here we assumed $\lambda_1\neq \lambda_2$ and we conclude $\lambda_1=\lambda_2$. Those opposites so that's a contradiction. The conclusion is that if $v_1\cdot v_2 \neq 0$ then a contraction means that it must be true that $v_1\cdot v_2 =0$ because contradictions can't happen :)
Comment: I dont get the idea what does the contradiction mean?
Comment: ahhhh I got you mate, i did not see this part if  v1⋅v2≠0. Of course i know the meaning of a contradiction. Ok i got it thanks!
Answer: Calculate the inner product
$$
\langle A v_1 | v_2 \rangle = \lambda_1 \langle v_1 | v_2\rangle  \tag{1}
$$
But on the other hand
$$
\langle A v_1 | v_2 \rangle = \langle v_1 | A^T v_2 \rangle = \langle  v_1 | \lambda_2 v_2 \rangle = \lambda_2 \langle  v_1 | v_2 \rangle \tag{2}
$$
From (1) and (2) you get to the conclusion
$$
(\lambda_1 - \lambda_2) \langle v_1 | v_2 \rangle = 0
$$
Since the eigenvalues are different
$$
\langle  v_1 | v_2 \rangle = 0
$$
Comment: @JohnSmith No, it is the same thing $\langle x | y \rangle = x \cdot y$
Comment: Is there a diference here using dot product?
Comment: The main difference is people also use the inner product notation $\left$ for inner product spaces where the inner product is not the dot product. Function spaces and complex numbers have different inner products. That's why I put a comment about real coefficients in my answer because you'd need $A^*=A$ instead of $A^T=A$ if you had complex coefficients and the usual complex inner product (conjugate transpose).
Comment: $A$ has real coefficients!

Eigenvalues and eigenvector


Help with Poisson regression accounting for repeated measures

Question: I'm not a statistician, but need to use these clever tools to analyse some data I have.
I have a really simple dataset to analyze (see below. cases=disease counts, pop=total number of subjects sampled per year), but can't seem to settle on the most appropriate model to use.
It is count of a disease across different years.
I am interested in studying the change in disease counts over the study period (2000-2008) and to see if this change is 'significant'.
I am under the impression that poisson regression would be appropriate to model how the disease count changes over time. The disease count is its prevalence - therefore accounting for old and new cases in a year. Therefore, my data violates the assumption that the counts are independent of each other as they are not. Counts in 2001 for example may have been from the same individuals sampled in 2000.
'data.frame':   9 obs. of  4 variables:
 $ year      : int  2000 2001 2002 2003 2004 2005 2006 2007 2008
 $ cases     : int  76 103 110 129 129 135 144 130 147
 $ pop       : int  3766 4012 3993 4111 4086 4100 4060 4132 4084

I have tried performing poisson regression using glmer() by specifying 'year' as a random effect to account for clustering of cases between years, but it gives me this error message:
boundary (singular) fit: see ?isSingular

Maybe it's because my dataset is too small??
My question is quite simply how do I use poisson regression to account for repeated measures in this case?
If I were to assume independence of observations/counts, using poisson regression in glm() gives me some interpretable results showing that the increase in counts over time is statistically significant. Not sure if i can trust these results if I haven't accounted for repeated measures...
Any help, comments are appreciated!
I have added a bit more detail - please check if I am doing this correctly for the type of data I have. Here it is:
> new.prevalence.data
  year cases  pop   logpop
1 2000    60 3700 8.216088
2 2001    70 4000 8.294050
3 2002   100 3990 8.291547
4 2003   130 4100 8.318742
5 2004   140 4086 8.315322
6 2005   140 4100 8.318742
7 2006   167 4060 8.308938
8 2007   170 4132 8.326517
9 2008   175 4084 8.314832

Distribution of counts looks like this

poisson.model.rate<-glm(cases~year+offset(logpop), family=poisson, data=new.prevalence.data)

    > summary(poisson.model.rate)

Call:
glm(formula = cases ~ year + offset(logpop), family = poisson,
    data = new.prevalence.data)

Deviance Residuals:
     Min        1Q    Median        3Q       Max
-1.81701  -1.37797   0.08085   1.02010   1.76332

Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept) -229.96091   23.70549  -9.701   <2e-16 ***
year           0.11301    0.01182   9.557   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for poisson family taken to be 1)

    Null deviance: 107.55  on 8  degrees of freedom
Residual deviance:  13.72  on 7  degrees of freedom
AIC: 77.391

Number of Fisher Scoring iterations: 4

My dispersion parameter here is 1.96. Therefore I've proceeded to use the quasipoisson model:
Call:
glm(formula = cases ~ year + offset(logpop), family = quasipoisson,
    data = new.prevalence.data)

Deviance Residuals:
     Min        1Q    Median        3Q       Max
-1.81701  -1.37797   0.08085   1.02010   1.76332

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) -229.9609    33.0696  -6.954 0.000220 ***
year           0.1130     0.0165   6.851 0.000242 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for quasipoisson family taken to be 1.946079)

    Null deviance: 107.55  on 8  degrees of freedom
Residual deviance:  13.72  on 7  degrees of freedom
AIC: NA

Number of Fisher Scoring iterations: 4

Does anyone see a problem with using this model to reach the conclusion that prevalence increases with time and that this increase is significant?
i.e. a 1 unit increase in year results in a multiplicative increase by a factor of exp(0.113) that is statistically significant (p<0.05)?
Thank you
Comment: Try `glm(..., family=quasipoisson)` for more background read here: https://stats.stackexchange.com/questions/201903/how-to-deal-with-overdispersion-in-poisson-regression-quasi-likelihood-negativ
Comment: The Poisson model makes a kind of restrictive assumption (mean = variance) that can cause the model not to converge--you can maybe use a negative binomial or quasipoisson model. You can use `pacf()` to check the partial autocorrelation of the residuals to see if your data have strong correlation as well. If they do you, you should account for the correlation structure in your model, but your sample size may not be large enough for meaningful estimates.
Comment: Thanks for your comments. My data do not appear to be overdispersed. If anything they are very slightly underdispersed. I'm not too concerned about accounting for over/underdispersion of data at this stage. I guess my main question is whether or not using poisson regression is a valid method to use with count data that is NOT independent? I think using quasipoisson or negative binomial regression do not really fix this particular issue? Any other suggestions/solutions?
Comment: @beta you may be confused here. First off: dispersion is intrinsically related to correlation. A quasipoisson model is for the case when the variance is not equal to the mean, but proportional to the mean. Positively correlated samples cause dispersion < 1, aka underdispersion. It is precisely quasipoisson modeling that handles this underdispersion.
Answer: If you have a panel of identified individuals that were evaluated each year, then one way to deal with intra-individual correlations would be to include the individuals' IDs as random effects. The data for example could be formatted with one row per individual per year, annotated with the year, the ID and the event counts for the year.
At the other extreme, if these represent samples of a small proportion taken from a large underlying population, then the chance of re-sampling the same individual is small and you don't have to worry much if at all about intra-individual correlations.
As AdamO suggests in comments, a quasiPoisson model should handle any remaining "unmeasured correlation" well in that latter situation, or if you are at an intermediate level of sampling from the population.
Comment: the code I used was gee(disease_status~Year, data=data_long, family = "binomial"(link="logit"), corstr = "unstructured",id=patient_ID, na.action = na.omit)
Comment: Yes I have this data. When you say 'event counts for the year' - for an individual (ID) in a particular year - the event count will only either be 0 or 1 (as it is binomial - disease present vs. disease not present). Is this what you mean or is there a different way of incorporating event counts for the year in this 'long' format of data.
Comment: Your second paragraph does somewhat reassure me - the only issue I have is that as I have data at an individual level, I know personally that an individual that is counted as having disease in one year has also been counted as having disease in other years. Your answer has been helpful. I have added a bit more detail to my question if you have time to have a look. Thank you & apologies if I'm not making much sense as I'm clearly a noob at this subject!
Comment: @beta the quasi-Poisson analyisi behaved as hoped: similar coefficient estimates, but higher standard errors that should account for the issues you identified. A couple of cautions. First, my second paragraph was written in the context of taking repeated _random_ samples. If you had a panel of the same individuals evaluated from year to year then a random-effect model might be more appropriate. Second, can an individual have disease one year and _not_ have it in a subsequent year? If that's not the case, and you have a panel, then be careful in how you interpret the apparent prevalence trend.
Comment: I am looking at the presence of infection rather than disease per se. So actually, the same person may have tested positive for infection in one year then tested negative in the following year. I have used GEE in R to incorporate random effects. As my outcome using the long format of data will be disease - yes/no - rather than count data - my regression analysis becomes a logistic one. I am not longer sure if this type of model specs helps me answer the question of prevalence trends over time. I get an intercept of -103 and a coefficient (year) of 0.04 and have no idea how to interpret this.
Comment: @beta coefficients in a logistic regression are based on [log-odds](https://en.wikipedia.org/wiki/Logit). Intercept: log-odds when covariates are at baseline/0 values. Intercept in your models might be easier to interpret if you subtract 2000 from the Year values so that the intercept represents the Year 2000 instead of Year 0. Slope coefficient: _change_ in log-odds per unit _change_ in year. If that coefficient for `Year` is significant then you have answered your question.
Comment: you've been extremely helpful. Thanks so much!


Determining if the function $u(x,y)=xy$ is continuous and its maximum and minimum values on the unit disk.

Question: Q1: How do we show that the function $f:\mathbb{R}^2 \to \mathbb{R}$, $(x,y) \mapsto xy$ is continuous?
Work: The open sets on $\mathbb{R}$ are the open intervals $(a,b)$. We need to show that the preimage of the open interval $(a,b)$ is also open. Would $f^{-1}(a,b)= (a,b) \times (a,b)$ the cartesian product of the interval $(a,b) \subseteq \mathbb{R}$ with itself? Since the cartesian product of two open sets is open, the preimage of $(a,b)$ is open, so $f$ is continuous.
Q2: Is there a systematic method for finding the maximum and minimum values of a function $g$ on some subset $\Omega$ of the domain? I think I remember learning something about it in calculus, but I can't find anything about it in my textbook "Comple Variables" by Flanigan.
Suppose $\overline{D} \subseteq \mathbb{R}^2$ is the closed unit disk centered at the origin $(0,0)$ and that $g$ is function $f$ defined above. How do we find the maximum and minimum values of the function $f(x,y)=xy$ on the closed disk?
Since the unit disk satisfies $x^2+y^2 \leqslant 1$ and the function $f$ gets larger for increasing values of $x$ and $y$, I know that the maximum values are determined by $(1/\sqrt{2},1\sqrt{2})$ and $(-1/\sqrt{2},-1\sqrt{2})$ and the mimimum values are determined by $(-1/\sqrt{2},1\sqrt{2})$ and$(1/\sqrt{2},-1\sqrt{2})$, but I would like to learn a concrete method for finding these values, in case $\Omega$ and $g$ are more complicated.
Answer: Sketch of answers:
Q1: you only need to show
$p_1(x,y)=x, \, p_2(x,y)=y$
are continuous.
Since $\bar D$ is compact, max and min are indeed attained. Clearly neither is attained in the interior (check why), so there is only the boundary $S^1$ left to examine. Then let $x=\sin \phi,\, y=\cos\phi$.
Q2: Yes. But, in practice, usually we need to presume differentiability of $g$, and $\Omega$ a subset whose boundary is an easily identifiable curve (or a set of such curves). We first work on $\text{int}\Omega$: let $D g=0$ to obtain all the stationary points, then calculate the values assumed by $g$ at these points. Then we move to $\partial \Omega$: if the curve is readily parametrisable, then we are happy because all we need to do is to plug the curve equation into the goal function, then optimise this one-variable function; if the curve appears in an implicit form, say $r(x,y)=0$, use Lagrange multiplier method. Finally, compare all candidate extreme values against each other to find out the max and min.
Comment: Thanks for the help. One more quick question: What do you mean by the equation "$Dg=0$? For the part about the boundary $\partial \Omega$, parametrizing the circle $C_1(0)$ by $x(t)=(\cos t, \sin t)$, would the function be $f(\cos t, \sin t )= \cos t \sin t$? I'll make sure to review the Lagrange multiplier method.
Comment: @user167857 The first one: $D g=0$ is an abbreviation for $(\partial g/\partial x, \partial g/\partial y)=(0,0)$. The second one: yes, you're right.


A question about rate of convergence of an iteration method

Question: I've got these two sequences

$${x_n}{_+}{_1}={x_n}{_-}{_1}{x_n}^2$$
or
$${x_n}{_+}{_1}={x_n}{x_n}{_-}{_1}^2$$

how can I know where do they converge to and what is the rate of convergence when they are not constant and also do converge (for sure)?
for the first one I guessed that it might converges to zero only
and the second one to zero or one but I'm not really sure about that and I don't know how to show that.
we usually use $${\epsilon_n}={x_n}-c$$
$${\epsilon_n}{_+}{_1}+c=({\epsilon_n}{_-}{_1}+c)({\epsilon_n}+c)^2$$
so if it really converges to zero then c=0 and we get
$${\epsilon_n}{_+}{_1}=({\epsilon_n}{_-}{_1})({\epsilon_n})^2$$
and then I don't know how to show what the rate is.
we usually get something like that
$${\epsilon_n}{_+}{_1}={\epsilon_n}{\epsilon_n}{_-}{_1}$$
and from that I can conclude what the order is.
any help would be appreciated.
thanks.
Comment: You're right sorry.
Comment: You get from products to sums by applying the logarithm.
Answer: Consider the logarithm, then the second equation as the easier example becomes
$$
\ln|x_{n+1}|=\ln|x_n|+2\ln|x_{n-1}|
$$
is a linear recursion. Its characteristic polynomial $q^2=q+2$ or $0=q^2-q-2=(q-2)(q+1)$ has roots $2$ and $-1$, so that
$$
x_n=A^{2^n}\cdot B^{(-1)^n}
$$
where the constants are determined from the initial values $x_0=AB$, $x_1=A^2B^{-1}$.
With $|A|<1$ one gets convergence to zero, $|A|=1$ and $B\ne1$ a bounded but not converging sequence and for $|A|>1$ a diverging sequence.
Comment: Thank you.
actually I've got to that characteristic polynomial but the thing is I didn't really know how to get to the rate of convergence


Loading BigQuery GA Data to Redshift

Question: Can you please help provide ways how I can load my google analytics data in biGQuery to Redshift? Can Cloud Function be used for this? or, how do I trigger this from the BigQuery side instead of using a python script to call bigquery?
Answer: You can utilise two cloud functions to get the data into S3. Once it's in S3, you can have your own mechanism (e.g. Lambda function) to import the data into Redshift.
Preamble: set up a Stackdriver export trigger
We will trigger our first Cloud Function whenever the latest Google Analytics daily sessions table is available. This is done through triggering a Pub/Sub message whenever Stackdriver Logging indicates that the latest table has been loaded. In order to set up this trigger, follow these steps (refer to the "Pub/Sub & Stackdriver" section).
Cloud Function 1: Export BigQuery table

Trigger: Pub/Sub message for when a new daily table has been loaded
Workflow

Export table as JSON (or Avro, Parquet)
Save JSON in Google Cloud Storage

Cloud Function 2: Transfer export file to S3

Trigger: New file in Google Cloud Storage bucket
Workflow

Utilising boto read file from Google Cloud Storage
Transfer file to S3
Delete or archive file
Comment: additionally you can also use the suggested method [here](https://stackoverflow.com/questions/39329580/exporting-data-from-google-cloud-storage-to-amazon-s3) to copy from gcs to s3
Comment: @vinoaj, thanks for this! I tried following the steps Pub/Sub, and got stuck at cloud function [1] exporting the bigquery table in GCS.But i get an error - updated in the question
Comment: @Justine that looks like it might be a syntax error. Are you able to provide your whole code to debug?
Comment: @Justine - re-looking at the error message, I suspect what you have is a multiline query statement. In that case your Python syntax should be (i.e. using triple quotation marks): QUERY="""......"""
Comment: @vinoaj, thank you, that helps, but I'm getting another error. I have updated my question above to include the code. Do i need to do additional setup?
Comment: @Justine that error message looks like you haven't installed the Google BigQuery python library on your system. Try: pip install --upgrade google-cloud-bigquery
Comment: @vinoaj, I managed to export successfully in GCS, however, I wanted to adjust the filename with date format. Since i'm taking the load from Google Anaytics data in BQ, i want to take data from yesterday, For example ga_session_date_yesterday -> ga_session_20191208? and if I wanted to export it different extension like .gz, will how does it work? Thanks a lot for your help.
Comment: I solved the date issue, but i'm having issue exporting the file into a readable csv with .gz compression, updated query above

Loading BigQuery GA Data to Redshift


Create Custom Search and result page

Question: I am trying to create a custom search page with two input fields.
Company Name and Certification Code are two fields to be entered by user.
I have a content type called certification and the search will be performed based on this Content type.
A field collection is used to enter certifications details(Certificate Name     ,Certificate Number    ,Certificate ,Issuing Date: ,Date of Expiry).
The result page will separated one or under the same page of search form.
I tried many ways and i am not convinced with the results. I created a page node--certification.tpl to show the result
Here i want two things :

In the search page Two input fields to be entered  ,the combination of two items in content type(company name,Certificate number.) Here certificate number is a field collection value.I couldn't search from the field collection.I used custom search module.
Show the result in this page directly when we search from the search page instead of showing the intermediate result page.

Search Page

Search Result looks like

I hope someone will help me to get a perfect search page.
Comment: What is the actual problem/question? Is this a presentation issue? Or a functionality one? You should update your question accordingly, will help you get a better answer from the community. Cheers!
Comment: yes i updated the answer
Answer: Installing module or modifying .tpl files are not required to get a simple search page.
Creating a view and using Exposed Filters setting will get the job done.
Providing you high level steps below, you may need to research a bit more to accomplish this task.
Steps

Create a View.
Choose "Table" in Format section.
Add "Certificate Name ,Certificate Number,Certificate,Issuing Date,Date of Expiry" in field section.
In filter criteria, select your content type.
Also add "Company name, Certificate number" in the Filter Criteria.
Click Advanced > Better Exposed Forms > Better Exposed Filters.

Expose the fields "Company name & Certification number" as text fields.
You can expose this fields as a Drupal Block and assign it anywhere in the page.
Comment: Thanks for the valuable answer.But the certificate number is coming from the field collection "certifications details", Can we mention the field collection items separate in better exposed filter?

Create Custom Search and result page


Help solving a geometry problem involving circles and arcs

Question: Can any of you offer some advice on how to solve this problem?
Show that if point $P$ is external to a circle and tangent $PT$ and secant $PAB$ are drawn where $T, A$ and $B$ lie on the circle, then $\angle TPA \circeq \frac 12 (BT-AT)$ where $BT$ and $AT$ represent the lengths of the arcs on the circle.
I know that this is a direct consequence of the theorem that states the angle between a chord and the tangent at one of its endpoints is equal in degrees to half the subtended arc. I'll probably need to use this somewhere in the proof, but I'm not sure where.
I appreciate any help.
Comment: Your equation looks as if there is an angle on the left and a length on the right??
Comment: Yes, sorry. BT and AT are meant to represent the lengths of the arcs on the circle.
Comment: It is still nonsense. Let's assume the statement holds for some configuration $P,T,A,B$. Then zoom out the whole picture from $P$: it preserves tangent and section points and the angle $\angle TPA$, but increases the arc lengths $AT$ and $BT$.
Answer:
$\angle PTA=\angle TBA, \angle TAB=\angle TPA+\angle PTA, \angle TPA=\angle TAB-\angle TBA. $
$\angle TDB=2\angle TAB, \angle TDA=2\angle TBA.  \angle TPA=\dfrac{\angle TDB-\angle TDA}{2}$
let $DT=DB=DA=r$
$\angle TDB=\dfrac{\stackrel{\frown}{TB}}{r},\angle TDA=\dfrac{\stackrel{\frown} {TA}}{r}, \angle TPA=\dfrac{1}{2r}(\stackrel{\frown}{TB}- \stackrel{\frown}{TA}).$
so you miss the r .


How to save the discord user that used the slash command corresponding to the button in discord.py

Question: I'm trying to make an application slash command that sends the application to a mod channel, where the mods can either accept or deny the application.
But currently if 2 users use the /apply2 command, the button that is send to the carrier-app-log channel will both give roles to the user that used the command second (because the global variable id is set to the new id)
@client.tree.command(name="apply2", description="Apply for carrier")
@app_commands.describe(player_name="Player Name", profile="Profile Name", floor="Floor", proof="Proof")
async def apply2(interaction: discord.Interaction, player_name: str, profile: str, floor: int, proof: discord.Attachment):
  global id
  global role

  get_player_profile(player_name, "dungeons", profile)
  if check_reqs_dungeon(floor):
    global role
    await interaction.response.send_message("Submitted Succesfully", ephemeral=True)
    channel = discord.utils.get(interaction.guild.channels, name="carrier-app-log")
    embed=discord.Embed(title=player_name.capitalize(), url=f"https://sky.shiiyu.moe/stats/{player_name}/{profile}", description=f"{interaction.user.mention} applied for floor {floor} carrier on account {player_name.capitalize()}", color=368)
    id=interaction.user
    role=floorRoles[floor-1]
    print(role)
    embed.set_image(url=proof.proxy_url)

    await channel.send(embed=embed, view=ApplyButtons())
  else:
    await interaction.response.send_message(f"You do not meet the minimum requirements for floor {floor}, the minimum requirement is {floorReqs[floor-1]}", ephemeral=True)

class ApplyButtons(discord.ui.View):
  def __init__(self):
      super().__init__(timeout=None)

  @discord.ui.button(label="Accept", style=discord.ButtonStyle.green, custom_id="accept")
  async def accept(self, interaction: discord.Interaction, button: discord.ui.Button):
    global role
    global id
    user = id
    print(user)
    print(role)
    if type(role) is int:
      role=interaction.guild.get_role(role)
      print(role)
      print("Converted")
    if role not in user.roles:
      print("Not in user roles")
      await user.add_roles(role)
      await interaction.response.send_message(f"{role.mention} role given to {user.mention}", ephemeral=False)
    else:
      print("In user roles")
      await interaction.response.send_message(f"Applicant already have {role.mention} role", ephemeral=False)

I tried to set the the custom_id of the button to be the original global id, and was hoping I can access the custom_id of the button for the username everytime the button is pressed. @discord.ui.button(label="Accept", style=discord.ButtonStyle.green, custom_id=id)
user = self.custom_id
but an attribute error is returned saying
"AttributeError: 'ApplyButtons' object has no attribute 'custom_id'"
Thanks ;-;
Answer: Do not use global variables. For Discord bots in particular, where commands are ran often by different users with different information, using global variables is bad practice.
The way to use a variable from a command function to a view class, is by taking it as an argument when initialising the class. In other words, make __init__ take an argument.
class MyView(discord.ui.View):
    def __init__(self, user): #  <-
        super().__init__()
        self.user = user #  Storing it as an attribute in the class

    @discord.ui.button(label="Accept", style=discord.ButtonStyle.green, custom_id="accept")
    async def accept(self, interaction: discord.Interaction, button: discord.ui.Button):
        user = self.user #  Getting the user argument

@client.tree.command(name="command")
async def command_callback(interaction: discord.Interaction):
     await interaction.response.send_message(...,
                                             view=MyView(interaction.user)) #  Parsing the argument

Another thing to note: try not to use built-in function names as a variable name.
id is a function, so you shouldn't be using it as a variable because it overwrites its use. This is especially important since in your code, id is a global variable. Otherwise, it would only overwrite the id function locally (in the function it is defined in).
@discord.ui.button(label="Accept", style=discord.ButtonStyle.green, custom_id=id) user = self.custom_id would also not work.
var = None

class MyView(discord.ui.View):
    def __init__(self):
        super().__init__()

    @discord.ui.button(label="Accept", style=discord.ButtonStyle.green, custom_id=var)
    async def accept(self, interaction: discord.Interaction, button: discord.ui.Button):
        var = self.custom_id

self refers to the class. Methods in a class, like the button callback, take in an argument we conventional name self. This is the class object.
If you want to access the custom_id, you instead want to take it from the button itself, not the class.
    @discord.ui.button(label="Accept", style=discord.ButtonStyle.green, custom_id="accept")
    async def accept(self, interaction: discord.Interaction, button: discord.ui.Button):
        custom_id = button.custom_id


Verificar se um device possui GPS e Serviço de localização está ativo

Question: Preciso verificar se no iPad que estou usando possui GPS e se o serviço de localização do app esta ativo durante a utilização do app, como eu posso conseguir essas informações?
Answer: Utilizando a classe CLLocationManager você pode utilizar o método locationServicesEnabled, que é retornado um valor BOOL.
if (![CLLocationManager locationServicesEnabled]) {
    // Localização não habilitada
}

Acredito que com isso você consiga o que quer.