Generating embeddings for languages without a written representation?

Question: I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as generating and embedding for a language that doesn't have a written representation at all, which is probably the case for most of the world's languages (in terms of the number of languages, not number of speakers).
What are the approaches for generating an embedding for a language without a written representation at all? The following approach seems feasible:

Record enough samples to capture all the phonemes in the language.
Generate a suitable encoding of the phonemes.
Record a large amount of speech in that language using the encoding from the previous step to convert it to a digital format.
Use the data to generate an unsupervised embedding. Use this as the starting point for downstream classification and more complex NLP tasks.

This seams perfectly feasible, but I feel I might be missing something:

Most languages don't have a one-to-one mapping between sequences of phonemes and sequences of symbols, e.g. homographs or homophones.
Is it as straightforward as I think it is to capture all the unitary phonemes from a sufficiently large sample? More specifically, this might be informed by my familiarity with English, Arabic, French (alphabet based), and Japanese (diphtongs) - all with a relatively small and discrete set of phonemes. But this might not be the case with other languages with more complex pronunciation rules, like tonal languages, or subsaharan African languages?
Some languages have demographic based pronunciation variations (based on gender, social class, etc...). Would these variations make a direct phoneme-to-binary encoding more challenging?

To clarify where I am coming from, based on Tim's response:

For the actual problem at hand, yes the data is available. It is a local dialect (Tunisian Arabic) of a standard language that already has a wide corpus and some embeddings available (Modern Standard Arabic). It is however different enough from MSA, as to be mutually unintelligible, and have significant variations in grammar, syntax, cases, etc.... Tunisian Arabic does have a large body of streaming and older video data (and a lot of it captioned into English or French actually).
I originally went down the path of trying to figure out whether the embeddings from the more standard parent language were useful, maybe they could be fine tuned to Tunisian Arabic. I assumed somebody had already researched the relevance of Mandarin embeddings for Cantonese NLP problems, or Spanish embeddings for Catalan and Andalusian, etc...
But then how do we solve for the lack of written standard, and the fact that different demographics use different alphabets for the task, either a modified Arabic alphabet to account for additional sounds not present in MSA, or latin letters augmented with numbers to account for both native MSA sounds and Tunisian specific sounds (e.g. the proper spelling of "Ali" is actually "3ali" using this more recent informal norm.
Then I realized that it might be easier to just treat it as a language with no written representation at all, since that problem might have been researched more often than the somewhat complex dialect type problem I just mentioned. Hence my post.
Answer: Embeddings, as other neutral networks, need huge datasets. People usually scrape thousands of books, news articles and web pages for that. For audio files, this would mean thousands of hours of recordings, considering audio quality issues, probably even more. For English, this would mean something like scraping all the audio from Netflix etc. Are you sure you have this data? If not, recording it sounds like a project for years.
Moreover, embeddings are usually trained in scenarios like "guess next word", so the data would be manually labeled by someone to mark beginnings and ends of words. This alone is also a lot of work.
The reasons above is why we have a lot of NLP models for English and not many for other languages. It is easy to scrape the Internet for thousands of documents in English. Same for written language: you don't have audio quality issues to worry about, written language is already recorded in unified form, words are easily separable in most languages etc. With audio there's much more additional preprocessing needed.
You are correct about the other issues mentioned in bullet points by you. You would probably need samples diverse in terms of regional variations of pronunciation, genders, age, education, etc. to make the data representative.
Comment: my post was slightly confusing. For the actual problem at hand, yes the data is available. It is a local dialect (Tunisian Arabic) of a standard language that already has a wide corpus and some embeddings available (Modern Standard Arabic). It is however different enough from MSA, as to be mutually unintelligible, and have significant variations in grammar, syntax, cases, etc....  Tunisian Arabic does have a large body of streaming and older video data (and a lot of it captioned into English or French actually).
Comment: I originally went down the path of trying to figure out whether the embeddings from the more standard parent language were useful, maybe they could be fine tuned to Tunisian Arabic. But how do we solve for the lack of written standard, and the fact that different demographics use different alphabets for the task, either a modified Arabic alphabet to account for additional sounds not present in MSA, or latin letters augmented with numbers to account for both native MSA sounds and Tunisian specific sounds (e.g. the proper spelling of "Ali" is actually "3ali" using this more recent informal code
Comment: Then I realized that it might be easier to just treat it as a language with no written representation at all, since that problem might have been researched more often than the somewhat complex dialect type problem I just mentioned. Hence my post
Comment: @SkanderH. than the problem is labeling the data, so you could do something like "predict next word". Otherwise, this does not seem to differ much from written text, doesn't it?
Comment: I'm not sure I am following: Embeddings are usually treated as unsupervised models first, where the focus is on finding structure in a very large corpus, and transforming it into a Euclidian type space. My question is around whether finding such a structure is possible when the input sequences (sentences and paragraphs) are captured directly from speech instead of from written documents.
Comment: @SkanderH. usually you train them in scenarios like: you are given previous word(s) and need to predict next word. In written text, words are separated, so you can easily distinguish them and split the text into individual words to predict. With audio, you don't have that. Maybe it'd be possible to predict few next seconds of audio given previous seconds of audio, so you would have something like a character-level model in written text. Obviously, it'd be much less interpretable than model for written text, but I don't know how you want to use the embeddings.


Find the sum of these definite integrals

Question:
(Ignore the pencil scrawls in the image, they're not by me).
This question was asked on a Singapore Facebook page supporting parents of kids taking O levels (that's age range 15 to 16). Basic calculus is taught in this course, but I found this question a little too challenging to be pegged at that level - at least the way I did it, so I wanted to ask if I'm missing a much easier approach. I also didn't need one particular piece of given info (the intermediate point $(4,7)$) so I'm left wondering if I am completely on the wrong track here. Anyway, here's the question in words, paraphrased.
You're given a sketch of a function $y = f(x)$ that's continuous, monotonic and concave upwards over the part of the domain of interest. You're told it passes through points $P(3,2);Q(4,7);R(6,9)$. You're asked to evaluate $\displaystyle \int_3^6 ydx + \int_2^9 xdy$.
Here's how I did it:
$\int_3^6 ydx + \int_2^9 xdy \\ = \int_3^6 ydx + \int_3^6 x\frac{dy}{dx}dx \\ = \int_3^6 (y + x\frac{dy}{dx}) dx \\ = \int_3^6 \frac{d}{dx} (xy) dx \\ = \int_6^{54} d(xy) \\ = 48$
Would be grateful for a critique or alternative solution. Thank you.
Comment: The area between the graph and the relevant axe,  sorry.
Comment: I do not know why the Q point is given....
Comment: @dmtri Indeed, point $Q$ is not needed or used. What *is* used, however, is that the function must be invertible, which is probably assumed from the drawing.
Comment: The integrals given are the area between the axes minus a rectangle,  so it suffices to calculate the area left =6*9-2*3=48
Comment: See for example the proof without words [here](https://en.wikipedia.org/wiki/Integral_of_inverse_functions#Statement_of_the_theorem).
Comment: I thought of the rectangle thing too, but I'm having a hard time getting $\int _2^9 xdy = \int _3^6 xy'dx$. Probably something stupidly obvious I'm not getting..
Comment: @AlvinLepik Change of bounds with chain rule. I'm careful to use the Leibniz notation in these cases to avoid ambiguity.
Answer: The way you did it is the standard approach, but the problem becomes much simpler if you think what the integrals given represent. The first integral represents the area covered by the function from  $P$ to $R$ with respect to the $x$-axis, while the second integral represents the area covered between the same points, wrt the $y$-axis.
Hence the addition of the integrals represents the difference in area of two rectangles with edges along axes.
Let the perpendicular from $P$ be called $A$ and $B$ on $x$- and $y$-axis respectively, and those from $R$ be $C$ and $D$, then the given sum of integrals represents:
$$ar(DRCO)-ar(BPAO)=9\cdot 6-3\cdot 2=48$$
Comment: I agree monotonicity is important but giving a single intermediate point is not sufficient to establish that. So specifying point $Q$ is neither necessary nor sufficient.
Comment: That's why I said that $Q$ is irrelevant.
Comment: Thank you. That makes sense, and it will appeal to those with more geometric insight. But what of the point $(4,7)$?
Comment: That is quite irrelevant. But the monotonocity condition is important
Comment: Thanks, I agree.
Answer: The first integral equals the pink area, the second one the blue one.
You just need to calculate a sum (or a difference, depending on an approach) of areas of two rectangles.


How to alter file type and then save to a new directory?

Question: I have been attempting to change all files in a folder of a certain type to another and then save them to another folder I have created.
In my example the files are being changed from '.dna' files to '.fasta' files. I have successfully completed this via this code:
files = Path(directory).glob('*.dna')
for file in files:
          record = snapgene_file_to_seqrecord(file)
          fasta = record.format("fasta")
          print(fasta)

My issue is now with saving these files to a new folder. My attempt has been to use this:
save_path = Path('/Users/user/Documents...')
for file in files:
    with open(file,'w') as a:
          record = snapgene_file_to_seqrecord(a)
          fasta = record.format("fasta").read()
          with open(save_path, file).open('w') as f:
              f.write(fasta)

No errors come up but it is definitely not working. I can see that there may be an issue with how I am writing this but I can't currently think of a better way to do it.
Thank you in advance!
Answer: Hi, You can use os lib to rename the file with the new extension (type)
import os
my_file = 'my_file.txt'
base = os.path.splitext(my_file)[0]
os.rename(my_file, base + '.bin')

And you can use shutil lib to move the file to a new directory.
import shutil

# absolute path
src_path = r"E:\pynative\reports\sales.txt"
dst_path = r"E:\pynative\account\sales.txt"
shutil.move(src_path, dst_path)

Hope that can be of help.


issue pending certificate, not apearing on client

Question: I am requesting a certificate that must be approved by the CA administrator.
When I make the request, it appears in the CA pending request folder and I right click to issue the certificate. but the approved certificates don't appear in the personal folder of the requesting computer.
I have tried refreshing the certificates mmc and updating the machine group policies but the certificate does not appear in the client computer.
My question what I need to do after issuing a certificate in order make it appear in the certificate vault of the computer.
Answer: You need to select top node (Certificates), right-click -> All Tasks -> Automatically Enroll and Retrieve Certificates. Follow certificate enrollment wizard to retrieve and install pending certificate.
Answer: First of all, you need your requests number; If you requested your cert using mmc, you'll not know your request number. But you'll find it in the CA Console, during approval or afterwards under "issued certificates".
With this number, you can:

certreq -retrieve X" where "X" is your request number
A popup appears; select the CA from which you requested your certificate
you get back a certificate with your public key from the ca, which you can download to desktop
now just import this certificate with mmc under the same user account you've done the request

As result, you should see the imported certificate has attached a key sign on the icon, which means you have a private key for this certificate. If so, you were successful.
Answer: Fix for me:

Launch CA mmc
Right click CA > Properties
Click "Policy Module"
Click Properties...
Select "Follow the settings in the certificate template, if applicable. Otherwise, automatically issue the certificate."


how can i got the recent Saved id of my SQL database in next form

Question: When i am trying to retrieve the last value from my data base it returns the first one.
By using the following Query.
SELECT [Id] from dbo.Tb_Patient;

I want to get the last value.
What query i should use?
Comment: Either `SELECT [Id] FROM [dbo].[Tb_Patient] ORDER BY [Id] DESC` or (better) look at http://stackoverflow.com/questions/7917695/sql-server-return-value-after-insert
Comment: ***last value*** ordered by what criteria? A SQL database (whatever product you're using) doesn't have any implicit order - you need to **define** the order you want by issuing `ORDER BY ......` in your query
Answer: Try this
SELECT Top(1) [Id] from dbo.Tb_Patient order by Id desc;

Alternatively if Id is Auto Incremented column then you can use MAX function as well to find Maximum value.
SELECT MAX([Id]) from dbo.Tb_Patient

Comment: Assuming `Id` is of a type that orders properly, if it's a GUID etc it won't work.


VB.NET Chat Server Error

Question: I recently got into this article. It makes a chat server and people can connect to it like an AIM. The problem how would I disconnect from the server from the client. Because if I close the client application when its connected to the server it will crash the server as well. Please let me know if anyone has any solutions...
Thanks
Answer: Wherever in your code the crash is occurring, you need to set a try/catch block to catch a Connection Closed exception.
Comment: What is the Connection Code exception called?
Comment: I just call it Try/ Catch ex as Exception/End try kind of thing

VB.NET Chat Server Error


Finding the equation of the graph y=a^x +b from given points (2,5) and (0,-4)

Question: I am trying to figure out the equation of an exponential graph from two given points.
I know you have to plug each set of points into the equation, but I forget exactly how to solve this.
As stated in the question, the equation for the graph is y=a^x +b
Fist set of points: (2,5)
set of points with y intercept: (0,-4)
Thank you all.
Comment: Well, if you know you have to "plug", what equations does that give you?
Answer: As you have said, just put the points into the equation:
\begin{align}
5 & = a^2 + b \\
-4  & = a^0 + b
\end{align}
The second equation directly gives you the first value you need, inserted into the first equation you can easily calculate $a$.
Comment: Technically, without an assumption that $a$ is positive (or that the graph should be some continuous function with domain $\mathbb R$) this doesn't determine $a$. But I'm absolutely certain that the author of the question (whether it was user115944 or not) either said $a$ is positive, or meant to.


Add another button removes the default values from my fields on node form

Question: I have a node form on which two fields are displaying.
One is a list of radio buttons which is required field.
I am getting a value from url using arg(3) and setting the default value of that field
$form['field_test'][LANGUAGE_NONE]['#default_value'] = arg(3);

I have a link field which is a multi valued field. It means this field as add another button.
When I click on this add another button and then submit the form, I see an error of Test field is required, and that default value of test field gone.
Without clicking the add another button everything works fine.
How can I set a default value of the test field in this case?
Code:
if (!$form['nid']['#value']) {
  if (arg(3)) {
    // Select the default value.
    $tg = arg(3);
    $form['field_test'][LANGUAGE_NONE]['#default_value'] = $tg;
  }
}

  // Make the field readonly.
  $form['field_test'][LANGUAGE_NONE]['#attributes']['disabled'] = 'disabled';

Comment: Looks like you are ignoring delta? Also please post the form code.
Comment: Added the code.
Answer: Ok. I found the solution. Let me share my code here.
I stored the argument in the $form_state['storage] so that it can be preserved during the ajax call.
if (arg(0) == 'node' && arg(3)) {
    $form_state['storage']['field_test_value'] = arg(3);
}

// Only on node creation.
if (!$form['nid']['#value'] && isset($form_state['storage'])) {
    $form['field_test'][LANGUAGE_NONE]['#default_value'] = $form_state['storage']['field_test_value'];
}

$form['field_test'][LANGUAGE_NONE]['#attributes']['disabled'] = 'disabled';

Hope this will help someone.


How to make a join select in Laravel?

Question: What I've got now is that I retrieve all the users and then loop each and check if there are an item, otherwise if there are no relation while we are retrieving the name of the item, we will get an error
<code>$users = User::all();

foreach($users as $user)
{
  if($user->item)
  {
     echo $user->item->name;
  }

}
</code>
My question is how to select all the user records that have a relation in the items table? using the eloquent Model in laravel-4?
I'am asking this because here http://laravel.com/docs/eloquent - are no examples of using joins, only joining them after the selecting, but this is not good because we select all the rows instead of only those who has relation, also we will not have to check if the item exists, because we already know that the item really exists...
Answer: All you need is:
<code>$users = User::has('item')->get();
</code>
1st example here http://laravel.com/docs/eloquent#querying-relations
Answer: http://laravel.com/docs/queries#joins
Try this
<code>DB::table('users')
            ->join('item', 'users.id', '=', 'item.user_id')
            ->select('item.name')
            ->get();
</code>

How to make a join select in Laravel?


How can I get my local network machines to call local web server by URL?

Question: I have four computers connected to a single switch - unconnected to the internet.
One of the computers (the large one in the picture) is more powerful than the others and is running Ubuntu, with three live Node.js web servers with different apps on different ports.
The other computers are running Windows.
I want the other three computers to access the web servers by calling a URL, for example

files.bugsteins   --> 192.168.0.5:3000
chat.bugsteins    --> 192.168.0.5:3001
devices.bugsteins  --> 192.168.0.5:3002

What is the best (priority to fastest set up time) way to accomplish this ?
Thus far I have tried Bind9 on the server machine and assigning static IPs and manual hardcoded  DNS on the three client machines but it is not working and as I work on it, I figure I will post this question in case someone has a better way.
Comment: What have you tried? What technologies do you think might be in play to get this working?
Comment: Good point. I updated my question
Comment: Hear you can find a way to do this using nginx and local DNS or host file entries. https://www.digitalocean.com/community/questions/multiple-domains-on-different-ports-of-the-same-droplet
Answer:
What is the best (priority to fastest set up time) way to accomplish this?

Not BIND.
Edit the c:\windows\system32\drivers\etc\hosts file on each desktop and point those names to 192.168.0.5.
192.168.0.5 files.bugsteins
192.168.0.5 chat.bugsteins
192.168.0.5 devices.bugsteins

Add a web listener on port 80 on the server, somehow with your Node.js setup, or a basic webserver install (Lighttpd, NginX, Apache, or heaps of others) serving a static page with a JavaScript in it which looks at the URL and redirects the page to the appropriate port.
Taking from https://stackoverflow.com/questions/18022636/redirection-based-on-url-javascript something like this (untested):

<title>Redirect page</title>

Comment: I would prefer setting up a web server at port 80 on the web server, and then having it proxy the request to the Node.js servers.
Comment: Take a look at Apache's mod_proxy and the directives ProxyPass & ProxyPassReverse to redirect inbound http traffic to other ports.  Combined with dnsmasq you have something that should work!
Comment: I think both of the comments go against `best (priority to fastest set up time)` in that configuring a web server proxy will take more time and effort than 'serving a static page'.
Comment: @TessellatingHeckler this worked great thanks a lot!
Answer: The fastest will be installing the dnsmasq package depending on your Linux distros(Arch/CentOS/RHEL/Ubuntu). Keep all the entries in /etc/hosts on your DNS server (which is using dnsmasq). Point all your workstations/linux hosts to here.
Also, make sure to set forwarders to outside internet DNS server for non-local requests.
For small environments say <100 machines, this is good enough and minimalistic DNS server.
Comment: I mentioned the server machine is running Ubuntu.
Comment: Can you explain what you mean regarding updating the /etc/hosts/ on my Ubuntu machine (which is the server)? Don't I want to do that on my client machines?
Comment: Sorry for the late reply. Doesn't really matter if it's CentOS/RHEL/ubuntu as long as it's dnsmasq software.

That means your Ubuntu (DNS server) - the one and only DNS server will need to have all these configs below. You don't need to configure any complex zone files like BIND service.

ubuntudnsserver001 (/etc/hosts):
192.168.1.1 ubuntupc001
192.168.1.2 ubuntupc002
192.168.1.3 ubuntupc003
...
Comment: Also, you don't have to configure dnsmasq on your client machines. Just the Ubuntu machine which is your DNS server.

All your workstations will point to your Ubuntu DNS server as normal through /etc/resolv.conf file.
Answer: Normally this would be handled with an internal DNS server.  If you do not have one running, you could just add entries to the local host file on each of the 3 client computers.


Prevent ajax submit function being called when it fails validation

Question: I have a form, that when it's submitted, should be validated, and if valid, the results of that form should then appear below the form.
This works fine without AJAX:
function _form_validate($form, &$form_state) {
  if (!is_numeric($form_state['vsalues']['zip'])) {
    form_set_error('zip', 'Zip must be number');
  }
}

function _form_submit($form, &$form_state) {
    $values = $form_state['values'];
    $html = "
zip: ". $values['zip'] . "
";
    $form_state['storage'] = $html;
    $form_state['rebuild'] = TRUE;
 }

and then this gets printed out my main form in
  $form['output'] = array(
    '#markup' => (isset($form_state['storage']))?$form_state['storage']:'',
  );

If I make it an AJAX form, and change the submit function to:
function _form_submit($form, &$form_state) {
    $values = $form_state['values'];
    $html = "
zip: ". $values['zip'] . "
";
    $form_state['storage'] = $html;
    $form_state['rebuild'] = FALSE;
    return array("#markup" => $html);
}

It works fine apart from the fact the submit callback is triggered even when validation fails (e.g. passing a non-numeric value shows both a validation error yet also prints "zip: ").
What am I doing wrong? How can I stop the submit function being called?
Comment: I'm getting the exact same error, I'll let you know if I find the answer.
Answer: Only thing you should have to put in your AJAX submit callback is this
  if (!form_get_errors()) {
    // Do your processing
  }
  else {
    // Return your form with errors.
   return $form;
  }

Answer: I admitedly couldn't find an elegant solution for this but I found a decent workaround that solved my problem.
In my validation callback:
// Track errors and submissions ourselves.
$form_state['form_has_errors'] = FALSE;
$form_state['form_was_submitted'] = FALSE;

if ($condition) {
  form_set_error($form['fieldname'], t('Error message'));
  $form_state['form_has_errors'] = TRUE;
}

In the submit callback:
// Check for our custom error flag.
// For some reason form_get_errors() is empty, even when validation errors ocurred.
if (!$form_state['form_has_errors']) {
  // Put your submit callback code here.

  $form_state['form_was_submitted'] = TRUE;
}

In the AJAX callback:
if ($form_state['form_was_submitted']) {
  // Do something different here.
}
else {
  // For has errors and wasn't submitted.
}

Again, this is admittedly NOT an elegant solution but I couldn't find any other way to make Drupal not call the submit callback if errors were found.
Answer: Try to put it in hook_node_validate. if your form is built using content type then it will definitely help you.
Another way might help you like, check it in hook_node_presave() and make exit() call if your validation got fails.
Answer: Its an old issue, but i just worked on the same topic:
  // prevent default submit if form does not validate
  $('form').find('input[type="submit"]').on('click', function (e) {
    var $form = $(this).closest('form');
    if ( !($form[0].checkValidity && $form[0].checkValidity())) {
      e.preventDefault();
    }
  });

This is using the browser form validation which i used in a Drupal.behaviors function.


Proof for $\lor$ Elim: rule in Soundness Theorem

Question: So far I have been told to assume the line is invalid and then arrive at a contradiction.
Suppose the first invalid step derives the sentence $C$ by an application of $\lor$ Elim to the sentences $A\lor B$ and $A$ and $B$ appearing earlier in the proof. Let $P_1,\ldots,P_n$ be a list of all the assumptions in force at $C$. If this is an invalid step, $C$ is not a tautological consequence of $P_1,\ldots,P_n$.
Since $C$ is the first invalid step in $p$, we know that $A\lor B$, $A$ and $B$ are all valid steps, that is, they are tautological consequences of the assumptions in force at those steps.
Since $\mathcal{F}_T$ allows us to cite sentences only in the main proof or subproofs whose assumptions are still in force, we know that the assumptions in force at steps $A\lor B$, $A$ and $B$ are also in force at $C$. Hence the assumptions for those steps are among $P_1,\ldots,P_n$.
But I'm not sure how to carry this on...
Comment: If so you are proving it by contradiction, i.e. you are assuming that the rule is **not** sound, i.e. that the rule produce a conclusion $C$ which is not a log cons of the premises.
Comment: But the proof is by induction on the lenght of the derivation, and this means that $A \lor B$ and $A$ and $B$ are all log cons of the open assumptions.
Comment: yes I think the first comment is correct. I have the soundness theorem and I want to prove the twelve rules. Do you know how this is done? Is proof by contradiction not a good approach?
Answer: Sketch of the proof
We have a derivation $\mathcal D$ of $A \lor B$ with assumptions in $\Delta$.
We have a derivation $\mathcal D_1$ of $C$ from $A$ with assumptions in $\Delta_1$ and a derivation $\mathcal D_2$ of $C$ from $B$ with assumptions in $\Delta_2$.
Let $\mathfrak A$ a model of $\Delta \cup \Delta_1 \cup \Delta_2$.
By induction hypothesis : $\Delta \vDash A \lor B$.
Two subcases : either $\mathfrak A$ is a model of $A$ or it is a model of $B$.
In the first subcase by induction hypothesis ($\Delta_1 \vDash C$) $\mathfrak A$ is a model of $C$.
In the second subcase by induction hypothesis ($\Delta_2 \vDash C$) $\mathfrak A$ is again a model of $C$ .
In both cases $\mathfrak A$ is a model of $C$.
Answer: There's no compelling reason to use proof by contradiction.  The rule consists of a valid one in intuitionstic positive logic, so like anything else in intuitiionistic positive logic it can get proved without the use of proof by contradiction.
Suppose that (A $\lor$ B) is true, (A$\rightarrow$C) is true, and (B$\rightarrow$C) is true also.  By the truth table for (A $\lor$ B) one of two cases gets satisfied.  A holds true consists of one case, and B holds true for the other case.  Suppose that A holds true.  Then, by modus ponens C will follow.  Similar reasoning shows that C holds for the other case.  Since that exhausts all cases, C follows from the set of premises.
Note the above does NOT use $\lor$ in the reasoning.  $\lor$ consists of an objective level construct, and is not as comprehensive as meta-linguistic case exhaustive analysis, since case exhaustive analysis could have many more cases than two, while $\lor$ as an objective level connective, defined by the definition of a well-formed formula, consists of a binary connective.
Comment: Ahh I had three examples of other rules and they all involved a contradiction and then went on to say that the rest can be solved in the same way. To assume an invalid claim and then derive a contradiction by assuming that all the lines before are valid


Local Representations of Forms on Manifolds

Question: I've been doing some work with differential forms on manifolds. I am still new to the subject so please excuse me for any minor gaffes. Currently, every discussion of manifolds I have done is with them in some ambient space, usually a dim $k$ manifold in $R^n$. Now, (and I think this is why some books say it is often easier to work without an ambient space).
Now, when we write a $k$ differential form on this manifold, the presence of the ambient space complicates things. Specifically, if I have a form defined on some coordinate patch of a k surface in $R^n$, the form has to be written as:
$$
\sum_I f_I dx_I
$$
But, if we have a diffeomorphism, we can use this diffeomorphism to pull-back the form to a top form in $R^k$, giving it the neat representation $f dx_1 \wedge \cdots \wedge dx_k$. My question is: is this pulled back form "equal" to the form in the ambient space? It seems like there would be some differences, for example, where the forms are being evaluated (evaluation being the k vectors that the form takes in and spits out a real number with).
Answer: The general idea of the formalism of differential geometry is to find rules for representations of objects which actually should not depend on the representation. This includes embeddings into some ambient space or local coordinate system. A $k$- form on a manifold is a purely geometric object, so this applies to $k$-forms, too.
In older books on the topic or in phisics textbooks you'll find complicate looking equations which verify the "correct" transformation behavior for coefficients of differential geometric objects (like vector field, $k$-forms or other tensor fields) which is nothing but the assertion that these objects are, in fact, independent of the coordinate system. A similar calculation (which I will refuse to write down right now) will show that the answer to your question is 'yes'. The pulled back form is equal to the form in the ambient space in the sense that, if you plug in a set of $k$ vectors tangent to the manifold in some point and calculate the result, the result will be the same regardless of the representation (of course you have to adapt the representation of the $k$ vectors, as well).
Note though, that there are geometric quantities which do depend on the embedding (e.g. some curvatures of a curve or surfaces).


appendChild não está funcionado

Question: Estou tentando passar os elementos do array com for( of ) para os elementos de html que eu criei pelo js. O problema é q quando tento colocar o  criado no js dentro do, usando appendChild,o código da erro.

Esse é o javascript:
var listElement = document.querySelector('#app ul');
var inputElement = document.querySelector('#app input');
var buttonElement = document.querySelector('#app button');

var todos = [
    'Fazer café',
    'Estudar Javascript',
    'Lavar a Louça'
];

function renderTodos(){

for(todo of todos){
    var todoElement = document.createElement('li');
    var todoText = document.createTextNode(todo);

    todoElement.appendChild(todoText);
    listElement.appendChild(todoElement);
   }
}

renderTodos();

Esse é o html;
<!DOCTYPE html>

<meta charset="UTF-8">
<meta http-equiv="Content Language" content="pt-br">
<title>Parte 3</title>

       <input type="text" placeholder="Digite um To do">
       <button>Adicionar</button>

Erro ocorrido:

script.js:18 Uncaught TypeError: Cannot read property 'appendChild' of
  null at renderTodos (script.js:18) at script.js:22
Comment: Qual a mensagem de erro? Copiei o código no JSFiddle funcionou normalmente.
Comment: script.js:18 Uncaught TypeError: Cannot read property 'appendChild' of null
    at renderTodos (script.js:18)
    at script.js:22
Comment: Seu script deve estar rodando antes do DOM estar pronto, então a **** ainda não existe, coloque o script depois da tag ****.
Answer: É a ordem da sua execução, você tem que usar ou DOMContentLoaded ou o atributo defer na tag

Outra maneira seria simplesmente tirar o script do head e colocar no final do body, assim:
<!DOCTYPE html>

<meta charset="UTF-8">
<meta http-equiv="Content Language" content="pt-br">
<title>Parte 3</title>

       <input type="text" placeholder="Digite um To do">
       <button>Adicionar</button>

Não é necessário usar defer ao mesmo tempo que DOMContentLoaded, apesar que ambos tem uma pequena diferença na hora/tempo que irão executar, mas ainda sim ambos serão após o DOM estar acessivel as funções como querySeletor, getElementById, etc
Comment: Vlw, funcionou. Vou cometer muito ainda esse tipo de erro, sou iniciante.

appendChild não está funcionado


no osgi ready dependencies

Question: Currently I'm working with osgi and karaf.
My problem is the no "osgi ready" dependencies , which means a jar that is not ready to be deployed as a bundle into karaf for example.
I tried two solutions in order to deal with this kind of problems :

I tried to to use "Embed-Dependency" which will include the jar
dependency with the project... I don't think this could be a solution
because when I try to embed the jar , it will ask me to include other
jars that the first jar depend on , and so on ..
I tried to convert the no "osgi ready" jars into bundles using bnd tool or from "Plug-in from Existing JAR Archive" from eclipse project.
And this led to the same result , each jar will call another jar that it depend on it..

I am not sure if I'm doing it the wrong way or what is the problem exactly.
Any tips how to deal with no osgi ready dependencies ?
Answer: The simplest way to start is to use the wrap: protocol to auto create a jar. Behind the scenes it uses bnd to create a bundle on the fly. Simply prepend wrap: to the mvn url of the jar.
When you try to install the jar using bundle:install -s wrap:mvn:... karaf will tell you which imported packages are missing. Install jars that provide these packages in the same way. The pom of the jar can give you a hint what is missing.
This can mean to install lots of jars if your initial jars has lots of dependencies.
Once you have a list of jars that are installable together you can either create a feature using wrap protocol or you can make bundles from the individual at build time.
In any case you should take a look are the servicemix bundles. It provides OSGi ready bundles for many libraries.
Comment: thanks for the answer , but after i did osgified the bundles , I always get a java.lang.NoClassDefFoundError.

Before that I had many exceptions of class not found , and I fixed that with exporting the needed packages.

no osgi ready dependencies


Match multiline XML with regex and sed or awk

Question: I know that it is not the best way to match xml with sed or awk and regex, but I have no other choice in the environment I am encountering this problem.
I did not find any answer which solves my problem.
Following XML content:
<testTag name="findThisName">
    <content>...</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>

The regex should match the whole tag with name findThisName including all content. With this regex it works, but only if the content is in one line:
<testTag name(?:(?!<\/testTag>).)*findThisName.*?<\/testTag>

Anybody has an idea how this can be solved with sed or awk? Thanks!
Comment: what environment are you working on so that you don't have access to other tools?
Comment: What are you trying to do with the tag once you've found it?
Comment: Just get rid of it, simple delete... ;)
Answer: awk does not have all of perl's regex features but this might work for you:
$ awk '/<testTag[^>]*name="findThisName"/,/<\/testTag>/{next} 1' file
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>

How it works
awk allows us to specify a range of lines in the form /regex1/,/regex2/ which will match lines starting with regex1 and ending with the first occurrence of regex2.  We use this to skip over the unwanted lines:

/<testTag[^>]*name="findThisName"/,/<\/testTag>/{next}
For all lines in the range that starts with <testTag[^>]*name="findThisName" and ends with <\/testTag>, skip to the next line.
The chosen starting regex, <testTag[^>]*name="findThisName", allows testTag to have several attributes.  We do not require that name="findThisName" be the first attribute.
1
For all other lines, tell awk to print them. 1 is awk's cryptic shorthand for print a line.  If you prefer being explicit, replace this with {print $0}.
Answer: Something like this seems to work in awk. Since you mentioned deleting this tag, I'm not printing out those lines.  Note this will fail with nested testTags.
awk 'BEGIN {open=0}
    $0 ~ /<testTag name="findThisName">/ {open=1}
    open==1 && $0 ~ /<\/testTag>/ {open=0; next;}
    open==1 {next;}
    open==0 {print;}'

Marks when its found the start tag, then checks if it ends on the same line and removes it if so.  If not, skips lines until its hits the closing tag.  Outside of the desired tag it just prints.
Using this test input:
<testTag name="findThisName">
    <content>1<content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>
<testTag name="findThisName">
    <content>2</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>
<testTag name="findThisName">
    <content>3</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>

It seems to work as expected:
~$ awk 'BEGIN {open=0}
        $0 ~ /<testTag name="findThisName">/ {open=1}
        open==1 && $0 ~ /<\/testTag>/ {open=0; next;}
        open==1 {next;}
        open==0 {print;}' testxml.txt

<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>

Answer: $ awk '/<testTag name="findThisName"/{f=1} !f; /<\/testTag>/{f=0}' file
<testTag name="doNOTfindThisName">
   <content>...</content>
</testTag>

Answer: If you don't need a full regex match then don't bother with one. Something like the following should work for you.
awk '
# Find the start line and set our flag.
/^<testTag name="findThisName">$/ {f=1}

# Print the line if we aren't currently in the flagged tag.
!f {print; next}

# Find the end of the flagged tag and unset our flag.
f && /^<\/testTag>$/ {f=0}
'

This won't work for nested <testTag> elements though as the first </testTag> will trigger the unflagging block.


Unable to load CSS and JS file after adding @REQUESTMAPPING annotation at class level

Question: I've added @RequestMapping annotation at class level for some views (jsp). When I try to access those pages with proper URL mapping, Pages displayed on but unable to render properly and it unable to load .js and .css files. In browser I get follwing error.
 GET 404 (Not Found)
where as in server log I get following messages.
org.springframework.web.servlet.PageNotFound noHandlerFound
WARNING: No mapping found for HTTP request with URI [/ccb/invoice/webjars/adminlte/2.3.3/plugins/iCheck/icheck.min.js] in DispatcherServlet with name 'spring-dispatcher'.
Not sure why is thing hapening although static resources mapping at Servlet config file is perfectly fine and its loading for those pages which dont have @requstmapping annotation at class level.
Can you Please let me know where I making mistake.
@Controller @RequestMapping("/invoice") @SessionAttributes("ListOrderItems") public class InvoiceController {
    @RequestMapping(value = "/index", method = RequestMethod.GET)
    public ModelAndView adminPage() {
        System.out.println("Im in /Index** Mapping");
        String username;
        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        username = auth.getName(); //get logged in username
        username.toUpperCase();
        // Logic to build menue based on the Role of the user.
        HashMap hm = new HashMap();
        OrderItems od = new OrderItems();
        ModelAndView model = new ModelAndView();
        model.addObject("usr", username);
        //Set the object for Menue Links on the Page
        model.addObject("mnue", hm);
        model.setViewName("index");
        model.addObject("odi",od);
        return model;

    } }

Dispatcher Servlet configuration file: <mvc:resources mapping="/webjars/**" location="/webjars/" />
    <mvc:resources mapping="/resources/**" location="/resources/" />
    <mvc:annotation-driven/>
    <bean id="viewResolver"
          class="org.springframework.web.servlet.view.InternalResourceViewResolver"
>
        <property name="prefix">
            <value>/WEB-INF/jsp/</value>
        </property>
        <property name="suffix">
            <value>.jsp</value>
        </property>
    </bean>

Answer: Try adding xmlns in your  dispatcher servlet
xmlns="http://www.springframework.org/schema/mvc"
xsi:schemaLocation="http://www.springframework.org/schema/mvc
     http://www.springframework.org/schema/mvc/spring-mvc.xsd

Then
<resources mapping="/resources/**" location="/resources/" />
    <beans:bean
        class="org.springframework.web.servlet.view.InternalResourceViewResolver">
        <beans:property name="prefix" value="/WEB-INF/views/" />
        <beans:property name="suffix" value=".jsp" />
    </beans:bean>


AsyncIO and concurrent.futures.ThreadPoolExecutor

Question: I'm building a web scraping API, and most of my scraping is done with AsyncIO coroutines, like this:
async def parse_event(self):
   do scraping

# call the func
asyncio.run(b.parse_event())

This works perfectly fine, but as I'm scraping multiple websites at the same time, I was using concurrent.futures.ThreadPoolExecutor at first to scrape with multiple threads.
But since I've implemented the coroutine logic, I cannot now use the asyncio.run method in my thread directly.
Before (without coroutine):
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
     w1_future = executor.submit(self.w1.parse_event)
     w2_future = executor.submit(self.w2.parse_event)
     w3_future = executor.submit(self.w3.parse_event)

After, I would have expected something like below
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
     w1_future = executor.submit(asyncio.run(self.w1.parse_event))
     w2_future = executor.submit(asyncio.run(self.w2.parse_event))
     w3_future = executor.submit(asyncio.run(self.w3.parse_event))

Unfortunately it is not working.
Comment: Due to GIL it is unlikely that multiple threads will actually result in a performance gain compared to async.
Comment: ``asyncio`` is already a way to use a single CPU concurrently, just as ``threading`` is a way to use a single CPU concurrently. You gain nothing by throwing both at the same task.
Comment: If you really want to, it should be ``executor.submit(asyncio.run, self.w1.parse_event)`` instead.
Comment: @MisterMiyagi is `asyncio.run` guaranteed to be thread safe? Since it is not mentioned in
the docs, I would assume it is not, and so this is not safe.
Comment: I've reopened the question. Passing `asyncio.run` with parameters to the executor is not safe.
Comment: @freakish ``asyncio.run`` doesn't do any concurrency by itself, so it's about as thread safe as ``+``. The operands are what makes the operation thread safe or not.
Comment: @MisterMiyagi `executor.submit(asyncio.run, self.w1.parse_event)` will run `asyncio.run` in seperate threads. So I'm asking you: is `asyncio.run` thread safe?
Comment: @freakish This is not speculation, this is working with ``asyncio`` for ages. The docs *do not* say that ``asyncio.run`` is thread-safe, just as the docs do not talk about thread safety for most things.
Comment: @MisterMiyagi yes, everything that is not **explicitly** stated as thread safe should be assumed to be not thread safe. Most things in Python are not thread safe. That's precisely why methods like `asyncio.run_coroutine_threadsafe` exist.
Comment: @freakish ``asyncio.run_coroutine_threadsafe`` is for submitting a coroutine to an event loop *across* threads. That's something completely different than running event loops that do not interact.
Comment: @MisterMiyagi you've missed the point: `run_coroutine_threadsafe` is thread safe because it is explicitly stated to be. Thread safety of `parse_event` is for the OP to take care of. Again: there is no indication that `asyncio.run` is thread safe and so no reason to believe it is. Even if it is thread safe now, it may not be tomorrow. The Python docs don't guarantee it is. Ergo the code you've propsed is simply not guaranteed to be thread safe. That's all there is to it, no point in discussing this anymore.
Comment: I've removed my answer. After further analysis I don't think this can be done in a thread safe way, unless the standard guarantees that loops themselves don't share any state and can be used independently from threads. Which I couldn't find. Still, even if they were threadsafe it is unlikely that the executor will result in a performance gain.
Comment: @MisterMiyagi What if are dealing with I/O so it is not CPU bound like a hard drive or camera. Then splitting something off to a separate thread makes sense right?
Comment: @TheCodeNovice In this case *No*, since that is what asyncio already does. Both asyncio and threading are for I/O bound tasks - there is little point using both unless it's for compatibility with existing code.
Answer: Both asyncio and threading are a means to use a single core for concurrent operations. However, this works via different mechanisms: asyncio uses the cooperative concurrency of async/await whereas threading uses the preemptive concurrency of the GIL.
Mixing the two does not speed up execution since both still use the same single core; instead, overhead of both mechanisms will slow down the program and the interaction of the mechanisms complicates writing correct code.
To achieve concurrency between multiple tasks, submit them all to a single asyncio event loop. The equivalent to executor.submit is asyncio.create_task; multiple tasks can be submitted at once using asyncio.gather. Note that both are called inside the loop as opposed to outside the executor.
async def parse_all():
    return await asyncio.gather(
        # all the parsing tasks that should run concurrently
        self.w1.parse_event,
        self.w2.parse_event,
        self.w3.parse_event,
    )

asyncio.run(parse_all())

If you absolutely do want to use separate, threaded event loops for each parse, you must use executor.submit(func, *args) instead of executor.submit(func(args)).
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
     w1_future = executor.submit(asyncio.run, self.w1.parse_event())
     w2_future = executor.submit(asyncio.run, self.w2.parse_event())
     w3_future = executor.submit(asyncio.run, self.w3.parse_event())

Note that mixing asyncio and threading adds complexity and constraints. You might want to use debug mode to detect some thread and context safety issues. However, thread and context safety constraints/guarantees are often not documented; manually test or inspect the operations for safety if needed.


Como comparar duas listas encadeadas em C?

Question: Galera estou com problema nessa questão:
-Tenho que comparar duas listas encadeadas e retornar 1 se forem iguais ou 0 se não forem.
A função está assim e não consigo progredir:
  struct Node
  {
 int data;
 struct Node *next;
 }

 int CompareLists(Node *headA, Node* headB)
 {
  /*se as duas forem vazias*/
   if(headA->next==NULL && headB->next==NULL)
  {
   return 1;
   }
   if(headA->next==NULL || headB->next==NULL)
   {
   return 0;
   }
   while(headA->next!=NULL && headB->next!=NULL)
   {
   if(headA->data=!headB->data)
   {
   return 0;
   }else{
   return 1;
   }
   headA=headA->next;
   headB=headB->next;

   }
   }

Acho que o problema está em percorrer a lista (headA=headA->next), mas não sei como fazer. Alguém pode me ajudar?
Comment: Por igual, você define o mesmo conteúdo na mesma ordem?
Comment: Não atualize o código da pergunta com código de respostas, pois isso invalida quaisquer respostas dadas até ao momento, e não é a forma como o StackOverflow funciona.
Comment: @Isac já consegui muito obrigado. Desculpa pelo incomodo.
Answer: Para comparar listas é necessário percorrer as listas até chegar a uma conclusão.
Esse while só irá acabar quando pelo menos uma delas chegar a NULL (que vai ser testado no while) logo:
while(headA->next != NULL && headB->next != NULL){
    ...
    headA = headA->next;
    headB = headB->next;
}

Essas são as últimas linhas, ou seja, quando headA estiver na penúltima estrutura (ou seja, headA->next seja NULL), o while não será realizado pois
headA atingiu a penúltima e headB não:
while(0 && 1) => while(0)

Ou seja, acabou a iteração.
Quando acaba a iteração, a função não dá resposta, simplesmente sai pois as únicas saídas que a função tem são os if's
int CompareLists(Node *headA, Node* headB)
{
   /*se as duas forem vazias*/
   if(headA->next==NULL && headB->next==NULL)
   {
       return 1;
   }
   if(headA->next==NULL || headB->next==NULL)
   {
       return 0;
   }
   while(headA->next!=NULL && headB->next!=NULL)
   {
       if(headA->data=!headB->data)
       {
           return 0;
       }else{
           return 1;
       }
       headA=headA->next;
       headB=headB->next;

   }
}

Comment: `while` com `return` no `if` e no `else` ? O seu else não seria antes fora do `while` ? E o `=!` não seria `!=`  ?
Comment: o que o @Isac disse é verdade, =! não existe... ou é == ! ou é !=
Na edição meti a minha forma de testar se 2 listas são iguais. Se quiser testar a minha forma a ver se está bem faça-o. Caso esteja correta leia e perceba. Alguma coisa mande mensagem para mim que eu ajudo.
Eu não tenho muita experiência com listas, mas farei o meu melhor para o ajudar.
Comment: @YODA a primeira iteração (dentro do while) vai dar 1 devido ao else
Comment: Troquei de lugar o return 1 e pus depois da { do while  e tirei o else, porém, a saída ta diferente do esperado. Está com a saída 1 1 1 1 e tem que retornar 1 0 0 0 segundo o que está no exercício.
Comment: Obrigado pela observação pessoal, mas continua dando erro. Será que falta alguma condição? Vou editar o código ai em cima
Comment: @DarkGV mudei agora o código da uma olhada ai em cima.
Comment: @YODA meta no fim da função return 1 e teste sff
Answer: #include <string.h>
#include <strings.h>
#include <stdio.h>
#include <conio.h>
#include <stdlib.h>

typedef struct No
{
    int nConteudo;
    struct No *prox;
}TNo;

typedef TNo* TLista;

void liberar_memoria(TLista p);

void verificaPar(TLista p1, TLista p2);

void mostraLista(TLista p1, TLista p2);

int main(void)
{
    TLista pNo1 = NULL;
    TLista pNo2 = NULL;
    TLista pNo3 = NULL;

    TLista pNo4 = NULL;
    TLista pNo5 = NULL;
    TLista pNo6 = NULL;

    pNo1 = (TLista) malloc(sizeof(TLista));
    pNo1->nConteudo = 110;

    pNo2 = (TLista) malloc(sizeof(TLista));
    pNo2->nConteudo = 210;

    pNo3 = (TLista) malloc(sizeof(TLista));
    pNo3->nConteudo = 310;

    pNo4 = (TLista) malloc(sizeof(TLista));
    pNo4->nConteudo = 110;

    pNo5 = (TLista) malloc(sizeof(TLista));
    pNo5->nConteudo = 210;

    pNo6 = (TLista) malloc(sizeof(TLista));
    pNo6->nConteudo = 310;

    pNo1->prox = pNo2;
    pNo2->prox = pNo3;
    pNo3->prox = NULL;

    pNo4->prox = pNo5;
    pNo5->prox = pNo6;
    pNo6->prox = NULL;

    mostraLista(pNo1,pNo4);

    verificaPar(pNo1, pNo4);

    liberar_memoria(pNo1);
    liberar_memoria(pNo4);

    return 0;
}

void liberar_memoria(TLista p)
{
    TLista atual = p;

    while (atual != NULL)
    {
        TLista proximo = atual->prox;

        free(atual);

        atual = proximo;
    }
}

void mostraLista(TLista p1, TLista p2)
{
    TLista i,j;
    printf("\nMostrando a lista:\n");

    if((p1)&&(p2))

    for (i=p1,j=p2; p1!=NULL;p2=p2->prox,p1=p1->prox)
    {
        printf("\n%d\t", i->nConteudo); printf("- %d\n", j->nConteudo);

        i=i->prox;  j=j->prox;
    }

    else

    printf("Lista vazia.");

    printf("\n\n");
}

void verificaPar(TLista p1, TLista p2)
{
    TLista i,j, cont=0;

    if((p1)&&(p2))
    {
        for (i=p1,j=p2; p1!=NULL;p2=p2->prox,p1=p1->prox)
        {
            if (i->nConteudo!=j->nConteudo)
            {
                printf("NAO SAO IGUAIS - LISTAS DIFERENTES!\n");
                    cont=0;
                        break;
            }
            else
            {
                cont++;
            }

            i=i->prox;

            j=j->prox;
        }
            if (cont)
            {
                printf("AS LISTAS SAO IDENTICAS\n");
            }
    }

    else

    printf("LISTAS IDENTICAS PORQUE AMBAS SAO VAZIAS.");
}


Terraform waits for ASG removal while doing graceful shutdown of EC2 instances

Question: I use Terraform to set up ASG for my worker servers running sidekiq. On deploy when AMI image_id changes the instances require long graceful shutdown before terminating to finish job processing (max. 30m).
My idea to accomplish that is to:

Use autoscaling module with an initial lifecycle hook that sets up a transition state to autoscaling:EC2_INSTANCE_TERMINATING when removing old ASG and creating new ASG on deploy.
Send transition state to SQS queue.
Set up AWS Lambda function that receives state messages from SQS, sends a remote command with SSM to initiate graceful shutdown on the EC2 instance. Graceful shutdown script running on the instance will send SIGTERM to sidekiq, monitor if sidekiq finished and then send shutdown -h now command at the end to stop the instance.
Lambda exits immediately and does not wait until shutdown script finishes which is expected. It sends SNS notification on success and failure.
Now, the problem is that Terraform waits until ASG is successfully removed and it hits timeout which by default is 10m. I would like to force Terraform to continue after ASG is only selected to be removed without actually waiting for it to finish removal.

I have force_delete=true and wait_for_capacity_timeout=0 options set up.
From docs:

force_delete - (Optional) Allows deleting the autoscaling group
  without waiting for all instances in the pool to terminate. You can
  force an autoscaling group to delete even if it's in the process of
  scaling a resource. Normally, Terraform drains all the instances
  before deleting the group. This bypasses that behavior and potentially
  leaves resources dangling.

Terraform exists after 10 minutes with the following error message:
Error: Error deleting autoscaling group: Auto Scaling Group still exists

Why isn't it working? Do you think it is a bug?
Answer: I think the issue is that the ASG won't delete itself until all the instances are removed, and its waiting for the hooks to finish running.
Thoughts:
1) Unless you need the queing mechanism of SQS for something, its usually easier to trigger lambda via a CW event
2) When sending the shutdown -h command, you may also want to have it send a command to finish the lifecycle hook so that you don't have to wait for the hook to timeout for the ASG to finish the scaling activity
https://docs.aws.amazon.com/cli/latest/reference/autoscaling/complete-lifecycle-action.html
3) The setting you are changing is for launching instances, the docs say "A maximum duration that Terraform should wait for ASG instances to be healthy before timing out".  Which implies its only for launching
https://www.terraform.io/docs/providers/aws/r/autoscaling_group.html
4) Try changing the 'delete' timeout:
https://www.terraform.io/docs/providers/aws/r/autoscaling_group.html#delete


Caching for every user

Question: In my website, users will be redirected to a "frontpage" page where they will be able to see the latest news, their special announcements, their own materials, etc. Now I have to use DB to retrieve all those information.
I also want to use caching for this purpose so the data on their frontpage refresh every 3 hours let's say instead of each visit.
I am using the following function to achieve this:
<code>function mymodule_cache($reset = FALSE) {
  static $my_data;
  global $user;
  if (!isset($my_data) || $reset) {
    if (!$reset && ($cache = cache_get('user_data:' . $user->uid)) && !empty($cache->data)) {
      $my_data = unserialize($cache->data);
    }
    else {
      $my_data = ... (This is where I do all the DB queries, etc.);
      cache_set('user_data:' . $user->uid, serialize($my_data), 'cache', time() + 10800);
    }
  }
  return $my_data;
</code>
}
now this works great, but my question is how if I have tons of users, let's say 10,000 users, would it be efficient to still using this caching method since we are filling up the table rows with data?
Or if not, what is the best solution?
Thanks
Luca
Answer: In general, yes, you can do this. There is no limitation on the number of cache entries that can be stored. However, if using the default database cache backend, it can fill up your database and when additionally using the default cache bin, it can slow down other cache lookups. So, I would recommend:

Use a custom cache bin instead of the generic cache table. Especially important if you don't follow the second point. Because the cache bin 'cache' usually holds information that is requested on almost every request.
Use a different cache backend, e.g. Memcache. That avoids that your database gets big due to these cache entries and makes cache lookups faster. See https://drupal.stackexchange.com/questions/12307/where-is-the-best-place-for-cache-bins-drupal-7

Two additional tipps:

Monitor the number reads/writes of that cache to see if the cache actually helps. If, say, 50% of the cache entries are only written and never read, you might want to consider to avoid caching this information.
Especially when using a cache backend like Memcache (which has a smaller overhead for cache calls), it might be helpful to split the data up into several separate cache entries. There a number of benefits for this:

You can cache certain information longer than others (maybe there only is a new announcement every few days but multiple new comments/whatever per hour
Some of the data might not even be user specific but only specific for the roles a user has or even global. So for example, all anouncements could be read from a single cache entry.
You can use cache_clear_all() to clear specific caches on defined events. For example, clear all news caches if a new news is published. By doing this, all users will be able to see the news immediatly. If this is working well, you can increase the cache expiration time.

Caching for every user


count data and Poisson GLMs to predict monetary amounts

Question: I have to predict money amounts, which are always greater than 0. The distribution is very tailed (i.e. there are many small values but also many large data).
Just wondering would a count data model (e.g. GLM + Poisson) be suitable for this kind of scenario?
Answer: No.

you shouldn't expect to get different results by changing from dollars to cents or from working in thousands to working in millions - but that's what happens if you use a Poisson model.
With money amounts you would generally expect spread to be proportional to mean rather than its square root (this would apply also to quasi-Poisson models). Indeed if one considers the effect of changes in inflation/interest under a Poisson or quasi-Poisson model (even if it's just constant either side of one change), we're left with inconsistent assumptions about variance.

Among GLMs "spread proportional to mean" would tend to suggest a gamma model.


DLP in a Cyclic group

Question: Let $q$ be a prime. $G$ is a cyclic group of order $q^2$. Show that for solving the DLP in $G$ it's enough to solve two distinct DLPs in two groups of order $q$.
---
We are looking for an $x$ such that $\alpha^x=\beta$ in this group $G$.
By the CRT $G \cong C_q \times C_q$ alright? So taking the CRT-isomorphism yields
$\phi(\alpha^x)=(\alpha^x \mod q, \alpha^x \mod q)\overset{!}{=} (\beta \mod q,\beta \mod q)=\phi(\beta)$
So we have to solve $\alpha^x=\beta$ in $C_q$ now? But is this correct? I doubt that because in the exercise they want two distinct DLPs ???
Comment: This is pretty hard to read because of the acronyms and the undefined function $\phi$.  But $G\equiv C_q \times C_q$ is definitely not true; the Chinese Remainder Theorem only applies to two relatively prime integers.
Comment: Well $\Phi$ is the well-known CRT isomorphism. Okay you are right this is not true. But how does this exercise work then?
Answer: Write $x=x_1+x_2q$. Solve $a^{x_1} = b \bmod{q}$ to find $x_1$.
Let $a^q \bmod{q} = c$, $b a^{-x_1} \bmod{q}=d$.
Find $x_2$ by solving $c^{x_2} = d \bmod{q}$.

DLP in a Cyclic group


Distribution of the $L^{2}$ norm of a vector of components drawn from Gaussian distributions

Question: I recently asked this question involving uniform distributions. I am wondering what would be the equivalent for Gaussian distributions. The problem states as follows.
We consider a random vector $\vec{v} = \left(x_{1}, x_{2}, \dots, x_{n}\right)$ built from $n$ real random variables drawn from a Gaussian distribution $\mathcal{N}\left(\mu, \sigma\right)$, $\mu$ and $\sigma$ being the same for all $x_{i}$.
What is the distribution $D$ of the $L^{2}$-norm of such random vectors $\vec{v}$: $\left\lVert\vec{v}\right\rVert_{2} = \sqrt{x_{1}^{2}+x_{2}^{2}+\dots+x_{n}^{2}}$?
In other words, what is the analytical expression of the distribution obtained through this numerical experiment:
<code># Packages
import numpy as np
import random as rd
import matplotlib.pyplot as plt

# Parameters
mu = 5
sigma = 2
n = 10
count = 100000

# Compute a random norm
def random_norm(mu, sigma, n):
    v = [rd.gauss(mu, sigma) for i in range(0, n)]
    return sum([x ** 2 for x in v]) ** (1./2.)

# Generate random vectors and compute their norm
norms = [random_norm(mu, sigma, n) for i in range(0, count)]

# Plot the resulting distribution
plt.hist(norms, 100)
plt.show()
</code>
Comment: It will be related to the [Chi-distribution](https://en.wikipedia.org/wiki/Chi_distribution), possibly up to some scaling and shifting.
Comment: Searching about the [non-central chi-squared distribution](https://en.wikipedia.org/wiki/Noncentral_chi-squared_distribution), maybe...
Comment: @rzch, the sum of n standardarized normals squared is chi-squared n but  that's not also true when you take the square root ? not clear what you mean upto some scaling and shifting ? thanks.
Comment: The Chi distribution is the distribution of the square root of a Chi-squared random variable. Since the question is being asked about non-standardised normals, I'm not immediately sure how the resulting distribution compares to the Chi distribution. I think having $\mu = 0$, $\sigma \neq 1$ results in a scaled Chi distribution, but having non-zero $\mu$ might complicate things a bit.
Comment: It turns out our site has *many* posts related to this: https://stats.stackexchange.com/search?q=non-central+chi.
Answer: The non-central Chi distribution with $k$ degrees of freedom gives the distribution of $\sqrt{\sum_{i = 1}^{k}\left(X_{i}/\sigma_{i}\right)^{2}}$ for independent Gaussian random variables with variances $\sigma_{1}^{2}, \dots, \sigma_{k}^{2}$ and means $\mu_{1}, \dots, \mu_{k}$. It has the density function
$$f\left(x;k,\lambda\right)=\dfrac{e^{-\left(x^2+\lambda^2\right)/2}x^k\lambda}
{\left(\lambda x\right)^{k/2}} I_{k/2-1}\left(\lambda x\right)$$
where $\lambda = \sqrt{\sum_{i = 1}^{k}\mu_{i}^{2}}$ and $I_{k/2-1}\left(\lambda x\right)$ is a modified Bessel function of the first kind.
So it follows that in our case with standard deviation $\sigma$ and mean $\mu$, the density function of the $L^{2}$ norm will be that of a scaled non-central Chi distribution with $n$ degrees of freedom $f\left(x/\sigma;n,\lambda\right)$, where $\lambda = \sqrt{n}\mu/\sigma$.
To see this, we can view our desired norm as being formed by
$$\sqrt{X_{1}^{2} + \dots + X_{n}^{2}} = \sigma\sqrt{Y_{1}^{2} + \dots + Y_{n}^{2}}$$
where $Y_{i} \sim  \mathcal{N}\left(\mu/\sigma, 1\right)$.


Recipient detail in SendGrid Dynamic Template

Question: I need to send a payload to multiple recipients in SendGrid. I use the Dynamic Templates to construct the email body, and all is working fine.
I would like to add a 'Hi <code>{{recipientName}}</code>' line to the email but I cannot find any documentation on it, is it possible?
I cannot include the recipient detail in the payload as a single payload goes out to many recipients
Comment: What programming language
Comment: Also, why not use a for loop
Answer: Use personalizations.

Personalizations allow you to override these various metadata for each email in an API request.

Your request will look like this:
<code>{
  "from": "firstname.lastname@example.com",
  "template_id": "YOUR TEMPLATE ID",
  "personalizations": [
    {
      "to": [
        {
          "email": "email@example.com"
        }
      ],
      "substitutions": {
        "%fname%": "John",
        "%CustomerID%": "C001"
      },

    },
    {
      "to": [
        {
          "email": "firstname.lastname@example.com"
        }
      ],
      "substitutions": {
        "%fname%": "Peter",
        "%CustomerID%": "C005"
      },
      "send_at": 1629723541
    }
  ]
}
</code>
and it will go as a single request.
Also look at this github comment.
Comment: gh link broken.


How to create a multi-level menu schema in Zod

Question: I am trying to create a multi-level schema for Zod that can be derived from this data:
{
    "data": [
        {
            "title": "Devices",
            "items": [
                {
                    "items": [
                        {
                            "title": "something",
                            "url": "/products/devices/something"
                        },
                        {
                            "title": "something3",
                            "url": "/products/devices/something3"
                        },
                        {
                            "title": "something2",
                            "url": "/products/devices/something2"
                        }
                    ],
                    "title": "Flower + Concentrates",
                    "url": "/devices/flower-and-concentrate"
                },
                {
                    "items": [
                        {
                            "title": "something",
                            "url": "/products/devices/something"
                        },
                        {
                            "title": "something3",
                            "url": "/products/devices/something3"
                        },
                        {
                            "title": "something2",
                            "url": "/products/devices/something2"
                        }
                    ],
                    "title": "something",
                    "url": "/devices/something"
                }
            ]
        },
        {
            "title": "CBD",
            "items": [
                {
                    "title": "something",
                    "url": "/products/devices/something"
                },
                {
                    "title": "something3",
                    "url": "/products/devices/something3"
                },
                {
                    "title": "something2",
                    "url": "/products/devices/something2"
                }
            ]
        }
  ...

So far I have:
export const menuItemSchema = z.object({
  url: z.string(),
  title: z.string(),
});

export const menuSchema = z.object({
  url: z.string().optional(),
  title: z.string(),
  items: z.array(menuItemSchema),
});

However, I can't figure out how to make menuSchema recursive to account for multi-level menus. I've read the documentation and don't see anything about this. For context, I am using a CMS to get this data and Zod to ensure all menus abide by this schema. So it should be able to also work on one-level menus.
EDIT:
I've found a piece of documentation for recursion, but it seems like I may have messed up somewhere since you can pass in additional keys where I only want title, items, and url:
const baseMenu = z.object({
  title: z.string().optional(),
  url: z.string().optional(),
});

type Menu = z.infer<typeof baseMenu> & {
  items?: Menu[];
};

const MenuSchema: z.ZodType<Menu> = baseMenu.extend({
  items: z.lazy(() => MenuSchema.array().optional()),
});

console.log(MenuSchema.safeParse({
  data: [
    {
      title: 'Devices',
      items: [
        {
          items: [
            {
              title: 'something',
              url: '/products/devices/something',
            },
            {
              title: 'something3',
              url: '/products/devices/something3',
            },
            {
              title: 'something2',
              url: '/products/devices/something2',
            },
          ],
          title: 'Flower + Concentrates',
          url: '/devices/flower-and-concentrate',
        },
        {
          items: [
            {
              title: 'something',
              url: '/products/devices/something',
            },
            {
              title: 'something3',
              url: '/products/devices/something3',
            },
            {
              title: 'something2',
              url: '/products/devices/something2',
            },
          ],
          title: 'something',
          url: '/devices/something',
        },
      ],
    },
    {
      title: 'CBD',
      items: [
        {
          title: 'something',
          url: '/products/devices/something',
        },
        {
          title: 'something3',
          url: '/products/devices/something3',
        },
        {
          title: 'something2',
          url: '/products/devices/something2',
        },
      ],
    },
  ],
}))

Answer: Looking at the data, I've identified two different types. There is the top level menu item type which doesn't have a URL and then there are menu items which have a url, title, and optionally a recursive list of menu items. I think the place where you're getting stuck is that the top level menu type feels very similar to the menu items, but I think it's better to consider them two distinct types. With that in mind, you can formulate your schemas as follows:
import { z } from "zod";

// This is a bit upside down, but we specify the recursive inner type
// first because we're going to use it with the top level menu schema later.
interface MenuItem {
  title: string;
  items?: MenuItem[];
  url: string;
}

// Recursive types rely on the type already existing, it can't infer
// what the type should be which is why there is an explicit interface
// definition.
const MenuItemSchema: z.ZodType<MenuItem> = z.lazy(() => z.object({
  title: z.string(),
  // Recursive use of the schema
  items: MenuItemSchema.array().optional(),
  url: z.string(),
}));

// A fully separate type for the outermost menu
const TopLevelMenuSchema = z.object({
  title: z.string(),
  items: MenuItemSchema.array(),
});

type TopLevelMenu = z.infer<typeof TopLevelMenuSchema>;

Then you can parse your input data with something like
z.object({ data: TopLevelMenuSchema.array() }).safeParse(data);


System.IO.FileNotFoundException when trying to load DLL

Question: I'm not a a very experienced Windows developer, so I hope this all makes sense.
I created a Managed Assembly DLL using Visual Studio 2010.  The DLL (Plip.dll) contains a C++ class that is using System.IO.SerialPort class to do some simple communication over a serial port.
In a second Visual Studio project I created a simple GUI that uses the class found in Plip.dll.  In my GUI project I have the line : #using "Plip.dll" .  In the Project Properties I set the 'Resolve #using References' value to the correct location of Plip.dll.  The GUI builds just fine.  If I copy the GUI.exe and Plip.dll to the same folder, the GUI runs just fine on my computer.
The problem I am having is that when I copy both files to a second computer, I cannot get the GUI executable to run.  I get the following error : "System.IO.FileNotFoundException.  Could not load file or assembly "Plip.dll" Vesion=.... ".  I get this error even though both the exe and dll are located in the same folder.
Any suggestions on how to resolve this issue?  Is there some option I need to set in my GUI project to load the DLL correctly at run time?
Answer: I suppose the problem is not the Plip.dll, but it's dependencies.
Use Dependency Walker  on the second computer to see if it needs any other dll's (they might be installed in System folder or in %PATH% on your development computer, but not on the other).
If this second computer doesn't have Visual Studio installed, you are probably missing Microsoft Visual C++ 2010 Redistributable Package (you need to install it on the other computer)
Also make sure that you compile in Release because debug builds need debug dependencies.
Comment: You got it exactly right.  The Dependency Walker showed that I was missing the MS C++ 2010 Redistributable.  Thanks for the help.
Answer: I found the answer to this problem to be much simpler than Dependency Walker (but admittedly, that was fun to look at).
In my case, the issue was a mis-match between the .DotNet versions in the DLL and with the application's .net version.  This was caused by building the "class library" using .DotNet 6.0 (dot net core?).
Instead, the entire class needed to be re-built using "Class Library (.NET Framework)"
enter image description here
I wrote an article on this problem.
https://keyliner.blogspot.com/2022/09/visual-studio-c-linked-dll-exception.html


What do different fill (background) colors mean in the MLB schedule?

Question: In the schedule, MLB official site displays games in different fill (background) colors. For instance, the Red Sox vs. Yankees game on July 31 is in grey fill color, while the game on July 7 is in white.
What does the coloring mean?
Answer: You didn't copy the important part of the table, because it's stated below what each color stands for.

To be honest, this isn't entirely true, because the white color is only for completed home games to distinguish between completed home and away games.

Sapphire = upcoming home games
Grey = all away games
White = completed home games

You can as well see this by the annotations in each field (vs. for home and @ for away). It's a perfect match.
Comment: Might be worth noting that the reason home games have two colorations (completed / upcoming) and away games one, is that the main purpose they're hoping people use this schedule for is shopping for tickets.
Comment: Thank you. Does `field` in "by the annotations in each field" mean `baseball field`, like Fenway Park?
Comment: @JJJohn no, I was referring to the table fields (days) of the calendar.


complex integration along closed contour

Question: Let  $I_r=  \int dz/(z(z-1)(z-2))$ along $C_r$,  where $C_r = \{z\in\mathbb C : |z|=r\}$, $r>0$. Then

a.    $I_r= 2\pi i$  if $r\in (2,3)$
b.    $I_r= 1/2$  if $r\in (0,1)$
c.    $I_r= -2\pi i$  if $r\in (1,2)$
d.    $I_r= 0$ if $r>3$.

I am stuck on this problem . Can anyone help me please?
all options are looking wrong by using residue theorem......
Comment: For rє(2,3), if I take r=2.5  etc…………….
All the options are wrong???
Comment: For rє(2,3), if I take r=2.5 etc............ and i used residue theorem but none of the options is correct in my calculaton.......am i right????
Comment: For rє(2,3),can I take r=2.5 etc. or not?????????
Comment: For rє(2,3), if I take r=2.5 etc……… using residue theorem Ir = 2Πi(sum of residue)=2Πi(2-1+2)=6Πi  so option a is wrong am i right??????????
Answer: Define $f(z)=\frac{1}{z(z-1)(z-2)}$ then we have $$\forall r\in(0,1) :\; I_{r}=2\pi iRes[f;0]$$
$$\forall r\in(1,2) :\; I_{r}=2\pi i(Res[f;0]+Res[f,1])$$
$$\forall r\in (2,\infty) :\; I_{r}=2\pi i(Res[f;0]+Res[f,1]+Res[f;2])$$
Now  because $f(z)=\frac{1}{z(z-1)(z-2)}$ is at shape $f(z)=\frac{p(z)}{q(z)}$ when $p(z)=1$ and $q(z)=z(z-1)(z-2)$ that both are analytic in points $z_{0}=0,1,2$ and $p(z_{0})\neq 0$ , $q(z_{0})=0$ and $q'(z_{0})\neq 0$ for these three points. So $f$ has three simple pole there. And so $Res[f;z_{0}]=lim_{z\longrightarrow z_{0}}(z-z_{0})f(z)$ in those points.
$$Res[f;0]=lim_{z\longrightarrow 0}(z)\frac{1}{z(z-1)(z-2)}=\lim_{z\longrightarrow 0}\frac{1}{(z-1)(z-2)}=\frac{1}{2}$$
$$Res[f;1]=lim_{z\longrightarrow 1}(z-1)\frac{1}{z(z-1)(z-2)}=\lim_{z\longrightarrow 1}\frac{1}{z(z-2)}=-1$$
$$Res[f;2]=lim_{z\longrightarrow 2}(z-2)\frac{1}{z(z-1)(z-2)}=\lim_{z\longrightarrow 2}\frac{1}{z(z-1)}=\frac{1}{2}$$
And at the end;
$$\forall r\in(0,1) :\; I_{r}=\pi i$$
$$\forall r\in(1,2) :\; I_{r}=-\pi i$$
$$\forall r\in(2,\infty) :\; I_{r}=0$$
So the last choice "d" is correct choice.


Difference in multiple regression with x=factor/coefficients matrix in R

Question: I have got 2 models. The first model uses a factor as the independent variable. The 2nd model uses a matrix with 0/1 coefficients as the independent variable.
# fit a model where an iris' petal width depends on species
# first up with a categorical variable
summary(lm(Petal.Width ~ 0+Species, data=iris))

# 2nd with a matrix indicating the species
smat = model.matrix(~ 0+Species, data=iris)
summary(lm(Petal.Width ~ 0+smat, data=iris))

The outcome of the two models above is the same. This is what I would expect:
Call:
lm(formula = Petal.Width ~ 0 + smat, data = iris)

Residuals:
   Min     1Q Median     3Q    Max
-0.626 -0.126 -0.026  0.154  0.474

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)
smatSpeciessetosa      0.24600    0.02894    8.50 1.96e-14 ***
smatSpeciesversicolor  1.32600    0.02894   45.82  < 2e-16 ***
smatSpeciesvirginica   2.02600    0.02894   70.00  < 2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.2047 on 147 degrees of freedom
Multiple R-squared:  0.9796,    Adjusted R-squared:  0.9792
F-statistic:  2357 on 3 and 147 DF,  p-value: < 2.2e-16

However, when the model gets more complicated the summary is not the same anymore:
x = c(rep("x1",19),rep("x2",23),rep("x3",8),rep("x4",101),rep("x5",66),rep("x6",26),rep("x7",88))
y = c(0.74, 0.37, -0.34, 0.18, 0.18, -1.2, 0.04, 0.61, 0.85, 0, -0.03,  -0.18, -0.12, 1.37, -0.34, -0.4, -1.71, -0.13, 3.63, 1.6, 0.47,  0.68, 0.14, 0.86, 0.68, 0.07, -0.33, 4.14, 5.3, 0.94, -0.54,  -2.14, 0.12, 0.81, 1.51, -1.53, 3.37, -1.83, -6.99, -4.69, 1.59,  -1.01, -0.09, 0.04, 0.01, 20.35, -0.28, -1.67, -1.08, -5.33,  1.26, 4.35, 0.07, 1.55, 0.65, 0.15, -0.08, -0.31, -1.61, 0.57,  -0.36, 0.47, 0.72, -0.66, 0.92, 0.77, -1.93, -1.06, -0.93, 0.33,  0, 1.23, -1.53, -0.15, -3.5, 1.01, 0.63, 0.98, 1.06, 0.77, 0.33,  0.72, 1.56, 1.81, 0.74, 1.06, 0.09, 0.15, 0.75, 0.17, 0.77, 0.63,  0.54, 0.22, 0.49, -0.81, -0.03, 2.19, 0.76, 0.11, 0.94, -1.7,  -2.11, 0.32, -4.9, 0.39, 1.59, -5.66, 2.08, -10.43, -0.08, -0.9,  -0.16, -0.86, 0.43, -2.37, -2.16, -0.5, -0.6, 0.27, -0.34, 1.41,  -0.42, -1.43, -2.25, -0.69, -0.71, -1.54, -3.12, -0.93, -2.01,  0.26, -0.53, -1.09, -1.45, -2.75, -1.63, 0.16, -0.13, 0.08, -2.93,  -1.46, 0.18, -0.45, -0.88, 0.05, 0.68, -0.62, -2.55, -3.33, -2.35,  -3.56, -0.07, -0.29, 1.81, 1.07, 0.01, 0.85, -0.21, -1.68, 1.13,  1.58, -0.31, 1.14, -0.73, 2.04, 0.82, 0.78, 0.69, 1.18, 3.78,  0.97, 0.7, 13.63, -2.35, 4.52, 7.46, 4.09, 1.36, 0.77, -0.51,  0.09, 3.12, -3.36, -1.04, 2.79, 0.1, 2.83, 1.96, 0.15, -0.09,  0.39, 0.65, -1.7, -0.84, 1.86, -0.6, 0.18, 0.49, -0.66, -0.16,  2.16, 0.46, 1.66, 0.13, 2.12, -0.03, -0.31, 0.13, -0.45, 0.18,  -0.17, 0.25, -0.16, -0.06, 0.01, 0.83, -2.04, 1.06, 0.35, 0.86,  0.8, 2.01, 2.55, 0.07, 0.55, 1.59, -2.79, -2.77, 1.15, 0.46,  0.45, -0.4, 1.54, 0.29, 0.17, -0.36, 0.99, -9.59, 2.6, 1.79,  -0.56, -0.49, 0.28, 1.46, -1.19, -0.48, -2.52, 1.08, -4.1, -2.91,  -4.89, -0.12, -10.36, -4.97, -7.07, -3.81, -4.34, -12.82, -2.51,  -8.47, -6.12, -1.56, -4.48, -2.88, -1.94, -6.01, -1.35, 1.23,  3.08, -2.3, -0.99, 0.63, 2.4, 2.27, -0.79, 1.76, 1.21, -4.43,  -1.34, -1.94, -0.57, -2.77, -3.07, -1.15, -2.28, 0.68, 1.5, -1.78,  0.46, 0.39, -2.04, -1.22, 0.66, 1.22, 2.61, 5.21, 1.17, 2.73,  2.48, -0.95, 1.59, 0.14, -2.62, -1.73, -0.55, -1.25, 0.15, -0.11,  -0.23, -3.47, 1.46, -1.32, 0.24, 1.48, 2.68, 0.07, 0.24, -0.76,  -1.18, -1.57, -4.08, -2.74, 0.03, 0.34, 0.34, 0.84, 1.26, -0.81,  -0.79, -1.59)

with the following output:
> summary(lm(y ~ 0+x))

Call:
lm(formula = y ~ 0 + x)

Residuals:
     Min       1Q   Median       3Q      Max
-11.6615  -1.0202   0.0515   1.1498  18.8562

Coefficients:
    Estimate Std. Error t value Pr(>|t|)
xx1  0.18526    0.59168   0.313    0.754
xx2  0.14000    0.53778   0.260    0.795
xx3  1.49375    0.91184   1.638    0.102
xx4 -0.41149    0.25663  -1.603    0.110
xx5  0.81182    0.31746   2.557    0.011 *
xx6  0.01077    0.50580   0.021    0.983
xx7 -1.15852    0.27493  -4.214 3.26e-05 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.579 on 324 degrees of freedom
Multiple R-squared:  0.08401,   Adjusted R-squared:  0.06422
F-statistic: 4.245 on 7 and 324 DF,  p-value: 0.0001676

> summary(lm(y ~ 0+model.matrix(~x)))

Call:
lm(formula = y ~ 0 + model.matrix(~x))

Residuals:
     Min       1Q   Median       3Q      Max
-11.6615  -1.0202   0.0515   1.1498  18.8562

Coefficients:
                            Estimate Std. Error t value Pr(>|t|)
model.matrix(~x)(Intercept)  0.18526    0.59168   0.313   0.7544
model.matrix(~x)xx2         -0.04526    0.79956  -0.057   0.9549
model.matrix(~x)xx3          1.30849    1.08699   1.204   0.2296
model.matrix(~x)xx4         -0.59675    0.64494  -0.925   0.3555
model.matrix(~x)xx5          0.62656    0.67147   0.933   0.3515
model.matrix(~x)xx6         -0.17449    0.77841  -0.224   0.8228
model.matrix(~x)xx7         -1.34379    0.65244  -2.060   0.0402 *
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.579 on 324 degrees of freedom
Multiple R-squared:  0.08401,   Adjusted R-squared:  0.06422
F-statistic: 4.245 on 7 and 324 DF,  p-value: 0.0001676

Why are the p-values of the coefficients different here?
If it's the order in which the coefficients are fitted, is there a way to get the same behaviour?
Comment: You have, whether you knew it or not, used a different coding scheme for the predictors in the two models, so the coefficients have different interpretations. Also, please try to write questions that don't rely on everyone understanding R code.
Comment: Ok, so the answer is: I forgot that the command `model.matrix(~x)` will fit coefficients *with* intercept, which I didn't want. The right syntax is `model.matrix(~0+x)`, and then the results are the same. (for R: I added 2 lines to explain the model fit)
Comment: I just realized that I made a mistake in the encoding of the coefficients in the last example - will see how this changes the outcome.
Comment: You have the ability to answer your own question. Doing so will show that this question has an answer, and improve the website for future visitors.
Answer: I forgot that the command
model.matrix(~x)

will fit coefficients with intercept, which I didn't want. The right syntax is
model.matrix(~0+x)

and then the results are the same.
So, to sum up, the following 2 models behave the same way:
summary(lm(y ~ 0+x))
summary(lm(y ~ 0+model.matrix(~0+x))) # notice that '0' appears twice

To get the same behaviour with intercept, the following syntax is needed:
summary(lm(y ~ x))
summary(lm(y ~ model.matrix(~x))) # notice that no '0' is in model.matrix()


How to test if a variable is a confounder in a repeated-measures design?

Question: I would like to test is a given variable is associated with both the dependent and the independent variables (and therefore a potential confounder) in a repeated-measures design.
My model has a continuous dependent variable and two independent variables: a within-subject factor with 6 levels and a between-subject factor with 2 levels. I also have a continuous independent variable that is statistically associated with my between-subject factor (tested using a 2 sample t-test).
How should I test the association between this continuous variable and the dependent variable?
(1) adding this variable to my original model and check the effect of this extra variable?
or
(2) fitting a new model with only my within-subject factor and this extra variable (i.e. without the between-subject factor)?
Answer: There is no test for confounding. A pre-treatment variable could be associated both with treatment and with the outcome and it might still not be a confounder; for example, it may be a collider, in which case controlling for it would bias your treatment effect estimate. You cannot use a collapsibility test (i.e., testing whether the treatment effect varies based on whether you control for the variable in question), because controlling for a collider would also change the treatment effect estimate. You need to rely on a causal model to determine whether a variable is a confounder or not, not a statistical test. If the causal model is justified and consistent with the data, causal definitions of confounders can be used to determine if the variable is a confounder.
Answer: I have been reading up on different ways to operationalize the idea of confounding, particularly in my own repeated measures data which has the added twist of being binary outcomes. In my situation, there are three points in time for each person (I believe some people call that a 3-level "within subject" time effect) and two groups (my 2-level between subjects effect). My potential confounder is each person's age, the mean ages differ between my two groups.
What I settled on was a particularly Epidemiological viewpoint on confounding. My effect measure is the difference in outcome between groups, at each point in time, adjusting for age. The technique I found allows me to look at that effect estimate from a model with age being adjusted and then calculate what those estimates would be without adjusting for that possible confounder. The idea is that "confounding" is the degree to which a putative confounder influences my effect measure.
I'm not able to provide a succinct and confident description of the method I used. If yo are interested, you can get this concept of confounding from Section 2.3 of this paper
https://cdn1.sph.harvard.edu/wp-content/uploads/sites/343/2013/03/Confounding_and_collapsibility.pdf
And a method for computing the muA0, muA1, muB0, muB1 components is given in this paper (here's the abstract, at least)
https://www.ncbi.nlm.nih.gov/pubmed/24603316


Pitman probability poisson approximation

Question: On pg118 of Pitman's Probability, a derivation of the Poisson approximation to $P(k)$ (the binomial probability of getting $k$ successes over $n$ trials, where the success probability $p$ is nearly 0 and $n$ is large) is presented in terms of the consecutive odds ratios $R(k) = P(k)/P(k-1)$.  The idea is to calculate $P(k) = P(0)R(1) \cdots R(k-1)R(k)$.  One thing in this presentation that I don't really follow is his approximation for $R(k)$.  He calculates $$R(k) = P(k)/P(k-1) = \frac{n - k + 1}{k}\frac{p}{1-p} = \frac{np}{k}\frac{(1-(k-1)/n)}{1-p} \approx \frac{\mu}{k}.$$  Implicit in this calculation is the approximation $$\frac{(1-(k-1)/n)}{1-p} \approx 1,$$ which I don't understand.  The reason I don't understand how this can always be so is that, in Pitman's discussion, there are no conditions on $k$ other than the obvious $0 \leq k \leq n$.  So if we take $k$ to be nearly $n$, then the numerator in the fraction directly above goes to zero while its denominator goes to 1.  Another, more understandable, derivation was presented in class, but I want to know what the reasoning is behind $\frac{np}{k}\frac{(1-(k-1)/n)}{1-p} \approx \frac{\mu}{k}$ in this particular explanation, because I'm not seeing it.
Comment: This is under the condition of $n\to \infty$ and $p\to 0$. The claim is that for $np=\mu$ with $\mu$ fixed, binomial($n,p$) approximates poisson($\mu$) as $n\to \infty$ and $p\to 0$. For instance binom(10,1/10), binom(100,1/100), binom(1000,1/1000),...have distributions approximating poisson(1).
Answer: The "truth" about this approximation is to actually consider sequences $p_n$ of probabilities such that

$np_n \stackrel{n \to \infty}{\longrightarrow}0$

From this it follows that

$p_n = o\left(\frac{1}{n}\right)$ ($p_n$ goes to $0$ even if you multiply it by $n$)

Then, for any fixed $k$ you have
$$\frac{1-\frac{k-1}{n}}{1-p_{\color{blue}{n}}} = \frac{1-\frac{k-1}{n}}{1-o\left(\frac{1}{n}\right) } \stackrel{n \to \infty}{\longrightarrow}\frac{1-0}{1-0} = 1$$


How can I fix yaml error: Block mapping value not allowed here

Question: I copied Semaphore's config of GoReleaser and saved it as .semaphore/semaphore.yml but when I run it on Semaphore, I can see:
Unprocessable YAML file.
Error: {{:throw, {:yamerl_exception, [{:yamerl_parsing_error, :error, 'Block mapping value not allowed here', 30, 20, :block_mapping_value_not_allowed, :undefined, []}]}}, "# .semaphore/semaphore.yml.\nversion: v1.0\nname: Build\nagent:\n machine:\n type: e1-standard-2\n os_image: ubuntu1804\n\nblocks:\n - name: \"Test\"\n task:\n prologue:\n commands:\n # set go version\n - sem-version go 1.11\n - \"export GOPATH=~/go\"\n - \"export PATH=/home/semaphore/go/bin:$PATH\"\n - checkout\n\n jobs:\n - name: \"Lint\"\n commands:\n - go get ./...\n - go test ./...\n\n# On Semaphore 2.0 deployment and delivery is managed with promotions,\n# which may be automatic or manual and optionally depend on conditions.\npromotions:\n - name: Release\n pipeline_file: goreleaser.yml\n auto_promote_on:\n - result: passed\n branch:\n - \"^refs/tags/v*\"\n"}

How can I fix it?
Answer: In this last part:
promotions:
    - name: Release
       pipeline_file: goreleaser.yml
       auto_promote_on:
         - result: passed
           branch:
             - "^refs/tags/v*"

The indentation of pipeline_file and auto_promote_on is wrong – both need to be at the same level as name.


Html Parsing vs. Regex

Question: I have a fixed well structured html source, incoming data is clear and small, just contains a little list of divs. I know that using a html parser for html parsing but this looks like a particular case and i am not sure which one that i should use. The problem conditions below

Data is clear and well structured
Data is small
Performance matters, application must be able to get as much as data that is possibble
Application will write data to MongoDB database
Implementation programming language will be Scala or Python

Any opinion is valuable so what should I do?
Answer: I would still stick to using an HTML Parser, because, at least, there is a specific data format and a specialized tool that understands the format.
If performance matters here, there is a blazingly fast lxml package. For the HTML, use lxml.html.
You can also use an awesome BeautifulSoup package and let it use lxml parser under-the-hood. Besides, if the data you need to parse is in a specific part of the HTML document, you can have a performance gain by asking BeautifulSoup to parse only the relevant part of the HTML document, see more at: Parsing only part of a document.
And, to follow the tradition for HTML+regex threads, here is the reference to the famous topic covering the reasons why you should not use regex for parsing HTML:

RegEx match open tags except XHTML self-contained tags
Comment: I know that what i shouldn't use regex to html parsing, i know what is regex and what it turns when implemented,yes i got automata lesson too, most of reasons are about html unstable structres and big amount of data, which is not true for our case, we have a well structred and small data to process. so, i appreciate your answer but this is not we are lookng for, i think.
Comment: @HüseyinZengin thanks. It's difficult to say without seeing what kind of data you have, how much of it and what data you need to parse from it. I guess your best bet would be to measure the performance yourself. For example, implement it using `lxml` and `regex`-only approach and benchmark it.

Html Parsing vs. Regex


Where does the Chrome web store "Support" tab come from?

Question: I'm creating a packaged Chrome app for the Chrome web store ... how do I enable a "Support" tab on my app page within the store?

I'm reading the documentation on manifest.json, but I don't see any property that enables this support tab.
Answer: This is a global setting available in your Dashboard:

After clicking:

Word of advice: don't do it and make some other support link available (which you can set in the item's edit page in the dashboard, and will show up on that tab). The support page function will not notify you anyhow about new questions, and the team is reluctant to even attempt to fix it.


FileSystemWatcher and DFSR

Question: I've got a class that watches for new folder being created. The code works well when I copy and past folders manually, but when a folder is replicated via Distributed File system Replication, the filesystemwatcher doesnt detect the new folder... Any ways to fix this?
I'm running on windows server 2008, the windows service is running in .net 4.5.
My plan B is polling the folder every second with a Task. But I'd rather stick with the filesystemwatcher over reinventing the wheel.
profileFolderMonitor = new FileSystemWatcher(path);
profileFolderMonitor.NotifyFilter = NotifyFilters.DirectoryName;
profileFolderMonitor.Error += (sender, args) => Monitor();
profileFolderMonitor.Created += OnFolderCreated;
profileFolderMonitor.EnableRaisingEvents = true;

Answer: I opted for having a simple class doing that keeps state of the currently registered files/foders, has a loop that verifies for any deltas (with a 1 second pause), and updates the state when deltas do occur. I submit FileSystemEventArgs thru event handlers just like the Microsoft FileSystemWatcher this way it integrates well with my code base.
Answer: I have continued to have issues with FileSystemWatcher for years. I ended up making something that merged the FileSystemWatcher with a polling option, so that events the FileSystemWatcher wouldn't pick up for some reason wouldn't be totally lost: https://github.com/phatboyg/Magnum/tree/d73a9be8f53e03132614e5787c6b3bfcc875ed09/src/Magnum/FileSystem.
We would have issues with stuff across the network, sometimes if the share was on a SAN, or just randomly in general the FileSystemWatcher would stop completely. Never did figure it out, thus the above workaround.
Comment: Thanks for your answer Travis, I didn't try your code, since I was running out of time. But I will upvote since theoratically this could replace the FileSystemWatcher.

FileSystemWatcher and DFSR