Equivalent definitions of Markov Decision Process

Question: I'm currently reading through Sutton's Reinforcement Learning where in Chapter 3 the notion of MDP is defined.
What it seems to me the author is saying is that an MDP is completely defined by means of the probability
$p(s_{t+1},r_t | s_t, a_t)$.
However according to wikipedia, and many other references I've found an MDP is a quadruple of set of states, set of actions, transition probability $P_{s_{t+1,s_t}}^{a_t} = p(s_{t+1} | s_t, a_t)$ and an expected reward $R_{s_{t+1},s_t}^{a_t}$.
Now I'm assuming one definition implies the other, unless I'm wrong so please tell me...
The thing is Sutton manages to derives from it's definitions the wikipedia one. However as I can't find the other way around I was trying to show the converse.
The only hunch I had was observing that
$$
R_{s_{t+1},s_t}^{a_t} = \sum_{r_{t+1},s_{t+1}} r_{t+1} p(r_{t+1}, s_{t+1} | s_t, a_t)
$$
and maybe there're conditions that actually would allow me to retrieve $p(r_{t+1},s_{t+1} | s_t, a_t)$ but I cannot honestly see it.
Can you tell me then if the two are equivalent and why?
Answer: I think the name of the question is a bit misleading...
In general in mathematics, the question of 'equivalent definitions' usually arises when there are two formally complete definitions of a thing that we actually expect to be the same thing.
Considering the case presented here I have the following feelings:

Sutton a bad reference because he does not formally define or prove most of the things. That makes this book really confusing to read. Example: They 'prove' the Bellman equation but actually they are missing out / leaving out / ignoring most of the proof which is quite complicated (see this answer of mine). NOTE: This is just my very personal point of view and I surely disagree with some people regading this opinion on this book. So whatever 'definition' Sutton uses, I am pretty sure it is ambiguous,misleading and/or confusing.

The same applies to Wikipedia. Thea say that there is a reward function $R_a(s,s')$ but what is that function supposed to do? How exactly does it integrate with the probability variables? What if the reward is not deterministic but rather a random thing as well (like in one of the very first examples in Reinforcement Learning, the bandit machine)? Do these random variables have a common density that somehow 'splits' or why exactly does wikipedia say that the MDP is completely described by the marginal distributions $p(s_{t+1}|s_t, a_t)$? What happens if we have two different points in time $t,r$ and $a_t = a_r$ and $s_{t+1}=s_{r+1}$ and $s_t = s_r$... is then $p(s_{t+1}|s_t,a_t) = p(s_{r+1}|s_r,a_r)$? This is not at all clear because formally, $S_t$ is a different random variable from $S_r$...

Summarized: We have two definitions, one is confusing and incomplete and the other one is rather thought of as a summary and is also incomplete. Hence, the question on whether or not they are equivalent is hard to answer. However, there is hope. There is one 'complete' definition that allows us to prove two things: The Bellman equation and the fact that is the state and action space are 'nice' then there is a provably best (surprisingly deterministic) policy! The proof for the Bellman equation can be found in the link above and the proof for the surprising fact that there exist a best policy can be found in the only book that I really found so far to be a valuable source of information about RL: Puterman, Markov Decision Processes.
This definition goes as follows:
Definition A Markov Decision process consists of sets $\mathcal{S}, \mathcal{A}, \mathcal{R}$ and a (potentially infinite) set of random variables $(S_t, A_t, R_t)_{t \in \text{time}}$ where $\text{time}$ is either $\mathbb{N}_0 = \{0,1,2,...\}$ or a set of the form $\text{time} = \{0,1,2,...,T\}$ for some $T \in \mathbb{N}$ and such that for all $t \in \text{time}$, the random variables $S_t,A_t,R_t,S_{t-1},A_{t-1},R_{t-1}, ..., S_0, A_0, R_0$ have a common density $f_{S_t,A_t,R_t,S_{t-1},A_{t-1},R_{t-1}, ..., S_0, A_0, R_0}$ and all $S_t$ map into $\mathcal{S}$, all $A_t$ map into the set $\mathcal{A}$ and all $R_t$ map into the set $\mathcal{R}$. We make the following assumptions on all of the common densities:

For every $t, s_{t+1}, r_t, a_t, s_t, r_{t-1}, a_{t-1}, s_{t-1}, ..., r_0, a_0, s_0$
$$p(s_{t+1}, r_{t} ,a_t | s_t, r_{t-1}, a_{t-1}, s_{t-1}, ..., r_{0}, a_0, s_0) = p(s_{t+1}, r_{t}, a_t | s_t)$$
This property is called the Markov property.
Let us denote the density of a random variable $X$ by $f_X$.
Let $t,q \in \text{time}$ and let $s, s' \in S, a \in A, r \in \mathcal{R}$ then

$f_{S_{t+1} | A_t, S_t}(s' | a, s) = f_{S_{q+1} | A_{q}, S_{q}}(s' | a, s)$
$f_{R_{t} | S_{t+1}, A_t, S_t}(r | s', a, s) = f_{R_{q} | S_{q+1}, A_{q}, S_{q}}(r | s', a, s)$
$f_{S_t|A_t}(s|a) = f_{S_{q}|A_{q}}(s|a)$
or rather formulated like this: if $s_{t+1} = s_{q+1}, r_{t} = r_{q}, a_t = a_{q}$ and $s_t = s_{q}$ then
$$p(s_{t+1} | a_t, s_t) = p(s_{q+1} | a_{q}, s_{q})$$
$$p(r_{t} | s_{t+1}, a_t, s_t) = p(r_{q} | s_{q+1}, a_{q}, s_{q})$$
and
$$p(s_t|a_t) = p(s_{q}|a_{q})$$

Notes:

It is not clear that the random variables have a common density, we must assume it because there are random variables without a density (not even talking about common density!).
It is not clear that these random variables are forgetful with respect to the 'past', we need to assume it!
It is not clear that the densities at different times coincide, we must assume it!
Let's say that we have two MDPs $M = (S_t, A_t, R_t)$ and $M' = (S'_t, A'_t, R'_t)$ over the same time index set. Then we say that $M$ and $M'$ are 'equal' if all their finite densities coincide, i.e. if for all $t$,
$$ f_{S_t,A_t,R_t,S_{t-1},A_{t-1},R_{t-1}, ..., S_0, A_0, R_0} = f_{S'_t,A'_t,R'_t,S'_{t-1},A'_{t-1},R'_{t-1}, ..., S'_0, A'_0, R'_0}$$
(as functions). Let's write $p(x)$ for the density $f_X$ in order to make things more simple. By definition and the first assumption, we have
$$\begin{align*}
&p(s_t,a_t,r_t,s_{t-1},a_{t-1},r_{t-1}, ..., s_0, a_0, r_0) \\
&= p(s_t,a_t,r_t|s_{t-1},a_{t-1},r_{t-1}, ..., s_0, a_0, r_0) * p(s_{t-1},a_{t-1},r_{t-1}, ..., s_0, a_0, r_0) \\
&= p(s_t,a_t,r_t|s_{t-1}) * p(s_{t-1},a_{t-1},r_{t-1}, ..., s_0, a_0, r_0)
  \end{align*}
  $$
Decomposing the latter part inductively we see that we can write $p(s_t,a_t,r_t,s_{t-1},a_{t-1},r_{t-1}, ..., s_0, a_0, r_0)$ as product of the densities $p(s_t,a_t,r_t|s_{t-1})$ and $p(s_0)$. By a straightforward computation we see that
$$p(s_{t+1}, r_t, a_t| s_t) = p(r_t|s_{t+1}, a_t, s_t) p(s_{t+1}|a_t,s_t)p(a_t|s_t)$$
By the second assumption, all these densities are the same (independently of the time).
This leads to the following very important insight: Everything we know to describe a MDP up to 'equality' as defined above is the four densities $p(r|s',a,s), p(s'|a,s), p(a|s)$ (where $s'$ takes the role of $s_{t+1}$, $s$ is $s_t$, $a$ is $a_t$ and $r$ is $r_t$) and finally, $p(s_0)$. So in that sense, a MDP is completely described by saying how the situation develops from one step in time to the next (in the sense that if two MDP's have the same way of going from onestep in time to the next then they are equal). Now an interesting question arises: Given these four densities, can we create an MDP (in some sense 'the unique' MDP, because any two different with the same four densities would be 'equal') from them? The answer is yes, but it is not as easy as you might think: See my question on mathoverflow.

Based on the last note, there is one more thing that one should say about this setup: There also is the notion of a Markov Decision Automata (MDA). Everytime you search for MDP, you will definitely stumble upon MDA. People draw some kind of weird graph next to the definition of an MDP but it is totally unclear how these two things fit together. By the last note, the MDP is completely described by four densities (in two senses: given these densities, we can actually create an MDP from them and any two MDPs we create from them are equal) and what an MDA defines is exactly those four densities. That means: Given an MDA which is nothing else than describing the four densities: initial distribution of the state $p(s_0)$, a reward distribution $p(r|s',a,s)$, a state transition distribution $p(s'|s)$ and a policy $p(a|s)$ usually denotes as $\pi(a|s)$ then the MDA gives rise to an MDP.
Note that when people try to optimize the policy they are actually doing something quite complicated: They have fixed densities $p(s_0), p(r|s',a,s), p(s'|s)$ and they start with some policy $\pi(a|s)$. Then they give rise to the MDP with these densities and use it to evaluate the performance of the policy. Then they change the polica to something else $\pi'(a|s)$ and do this construction (giving rise to an MDP from an MDA) again and re-evaluate the performance, etc etc etc.
Sorry for the long answer but I invested a lot of time in order to understand what is actually going on under the hood and the sources that I found left me very very puzzled (e.g. by simply using assumptions without stating them, etc).
Final note: no measure theory neede din order to understand this answer ;-)
APPENDIX:
I was asked what 'common density' means. A random variable (let's say $X$) is a function from a probablity space $\Omega$ into some other space $\mathcal{X}$. Notice that the 'P'-symbol in probability theory is not something 'bogus'/magical but rather has a very clear definition (it is a measure on (the sigma algebra on) $\Omega$). We say that $X$ has a density $f$ iff. the expressions $P[X \in A]$ can be expressed as $\int_A f(x) dx$ (here, dx is a 'natural' measure on $\mathcal{X}$, e.g. the Lebesgue measure if $\mathcal{X}=\mathbb{R}$ and the counting measure if $\mathcal{X}$ is finite) for all sets $A$ (in the sigma algebra on $\mathcal{X}$). If we are given a touple of random variables $(X,Y)$ then we can regard them as one single function into $\mathcal{X} \times \mathcal{Y}$ and if this random variable has a density then we say that $X,Y$ have a 'common density'. Intuitively that means not much more than being able to compute with expressions $p(x,y)$ instead of the clumsy $P[X \in A, Y \in B]$.
Comment: I agree on the book, it's terrible. But it's the only reference I found for Reinforcement learning.
Comment: can you clarify what do you mean with "common density"?
Comment: Attempted to extend the answer... however, by the nature of things, a density is something heavily measure theory related ... :-(
Comment: Can we have a chat? Cause I have some understanding of measure theory.
Comment: @user8469759: Could you create a chat and link it here? I am really busy at work these days... might be that it takes a while for me to respond though :-(


Odd root like function, but with asymptotes

Question: I have the following function:
$\frac{1}{2}\left(-x\right)^{\frac{1}{1+2\cdot d}}+\frac{1}{2}$
Where d  is an integer, ensuring that (-(x+f)) is taken to an odd root. I'm using the function as part of a machine learning algorithm; the key feature I need is the sudden drop in values at a certain point, but instead of the outputs of the function becoming negative as x increases, I need it to approach a horizontal asymptote at 0 and 1.
Is there any function that might satisfy those requirements?
Answer: The closest I could find was inverse tangent:
$tan^{-1}(x)$
Though it hasn't been transformed to fit the original question's needs.


Date formatting: Code or Template?

Question: A theory/best practice question as opposed to a technical one. Should it be up to a module's code to format a timestamp into a date string, or the template that displays it?
E.g. the two ways of doing it (both with the same result):
MYMODULE.module
$variables['date'] = date('m-d-Y', $timestamp);
return theme_render_template('template.tpl.php', $variables);

TEMPLATE_FILE.tpl.php
<?php print $date; ?>

Or:
MYMODULE.module
$variables['date'] = $timestamp;
return theme_render_template('template.tpl.php', $variables);

TEMPLATE_FILE.tpl.php
<?php print date('m-d-Y', $date); ?>

Answer: Instead of date() you should use format_date() instead per Drupal.
https://api.drupal.org/api/drupal/includes%21common.inc/function/format_date/7.x
format_date takes various settings and user config into account when formatting the date, whereas just date is a php function, and thus, will not. You can also pass it a custom date format name.
While you can use format_date() at the template level, usually it is best to do it in a preprocess function and pass it along to the template. If you have a module that is supplying variables to a template, supply them upfront.
Variables should be print-ready for the user, with the option of overriding later (extending or preprocessing at the theme level, etc).

Date formatting: Code or Template?


Why you need a rest here in this score?

Question: Beginner warning :-)
This is the piano sheet music for Elgar's Variation, Nimrod.
Please do note: I am currently learning time and key signatures with my theory teacher, but in looking at the music here I can not understand why this rest is required.
Answer: It prevents the appearance of an overfull measure. Without it, what we see on this staff is a quarter-note chord, a half-note chord, and a quarter-note chord. 1/4 + 1/2 = 3/4, and we're in the time signature 3/4, so the measure should be done already. Then where does that quarter-note chord on the third beat come from?
It comes from another voice, as Todd Wilcox says — a voice that's silent on the second beat. By writing a quarter-rest for that voice on the second beat, and writing it on the staff where it's about to have some notes, we properly set up, in advance, when that chord takes place. Yes, the quarter-note duration already exists on the upper staff, but that's a long long way away. The rest establishes that information locally, on the lower staff, to improve clarity.
Comment: This is the correct answer. Todd is correct about there being multiple voices, but the reason for the rest is to prevent the reader from getting confused and thinking the measure is in 4/4.
Comment: I note with amused irony that the rest is "to improve clarity", yet has first prompted this question, and then so much uncertainty/controversy in the answers and comments.
Comment: @Agargara While this answer may be technically more correct, it's also a bit harder to grasp. Some mention of the opposition of the note stems (i.e. why that's important) would be helpful.
Comment: So are you saying in this answer that the voice that plays the quarter note C and Eb in the left hand on the third beat isn't even notated at all on the first beat? And that the rest therefore "introduces" that voice?
Comment: @ToddWilcox No, more like "introduces it to this staff". I agree with you that it's the same voice that has the stem-down quarter notes. I was just trying to avoid duplicating your explanation. I also didn't expect this answer to be nearly so popular :)
Answer: The score has three voices. We can call them the "lower voice", "middle voice" and "upper voice". The lower voice is completely notated on the bass clef. The upper voice is completely notated on the treble clef. The middle voice switches back and forth. When the middle voice is in the treble clef, its notes have down stems, and vice versa when the middle voice is in the bass clef.
In the first measure, the middle voice plays the G and Bb quarter notes (crotchets) that are in the treble clef with down stems below middle C. The second beat the middle voice plays the rest that you are asking about. For the third beat the middle voice plays the C below middle C and the Eb just above it, notated on the bass clef with up stems.
A bigger question is where is the middle voice for the first two beats of measure two? And I'm a bit wrong because measure three seems to have four voices in it. Perhaps that's the titular enigma.
Edit:
On further consideration, I think there are four voices the whole time, and it's just notated in an unclear manner. I think the lower middle voice plays the Eb below middle C that is the top note of the first chord in the bass clef, then the lower middle voice rests for the second beat, and then the third beat is the C and Eb in the bass clef. The upper middle voice is the G and Bb below middle C in the treble clef for the first beat, then the second and third beats are the middle Cs that are the lowest notes of the treble clef chords. Then they proceed from there.
Comment: @GeneralNuisance In this case, I don't intend the word "voices" to imply singers or human vocalist. A "voice" in this sense is just a distinct musical line.
Comment: 5 voices for bass, maybe? I know it is a little unusual, but if you look at the measure, it starts with a one beat, three note chord. Then there's a two count, three note chord that plays for the rest of the measure, and on the last beat, a two note chord plays. That would require, for the last beat, five notes *just* for bass clef. Ahdunno, I'm no expert in voicing or arranging.
Comment: I'd agree about the voices - almost satb, not quite. However, going along those lines, the first bar shown could be written better, with a rest on beat 1 in 'bass', and a minim rest in 'alto'. otherwise the existing rest really has no more validation than those I just mentioned. Or change direction of some of the stems.
Comment: I like this answer, though I would say there are only two large-sense "voices" for the first two bars, one mostly in the lower staff, and the other mostly in the upper (although individual sub-voices become discernable around the eighth-notes).  The  chord in beat 3 does not introduce a new voice, but actually belongs to the  chord above it; they are part of the same "voice".  The only reason it is notated in the bass clef is to indicate it should be played with the LH, necessitating the quarter-rest in question.
Answer: Though clear, it would be marked wrong in a theory exam.  The correct notation would actually be this.  Each voice technically needs to show a complete bars worth of beats.
Comment: Modern practice, even in choral music where people sing distinct parts, is to write homophonic groups of notes using stems in a single direction rather than splitting them. The first beat is a quarter note in all "voices", so there's no need to write some parts stem up and some part down, and there's no physical "voice" that needs to be silent for that first beat.  If one wanted to write in "theory-correct" notation, I think changing the mid-bar quarter rest to a beat-one half rest (and reversing the stem direction for the beat-one quarter notes) would be clearer than adding your quarter rest.
Comment: Right. Splitting up the first chord to include both voices would also make it technically correct, while probably looking less jarring.
Answer: As others have said, there are multiple voices. Pictures are helpful:
Comment: Common practice allows voice splitting and joining within a measure if all notes that are sounding at a given point of the time start simultaneously and have the same duration.  I'd read m1b1 of the upper staff as having two voices while m1b1 of the lower staff is homophonic (all voices glommed together).  Then m1b2-3 of the upper staff are homophonic while m1b2-3 of the lower staff have two voices.
Comment: @MattPutnam I would see a different voice subdivision. Yours has stems in both directions in the 'lower green' voice. Isn't that a counter-indication to this way of splitting?
Comment: @MarnixKlooster Yes, but that's only because the lower "voice" is not being emphasized as a voice.  Standard stem-direction rules are being applied in the bass except where it "splits" into two voices (notationally, if not functionally speaking).
Answer: Common practice nowadays is to show all notes that are played simultaneously on a staff using a single stem if they start and end at the same time, and if nothing else will happen on that staff during that time.  This is done even in choral music where a staff is shared e.g. between soprano and alto, or between tenor and bass.  Notation is only divided into up-stem and down-stem portions at beats where different notes start and end at different times, or when there are distinguishable voices (e.g. sopranos and altos) and some but not all should refrain from singing on that beat.
In beat 2 of measure 1, bass clef, not everything is supposed to sustain a half note, but something else is supposed to happen during that time (i.e. some quarter notes are supposed to play on beat 3).  Thus, at beat 2 the bass clef splits into two logical voices, one of which plays half notes, and the other of which rests for a quarter note and then plays quarter notes.
Note that on beat 3 of measure 2, the treble clef actually has three voices, with the middle one playing a quarter note while the outer ones play eighths.  The bottom voice is beamed with the eighth note on the second half of beat 3 in the bass clef because doing so avoids having to write an eighth rest for the first half of that beat, and because it makes clear that that note starts at the same time as the quarter note but lasts a different duration.
Had the bass clef not had an eighth note that could be beamed to that note in the treble clef, other ways of notating that measure would be:

Notate the treble clef with homophonic eighth notes including a "d" which is tied to another "d", avoiding the need for quarter notes.  This would avoid crossed stems, but such ties can make music hard to read.
Place the "a" on the same stem as the "e" and "g" above.  This would avoid the need for ties, but at the expense of crossed stems.
Stack the notes as they are, but use a "flagged" eighth note on beat 3, and add an eighth rest for the second half of that beat.

Compared with the above, the use of the cross-staff eighth-note beam was a reasonable "least of evils" judgment.
Comment: This answer is most correct.  This is common notation for piano music, which indicates voicing only when feasible, logically necessary (as here), and/or helpful as a hint to the performer.  There are no discernable "voices" (in a functional sense) in the bass clef as other answers suggest.
Comment: As a performer, I do find the cross-staff stem in b2m3 helpful as an indication of how to resolve that voice in b3m1, apparently to the g in the upper staff, even though I play the f with the LH.  However, I don't know why this answer focuses so much on that stem, which wasn't even part of the question.
Comment: (I meant "beam" not "stem".)
Comment: @Gooseberry: In general, a vertically-stacked group of notes can only be treated as homophonic if they all start and end at the same time and nothing else happens in the mean time.  Applying the same principle to the second measure as the first would imply that there should be an eighth-note rest in the second measure bass clef, analogous to the quarter rest in the first measure, except that cross-stem beaming provides an alternative way to show that the bass clef eighth note in the second measure starts a half-beat after the last quarter note.
Comment: In Barbershop music scores, each of the four voices has a unique combination of staff and direction.  Tenor is treble clef, stems up.  Lead is treble clef, stems down, Baritone is bass clef, stems up.  Bass is bass clef, stems down.  No matter if the voices cross, or how high on the staff they appear.  If for example, the baritone and bass have the same note, it will be notated with both a staff up and a staff down.
Answer: A chord on a staff is executed immediately after the preceding chord, held for its full duration.  If the quarter-rest were absent, the (c e-flat) chord at the end of the 1st bar would be played on the 4th beat, instead of the 3rd, as we would infer from its alignment with the chord in the upper staff.  The quarter-rest explicitly indicates a splitting of the lower staff into two simultaneous "voices", but only in a notational sense.  There is no functional splitting of voices.
At a large scale, I would say the voices act functionally as follows, though individual subvoices become discernable around the eighth-notes:
Comment: The (vertical) alignment of C/Eb at the end of bar 1 can only mean it is played on beat 3. There cannot be beat 4 as the time sig. is 3/4, so I don't see that the rest indicates much. Also, as the C gets played again at the end, with pedal, it could have been written out better anyway.
Comment: Agreed, but that reasoning permits the removal of ordinary rests as well.  For example, suppose the half-note chord were absent.  Then, by that reasoning, we could also omit the quarter-rest, leaving a blank space at beat 2, because the alignment of C/Eb can only mean it is played on beat 3.  This is valid reasoning, and I wouldn't have any trouble playing such a score, but notation rules require us to make an explicit accounting of all beats within a staff, irrespective of the surrounding context.
Comment: I agree that, as a performer, I would probably not even miss the quarter rest if omitted; the score would be perfectly clear without it.  The rest is only present as a general notational rule, just as most English sentences would be perfectly easy to understand without proper punctuation.
Comment: Whilst that is true, then the down stem G and Bb on beat one have no rests for the remainder of that bar. By those rules, the rests ought to be shown, otherwise the stem on G/Bb should be up. Put the 2 A notes into 'alto' position - problem solved !  can't understand why that bar is written as such, it's not in keeping with the subsequent bars. Without punctuation, a lot of sentences could be highly ambiguous...
Comment: @Tim Those rests are not shown because this score indicates "voices" only when notationally/logically necessary.  It does not indicate the continuity of voices as they extend over entire phrases.  The "voice" in beat 1 of the upper staff, is notational, not functional, and ends on the same beat.  This is common in piano music.
Comment: I understand that. Thus the *'alto'*. But following how the rest of the bars shown are written, it appears, to me, that the first bar is written in a different way, and I can't work out why. Just like the OP.
Comment: @Tim I think the main reason the A notes are not written in an alto position is 1) To make it easier to read as a piano score, and 2) It was not important to preserve that particular voicing as a hint to a performer.  I was working on an orchestral reduction last year and found it difficult to preserve very many of the original voicings in piano notation.  Sometimes, the result was very difficult to sight-read.  I think in creating piano arrangements, some notational compromise is always necessary.

Why you need a rest here in this score?


Highlight cells if column contains a given number X times

Question: I have a large Excel spreadsheet with a column that has an identifying number.  Each unique number will occur at least 12 times.  I would like to highlight those numbers that appear more than 12 times.  Is there a way to do this non-programmatically?
Comment: I actually think @Remnant's version is better, you just need to select the whole sheet, then put it in and Excel will handle all of the relative referencing.  It can't be used in the cell proper, but for CF it works great.
Answer: Suppose in column A you have all your numbers. Then to highlight values that occur more than 12 times do as follows:

Format > Conditional Formatting...
In first dropdown select Formula Is
In the adjacent box type <code>=COUNTIF(A:A, A1) > 12</code>
Now select the desired color to highlight

Does this help?
Comment: @dzilla, start with the 'New Rule' menu item.
Comment: i'm using Excel 2007 and under conditional formatting i don't see an option for 'formula is'.  something that i'm missing?
Answer: Use Conditional Formatting:
<code>=COUNTIF(INDIRECT(ADDRESS(1,COLUMN())&":"&
         ADDRESS(65536,COLUMN())),INDIRECT(ADDRESS(ROW(), COLUMN())))>=12
</code>
Select the entire sheet, then choose Conditional Formatting.  Apply this formula, and pick the formatting you'd like.  It will work for any cell, in any column.


Strange ports on default install of W7

Question: I have a base new install of windows 7, and when I went to look for something else I saw the attached netstat output.
What concerns me is that this is Windows + Truecrypt + drivers, nothing else installed.
The sequential high ranged ports belonging to several different seemingly not out of place services seemed odd.
So I torched the install, used Active@ to scrub the disk, re-downloaded the ISO from MSDN, and did a fresh reinstall, viola, they are there again.
It just seems out of place, I have seen a many netstats over the years, this one just strikes me as odd, so I started thinking rootkit? (JUst FYI, when I reloaded I named the machine "Error" so that is why the task manager reads the computer name as such.)
So I would like to know if anyone else could explain it, and therefore is may be normal, or would they be worried as well, and should I start considering I have some very strange thing occuring on my network?
Comment: What services are actually in the svchosts in question? The Services tab should tell you what PID is hosting what service (assuming something isn't causing it to lie to you)
Comment: Multiple, for instance one reports to be "Security Center, TCP/IP Netbios Helper, Windows event Log, DHCP Client, and Windows Audio" (Port 49152) the next is "Windows Update, WMI, Shell Hardware Detection, Task Scheduler, and User Profile Service" (Port 49153)

Then "Services.exe" (Port 49155) does not show in the PID list of the running services at all.

All signs to me raise red flags of why the sequential listening ports in the high numbers, and why sequential between processes? But where could it be coming from if I am infected?
Answer: In the end this ended up being the fact that I did not know (too many years buried in code, less in the guts of my OS) that the ephemeral port range had been changed from > 1024 to start at 49152 as confirmed by...
<code>netsh int ipv4 show dynamicportrange tcp

Protocol tcp Dynamic Port Range
---------------------------------
Start Port      : 49152
Number of Ports : 16384
</code>
I confirmed all executables listening were correct versions, and the alarm was false, just a misunderstanding and not anticipating them being so high.

Strange ports on default install of W7


real part of a holomorphic function from a PDE

Question: I have some problem that I can't figure out myself. Hope that someone can help me out. The problem is: Let $f : U \to \mathbb{R}$ be some real function on a simply-connected domain $U \subset \mathbb{C}^{n}$ with $\partial\bar{\partial}f = 0$ on $U$. Then $f$ is the real part of a holomorphic function on $U$.
The question is: Is this holomorphic function unique? If not what choices are there to choose such a function? Are there any choices? How does the proof go?
I hope that someone has some answers for me and I will be very happy for a lot of answers. I have to apologize if this question is too trivial for some of you :).
Best regards
philippe
Answer: Suppose $g_1, g_2 : U \to \mathbb{C}$ are holomorphic functions such that $\Re(g_1) = \Re(g_2) = f$. Then $\Re(g_1 - g_2) = 0$, that is $g_1 - g_2 = 0 + iv(x, y)$; that is, $g_1 - g_2$ is a purely imaginary function. Furthermore, as both $g_1$ and $g_2$ are holomorphic, so is $g_1 - g_2$. Hence we can apply the Cauchy-Riemann equations which tell us that $v(x, y)$ is locally constant.
In summary, if $U$ is connected, any two holomorphic functions with real part $f$ differ by an imaginary constant.
Comment: There is nothing wrong with your *uniqueness* argument. @Yemon remarked that one needs additional assumption to obtain existence, but apparently the OP did not ask about existence.
Comment: @LVK is right - sorry for the unclear and somewhat muddled comment. As you point out: the question's assumption gives us a $g$ and asks about uniqueness; I was thinking of the variant of the problem where one is given a function and asked to find a harmonic conjugate.
Comment: Existence argument can be found [here](http://math.stackexchange.com/a/107727/31373).
Comment: Connected is not enough -- think of an annulus in the plane
Comment: @YemonChoi: Could you be a bit more specific? I can't see what is wrong with the argument above.
Comment: I would  be very thankfull if you could show me the existence arguement :).
Comment: @LVK: Thanks for clarifying.


Finding changepoints in the movement of a car with respect to a lead car (using ecp package in R)

Question: Main Question:
How do I apply e.divisive method from ecp package effectively on multivariate time-series, so that the resulting segments and changepoints are consistent with domain knowledge and the process is efficient?
Background
I have data of cars moving in a lane. In the car-following literature, the movement of a subject car relative to its lead car in the same lane can be divided into at-least 3 distinct states:
Free-driving:
Distance between the two cars is large, so the subject car (for which the data is collected) either accelerates or maintains speed. So, the position of the gas pedal is expected to be fully or partially pressed and maintained for some time.
Approaching:
Speed difference between subject car and lead car is quite large because subject car was moving at higher speed. But due to decrease in distance, the subject car reduces speed. So, it is expected that driver in subject car completely steps off the gas pedal and in some cases follow by pressing brake pedal.
Following:
Subject car reduces speed until the speed difference becomes zero and the distance reduces to the distance that subject car wants to keep while closely following lead car.  So, it is expected that we see slight up and down movements in the position of gas pedal in subject car because driver tries to maintain the current speed equal to lead car's speed.
Example:
Here is one example that illustrates 2 states:

I am showing plots for 5 variables for a subject car called "Cars_02". The x-axis contains the time frames (60 frames in 1 second), and y axis contains the given variable. On each plot I also manually put vertical lines where I think the Approaching and Following states start.  For keeping this question medium sized, I'm skipping what happened before these 2 states.
You can see that Approaching starts when the driver steps off the gas pedal (see gas pedal position, plot # 2) which causes a decrease in speed (see plot # 1). Afterward, Following starts when speed difference becomes zero (plot # 4) and distance between the car gets shorter and remains consistent for some time (plot # 5). Also, the gas pedal position has up and down movements which shows that driver tried to maintain the speed.
What I want to do?
I want to divide these time-series into segments where each segment represents a car-following state (basically automating my manual effort to identify pattern in multiple variables and detecting where the state changed). This is my first time using ecp so I have several questions:

Is using ecp going to be useful in this case?
Should I normalize all variables first?
Should I smooth all variables as they have tiny undulations?
How can I make e.divisive run faster? The above
time series database is about 6000 data points. And I have 30 other
time series.

What I have tried?
I tried using e.divisive on a sliced gas pedal position variable only (first 2000 data points). I kept the permutations parameter R very small because otherwise it takes very, very long on my 12 GB RAM laptop.
library(ecp)
Xnorm <- as.matrix(df$acc_pedal_pos)
output1 <- e.divisive(Xnorm, R = 20, alpha = 2)
ts.plot(Xnorm,ylab='Value',main='Change points')
abline(v=output1$estimates[c(-1,-21)],col='red',lty=2)

You can see that the small steps in the gas pedal positions cause detection of multiple changepoints, however, they should actually be part of the same behavior: `accelerating to reach a desired speed in Free-driving'.
So, what is the most effective way to use ecp for my problem?
Answer: Due to the length of the data and the gradual change you are expecting (and have seen with the final plot) i'm not sure that ecp is the best approach to use.
Instead you may want to fit a changepoint model which allows for slopes and large datasets.  I suggest you give the EnvCpt package a try.  The primary focus of the package is for model selection and so the main function fits 8 different models to allow the user to choose the most appropriate.
Having said this you can use the underlying functions to fit a specific model form, such as the trend you see in your plots.  Use the EnvCpt:::cpt.reg function to do this.  In contrast to ecp this means making model assumptions but it looks like it might give you the segmentation you want.
Comment: Thanks for your answer. Could you please share any links to blogs/slides that used EnvCpt. I could only find the reference manual on CRAN. I am searching but Google is not much helpful so far. Also, if you've used it yourself can you share your own examples? I'd really appreciate that.


Varnish 4.0 vcl for Drupal 7

Question: Drupal7, Varnish 4.0, Ubuntu 12.4
Varnish is installed an working, but returning intermittent 503, I think whenever Varnish goes to the backend.  Even though the request returns 503, a page refresh may return 200, HIT from cache.  Another refresh may be 200 or 503.  I think a subsequent 503 is a HIT-FOR-PASS.
We've opened up the timeouts, as follows:
vcl 4.0;
backend default {
.host = "127.0.0.1";
.port = "8080";
.probe = {
.url = "/ping";
.timeout = 600s;
.interval = 600s;
.window = 5;
.threshold = 2;
}
.first_byte_timeout = 900s; # How long to wait before we receive a first byte from our backend?
.connect_timeout = 500s; # How long to wait for a backend connection?
.between_bytes_timeout = 200s; # How long to wait between bytes received from our backend?
}

So I suppose it's not that Varnish is timing out in a blink of the eye.
Probably the problem is in the vcl; this is what I'm using:
https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
Here are fragments from the varnish log for 3 successive calls to the same page: 503, then 200, then 503 again:
First call - returns 503
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   FetchError     no backend connection
-   Timestamp      Beresp: 1408509130.524000 0.000219 0.000219
-   Timestamp      Error: 1408509130.524013 0.000232 0.000013
-   BerespProtocol HTTP/1.1
-   BerespStatus   503
-   BerespReason   Service Unavailable
-   BerespReason   Backend fetch failed
-   BerespHeader   Date: Wed, 20 Aug 2014 04:32:10 GMT
-   BerespHeader   Server: Varnish
-   VCL_call       BACKEND_ERROR
-   BerespHeader   Content-Type: text/html; charset=utf-8
-   BerespHeader   Retry-After: 5

Page refresh - Returns 200 from the cache
   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   Hit            2147745802
-   VCL_call       HIT
-   VCL_return     deliver
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
Another refresh - returns 503 again.

VCL_call       HASH
VCL_return     lookup
Debug          "XXXX HIT-FOR-PASS"
HitPass        2147811368
VCL_call       PASS
VCL_return     fetch
Link           bereq 196777 pass
Timestamp      Fetch: 1408509419.224633 0.000666 0.000666
RespProtocol   HTTP/1.1
RespStatus     503
RespReason     Backend fetch failed

Any clues on getting the vcl correct appreciated.
Answer: The VCL you have linked to and your example in the question show you only using one backend. In this case using the probe to do a health check serves no purpose. As per drifteaur's answer disabling probe resolves many 503 issues and as you have no need for it, problem is solved.
However, if you are testing working towards rolling out on multiple backends then it still needs to be fixed(understood).
Looking at the VCL you linked (https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl) it uses many sections which have been taken from https://www.varnish-software.com/ so we can presume (hopefully) that they are correct.
Currently, your probe is checking the response from "/ping" at your host and port, 127.0.0.1:8080, so that is 127.0.0.1:8080/ping .
If your set-up is multiple virtualhosts on one server with varnish sitting infront then using probe like this will not get a response(as this full url has most probably not been configured) and in turn this will lead to the backend being marked as "sick / unhealthy" and a 503 will be produced as Varnish will not even attempt to access the backend.
To confirm that it is your server setup add the following to your probe
backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .probe = {
              .request =
              "HEAD / HTTP/1.1"
              "Host: www.EXAMPLE.com"
              "Connection: close";
              }
}

You will need to change www.EXAMPLE.com to a real domain you are using on the server.
If this gives you 200s rather than 503s than it is safe to confirm that using probe as you were with your server set-up was incompatible.
Probes info: https://www.varnish-cache.org/docs/trunk/reference/vcl.html#reference-vcl-probes
Answer: I had the same problem (503) with the same example config and Drupal. I resolved it by commenting out the .probe part of the backend configuration. Not sure why that helped though.

Varnish 4.0 vcl for Drupal 7


Which partition is my OS installed on?

Question: I want to know which partition OS installed on my server.
Below is the fdisk and df command outputs from my server.
[root@ ~]# fdisk -l
Disk /dev/sda: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        1058     8393962+  82  Linux swap / Solaris
/dev/sda3            1059        7457    51399967+  83  Linux

[root@ ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              48G   12G   34G  26% /
/dev/sda1              99M   12M   82M  13% /boot
tmpfs                 7.9G     0  7.9G   0% /dev/shm

I noticed that sda1 has an asterisk on the Boot field. Does it signify that the OS is installed in that partition?
I also came upon this article, but I just want to confirm if the OS is installed on sda1.
Is there any linux command to run or files to open to see which partition the OS is installed on?
Answer: Well technically speaking, in your case, it's installed on two partitions - sda1 and sda3. sda1 is mounted as /boot. This is where the kernel is held. sda3 is the root partition, where the rest of the OS is installed.
Comment: Thanks @EEAA. That is why I cannot unmount sda3. I was supposed to resize that partition. Anyways that would be for another question. Thanks.
Comment: If you consider swap space as part of the OS (I do) then technically it's on *three* partitions (the swap space is on `/dev/sda3`, and while you *can* run a Linux system without swap this one is probably configured to expect it and might do Bad Things if it were not present).  Your OS *files* though are on the partitions mentioned above.
Answer: As an alternate (lower level) way of looking at things from EEAA's answer, your OS is installed on /dev/sda -- all three partitions you've listed are required for proper functioning of your system.

/dev/sda3 is / - the root filesystem.
All the stuff you deal with when using your system is located on this partition.
/dev/sda1 is /boot - as the name implies this is where Linux puts "bootup stuff"
This includes GRUB configurations, kernel modules, etc.
/dev/sda2 is your [SWAP] partition.
Linux uses this as a place to shove program data if you use up all your RAM.
As your system is apparently configured to expect a swap partition you need this to boot, otherwise when Linux tries to activate swap it will throw an error and ask you to fix it on the console.

NOT SHOWN: The boot block (the really tiny bit of code that gets things going after BIOS is done with its power-on tasks).
The boot block get put in the first disk block of the boot drive (almost certainly /dev/sda in your case), and is really "outside" of the partition layout that you see with df or fdisk.

Regarding the link you posted, for more information on partitioning, see this question and its related answers, you may find it interesting.
Comment: Sorry for editing your answer, but you misplaced /dev/sda1 and /dev/sda3, and this in fact completely reversed your answer's meaning.
Answer: From what i understand -
Your OS is installed on /dev/sda3 as you can see it is mounted on /
The system will boot from sda1
Comment: The system will require the files in `/boot` (little things like the GRUB configuration and the kernel) in order to actually boot...

Which partition is my OS installed on?


In some sense, is linear regression an estimate of an estimate of an estimate?

Question: Consider the problem of estimating a random variable $Y$ using another random variable $X$.
The best estimator of $Y$ by a function of $X$ is the conditional expectation $E[Y|X]$. It minimizes the mean-squared error $E[(Y - h(X))^2]$ over all $L^2$-functions $h(X)$ of $X$.
However, $E[Y|X]$ seldom has a nice closed expression. A further approximation is to consider the best linear unbiased estimator
$$ linear[Y|X] = \mu_Y + \rho \frac{\sigma_Y}{\sigma_X} (X - \mu_X) $$
However, in practice we don't even know the population variances and population correlation. So we use linear regression techniques such as the least-squared solution to get an estimate from a random sample.
In this sense, can we say that linear regression is an estimate of an estimate ($linear[Y|X]$) of an estimate ($E[Y|X]$) of the original random variable ($Y$)?
Answer: To some extent, you had some very good point.  The biggest problem in your interpretation is that you confused the concepts of approximation and estimation.
By probability theory, there exists a Borel function $f: \mathbb{R} \to \mathbb{R}$ such that $E[Y|X] = f(X)$ (almost surely).  As you stated, for general distribution of $(X, Y)$, $f$ seldom has a nice closed form. On the other hand, suppose by some means, we have collected $n$ observations $S = \{(x_1, y_1), \ldots, (x_n, y_n)\}$ from the underlying distribution $(X, Y)$. A natural problem in statistics is then: can we use the sample $S$ to make some inference on the unknown $f$?  Note that, while it is standard to say "estimate the functional form of $f$ using $S$", it is conceptually incorrect to say "estimate the random variable $Y$ using another random variable $X$", for the following two reasons:

No random variable $Y$ can be "estimated" by another random variable $X$.  This has been refuted in Tim's answer. I just want to add that, if you recall that a random variable is essentially a real-valued function, does it make much sense to say use one function to "estimate" another function?  From probability perspective, the statement "$E(Y|X)$ minimizes the mean-squared error $E[(Y - h(X))^2]$ over all $L^2$-functions $h(X)$ of $X$" is good enough but needs some correction as well: the "mean-squared error" has to be the "conditional mean-squared error" $E[(Y - h(X))^2|X]$. Do not use "estimator" for $E(Y|X)$ because $Y$ is not a valid estimand from statistics perspective (also see elaborations below).
In statistics (at least in frequentist statistical inference), the terminology "estimation" specifically means using an observed sample to draw some information on some unknown, yet non-random quantities (called "parameters") of an underlying population (or equivalently, distribution). From this perspective, your misuse of the word "estimate" is obvious: throughout your question, there is only one place you mentioned "sample": "So we use linear regression techniques such as the least-squared solution to get an estimate from a random sample." To be fair, this is the only place that you used the word "estimate" correctly, whereas "estimate/estimator" appeared in other places do not align with their standard statistical usages.

The more appropriate word for the problem you described is "approximation" (you actually also mentioned this term once but for the most time confused it with "estimation"): since $E[Y|X] = f(X)$ in general does not have an analytical form (i.e., "probability deduction" failed to work here), we need to turn to the help of statistical inference.  But in order to get the statistics machine running, the first question we need to face is: what statistical tools should we use?  Parametric inference or non-parametric inference?  It turns out that the linear model is the simplest parametric inference weapon that practitioners like to use, which means, you specify $f(X)$ (it may well be a completely wrong specification, but the advantage is its simplicity and interpretability) as a linear function of $X$, i.e., $f(X) = \alpha + \beta X$, and then go head and use $S$ to estimate the parameters $\alpha$ and $\beta$.  The procedure of specifying the unknown functional form $f$ as a linear function $\alpha + \beta X$ with just two unknown parameters is approximation (or model specification), it is not estimation, which is actually the next step that follows model specification. It is clear that while estimation cannot be done without sample/data, approximation in principle can be done without data (because it is just about selecting a simpler function to proxy a complicated function).  However, to make a satisfactory approximation (i.e., build a decent model) requires the guidance of data as well and is usually interweaved with estimation in an iterative style.  In this sense, "approximation" and "estimation" are closely related.
It is worth mentioning that when the joint distribution of $(X, Y)$ is bivariate Gaussian, then approximating $f$ by $\alpha + \beta X$ becomes exact.  However, this doesn't make $\alpha + \beta X$ the best linear unbiased estimator of $f(X)$ when $(X, Y)$ is non-Gaussian. The "best linear unbiased estimator" refers to the estimator $(\hat{\alpha}, \hat{\beta})$ that minimizes variance after you have approximated $f(X)$ by $\alpha + \beta X$. It is well known that when the error distribution is spherical, the best linear unbiased estimator is the ordinary least-squares estimator.
Finally, let me quote the opening remark of Chapter 5 in The Elements of Statistical Learning to consolidate the point made above. If you want to get a better, more realistic approximation to $f$ than linear model, you can start looking into this chapter too.

We have already made use of models linear in the input features, both for
regression and classification. Linear regression, linear discriminant analysis,
logistic regression and separating hyperplanes all rely on a linear model.
It is extremely unlikely that the true function $f(X)$ is actually linear in
$X$. In regression problems, $f(X) = E(Y |X)$ will typically be nonlinear and
nonadditive in $X$, and representing $f(X)$ by a linear model is usually a convenient, and sometimes a necessary, approximation. Convenient because a
linear model is easy to interpret, and is the first-order Taylor approximation
to $f(X)$.
Answer: It's not.

Consider the problem of estimating a random variable $Y$ using another random variable $X$.

We don't estimate random variables, but things about random variables. What would it mean to "estimate the random variable"? If I told you that I "estimated that the length of life is 70 years" what would it mean? Everybody would live exactly 70 years? It's maximum, minimum, average, mode, or median..? The statement would be quite meaningless. Also, why would you "estimate" a random variable with a single value when you can estimate its distribution (e.g. with empirical distribution or kernel density)?

The best estimator of $Y$ by a function of $X$ is the conditional expectation $E[Y|X]$. [...]

$E[Y]$ or $E[Y|X]$ are not estimators, but properties of random variables. The estimator is a function of a sample, the expected value is a property of a random variable.

However, $E[Y|X]$ seldom has a nice closed expression. [...] So we use linear regression techniques such as the least-squared solution to get an estimate from a random sample.

One has nothing to do with the other. To find $E[Y]$ you need to solve an integral. $E[Y]$ applies to mathematical objects (random variables). Linear regression needs data, if I asked you to tell me what is the expected value given that you know the probability density function of the variable, you wouldn't be able to calculate linear regression on the given function. There wouldn't be a closed-form solution either. The opposite is also true, you cannot calculate the expected value from the data, but you can use some estimator to approximate it.
Answer: A frequentist would use the term "prediction" for what you call "estimation of a random variable". Furthermore, if you don't assume a linear relationship but just use this as approximation, using the term "estmation" for "approximation" is confusing. And even then what is "estimated" is the coefficients of the regression, not the prediction of $Y$. So you're estimating the coefficients of an approximation in order to predict $Y$.
Things are not so clear in Bayesian analysis where parameters to be estimated are also random variables. There $E[Y|X]$ may be legitimately called "estimator" (even though some would probably still see this as confusing use of terminology), and you may estimate involved regression parameters on top of it. I don't see any way how to justify the term "estimator" for the linear approximation though in a Bayesian framework.


How to setup ContentType for Azure ServiceBus from Spring JMS

Question: I'm trying to use library azure-servicebus-jms-spring-boot-starter to send messages to topic. Everything works, however messages are being stored in subscriptions as application/xml type and I can't find the way how to setup this correctly to have them stored as application/json.
I've tried to configure message converter to send ContentType as described here but that doesn't work either.
  @Bean
  public MessageConverter jacksonJmsMessageConverter() {
    final MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter(){
      @Override
      protected TextMessage mapToTextMessage(Object object, Session session, ObjectWriter objectWriter)
          throws JMSException, IOException {
        final TextMessage message = super.mapToTextMessage(object, session, objectWriter);
        message.setStringProperty("ContentType", "application/json");
        return message;
      }
    };
    converter.setTargetType(MessageType.TEXT);
    converter.setTypeIdPropertyName("_type");
    converter.setObjectMapper(objectMapper);
    return converter;
  }

Comment: Content-Type needs to be stored as a System Property for SB to properly recognize it. The setStringProperty() seems to just set a User Property. Question is: how can do write a system property in JMS. Didn't find something quickly
Comment: Try with `content_type` instead.
Comment: @SergVasylchak [here](https://github.com/Azure/azure-sdk-for-java/issues/18852#issuecomment-773922436)
Comment: content_type doesn't work either :/
Comment: @bilak How did you set ContentType. Are you able to get it working?
Comment: @SathishMurugesan see [here](https://github.com/Azure/azure-sdk-for-java/issues/18852#issuecomment-773922436)
Comment: @bilak did you ever make it work? I've only come up with a custom byte array message converter for my specific case so far, but it doen't really makes me happy.
Comment: Thanks @bilak, pretty much the same I've come up with. Looks like there is no way to send json text messages with correct headers.
Answer: There is no exposed means of setting the content-type on the messages sent from the Qpid JMS client.  The client itself uses this field as part of the JMS mapping to AMQP to distinguish certain message types that it sends and to determine at receive time what certain messages should be presented as.
It is technically possible to use reflection to reach in and so the value but the APIs you have to use from the JmsMessageFacade class are not public and could change with any release so choosing to do so comes with significant risk.
Comment: well, you are [not right](https://github.com/Azure/azure-sdk-for-java/issues/18852#issuecomment-773922436)


Quando uma lista ordenada deve ser, de fato, uma lista ordenada no HTML?

Question: No HTML existem dois elementos básicos para se definir uma lista:

, que representa uma lista ordenada;

, que representa uma lista desordenada;

Mas quando, de fato, faz sentido representar uma lista ordenada/desordenada com tais elementos?
Por exemplo, a lista de perguntas na página inicial do Stack Overflow fica organizada de forma ordenada com base na data da última alteração, da mais recente para a mais antiga. Ao meu ver, isso representa uma lista ordenada, pois inverter a posição de diferentes perguntas dentro da lista quebraria a condição de ordem. Avaliando o código fonte da página, percebi que o site utiliza
 para representar a estrutura, sem nenhum atributo que auxiliasse na definição da ordem.
Analisando a estrutura apenas pelo código é impossível definir que as perguntas possuem tal ordenação, diferente se fossem estruturadas dentro de um elemento
, onde deixaria explicito que a ordenação existe.
Além deste exemplo, poderia citar muitos outros: lista de publicações em um blog, lista de produtos em uma loja virtual - que poderia estar ordenada por preço, popularidade, vendas, etc. Dentre todos, é difícil ver uma página que define tais estruturas com base nos elementos
 ou
.

Existe algum motivo que faça ser inviável criar tal estrutura?
Utilizar a
 para representar os elementos não quebra a semântica, visto que não transpassa a ideia de ordenação?
Comment: Ao meu ver, quase todos os dados dinâmicos serão mostrados como tabela ou como lista, na segunda opção geralmente se usa `div`, mas também pode se usar `section` ou `article` que são mais específicas. O não uso de listas deve-se, provavelmente, as facilidades de uma div (não ter que retirar o marcador, margens internas e externas)
Answer: UL vs OL
UL
UL é um conjunto de elementos onde a ordem não importar para o entendimento do conteúdo, ou seja, independente da ordenação desses elementos o usuário sempre será capaz de entende-lo. Ex. Imagina a NetShoes onde vc pode ordenar uma lista de itens por valor, cor ou marca. A ordem desses itens não interfere no entendimento do conteúdo, então para esse tipo de conteúdo uma UL (lista "desordenada") seria o mais indicado. Acredito que qualquer tipo de lista que tenha um filtro, ou seja, uma lista de elementos que possa ser rearranjada de acordo com o que o usuário deseja, deva ser tratada como um UL.

Por exemplo, a lista de perguntas na página inicial do Stack Overflow
fica organizada de forma ordenada com base na data da última
alteração, da mais recente para a mais antiga. Ao meu ver, isso
representa uma lista ordenada...

A lista de perguntas do StackOverflow seria o caso de uma lista não ordenada, uma UL, pois é uma lista onde a ordem não é rígida, inclusive vc não pode ter uma lista Ordenada que cada hora o usuário pode fazer um filtro mudando a ordem ao seu gosto, por exemplo de Data para Ativas ou Sem Resposta. Logo as respostas não precisam respeitar uma ordem para o entendimento, e cada Pergunta pode ser vista como um elemento único, como os Tênis no caso da NetShoes, são elementos do mesmo grupo, mas não necessariamente precisam de uma ordem.

OL
OL é uma lista ordenada, a ordem do conteúdo dessa lista está diretamente ligado a forma como o usuário vai entender a mensagem, e alterar a ordem dessa lista pode afetar diretamente no entendimento do conteúdo.
Nesse tipo de lista os elementos têm uma ordem fixa que não deve ser ser alterada, e geralmente não apresentam filtros ou outros artifícios para reordenação do conteúdo. Ex. o Índice de um Livro ou a Receita de um Bolo, a ordem de preparo do Bolo por exemplo é uma lista de tarefes que deve ser cumprida a risca. Assim como alterar a ordem dos capítulos de um Livro pode deixar o usuário completamente perdido na história. Então para esse tipo de conteúdo a Lista Ordenada OL é o mais indicado, pois exite uma linearidade regida na ordem do conteúdo e dos elementos.

Visto a imagem percebe-se que todos os passos do modo de preparo pertencem a um mesmo grupo, porém esses elementos precisam estar em uma determinada ordem para fazerem sentido para o usuário. Pensando justamente na listagem de perguntas da StackOverflow a OL não seria o ideal, pois o conteúdo independe da ordem para fazer sentido, além disso é uma lista que o próprio usuário pode determinar a ordenação do conteúdo, o que não justificaria o uso de uma OL.
A lado Semântico da história:
Agora que já ficou claro o que é e quando deve ser usado cada tipo de listagem pense na semântica das coisas. Imagine um leitores de tela de um usuário deficiente visual, ou um bot de mecanismo de busca como o Google. Essas duas personas não usam a visão para interpretar o conteúdo que estão acessando. Um listagem com um conjunto de tenis pode ser indexada em qualquer ordem. Já um listagem com o passo a passo de uma determinada tarefa deve ser seguida em uma determinada ordem imutável. Logo se vc declara um procedimento de cadastro usando uma UL, certamente o crawler de pesquisa ou o usuário deficiente visual teriam dificuldades para seguir a ordem correta das coisas, semanticamente nesse caso a OL faria mais sentido para um software que lê o código da página para o usuário.
Essa lista tem que seguir uma ordem imutável:

Bata as claras em neve e reserve

Misture as gemas, a margarina e o açúcar até obter uma massa homogênea

Acrescente o leite e a farinha de trigo aos poucos, sem parar de bater

Por último, adicione as claras em neve e o fermento

Despeje a massa em uma forma grande de furo central untada e enfarinhada

Essa lista não necessita de ser ordenada, e inclusive pode ser reordenada sem prejudicar o entendimento do conteúdo.

Tênis Kappa Impact Masculino

Tênis Mizuno Jet 2 N Masculino

Tênis Asics Patriot 8 Masculino

Tênis Olympikus Spirit 2 Masculino

O estilo padrão do CSS dos diferentes tipo de listagem ajudam o usuário a compreender o tipo de conteúdo que está acessando. Uma lista não ordenada não precisa de numeração para ordenar os elementos do conjunto, pois essa ordem pouco importa para o usuário. Já uma lista ordenada é numerada, justamente para auxiliar o usuário a entender que ali existe uma ordem semântica das coisas, e os elementos devem seguir uma determinada ordem.

Grid View vs List View

Veja que vc sempre deve levar em consideração o conteúdo que sera apresentado para o usuário e como esse conteúdo deve ser tratado para oferecer a melhor experiencia de uso para quem acessa a informação. Um exemplo disso é por exemplo formatar o conteúdo em um List View ou em um Grid View. Acredito que é aqui que a maioria das pessoas escolhem entre usar uma estrutura de Lista semântica UL/OL ou um sistema de Grid com Divs, A vantagem de se usar Divs é que vc não precisa remover o css padrão do user-agente das listas (bullets e números), no entanto vc perde o valor semântico do conjunto de elementos que pertencem a mesma lista.
Se vc vai listar os Tênis da NetShoes para o cliente o Grid View vai ser uma experiencia mais agradável para o usuário. Já para o modo de preparo do Bolo uma listagem tipo List View faria mais sentido do que um Grid.
Mas como podemos usar um Grid View para uma Lista Ordena?
Vário exemplos disso vc pode encontrar em Manuais de Montagem, Diagramas, etc. Veja que nessa imagem temos uma Lista Ordena exibida em forma de Grid View.

Pontos a considerar: Independente de ser um Grid ou Lista, pense na experiencia do usuário (UX), e em qual será a melhor forma de passar esse conteúdo. No exemplo acima, com um avião caindo, é pouco provável que alguém tenha tempo para ler algo, além do mais cado passageiro pode falar uma língua diferente... Então uma listagem com imagem é um modelo ótimo para essa situação. Sempre se coloque no lado do usuário e sempre tenha a semântica em mente. Até pq o Google considera isso como boa prática e pode interferir até no SEO e rankeamento do seu site
Artigo de referência onde vc pode ver as vantagens e desvantagens do Grid View vs List View: https://uxplanet.org/mobile-ux-design-list-view-and-grid-view-8f129b56fd5b
Fator FrameWork
Muitas vezes, principalmente em Frameworks que possuem Grids como o Bootstrap, Foundation ou Materialize precisamos distribuir nosso conteúdo em Rows e Cols, para pessoas que não têm experiencia com CSS fica inviável construir um Grid View com UL/LI que siga a responsividade do Grid System, acredito que por isso vemos muitas listas construídas com Divs até hoje...
Aqui por exemplo tem um exemplo de lista semântica construida com UL/LI usando o Grid do Boostrap.

[class*="col-"] {
 border: 1px solid red !important;
}
ul {
  list-style: none;
}
<link rel="stylesheet" type="text/css" media="screen" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" />

 conteúdo dentro de li

 conteúdo dentro de li

 conteúdo dentro de li

 conteúdo dentro de li

 conteúdo dentro de li

 conteúdo dentro de li

DIV List e Acessibilidade
Em último caso, em uma situação onde vc não tenha a opção de optar por Listas e tenha que exibir seu conteúdo em Divs vc pode usar os atributos role para indicar aos leitores de tela que esse conjunto de divs deve ser interpretado como uma lista. Para isso declare a div container como role="list" e os itens internos como role="listitem" .
Assim a semântica e acessibilidade fica mais claras, apesar de que não podemos garantir como os leitores de tela vão tratar esse conteúdo, visto que os atributos role ainda não são um padrão oficial e com suporte total tando dos browser quanto dos leitores de tela. Vc pode se informar mais sobre esse atributo na documentação oficial da W3C: https://www.w3.org/wiki/PF/XTech/HTML5/RoleAttribute
Exemplo de listagem semântica feita com div:

dog

cat

sparrow

wolf!

Comment: Excelente resposta @hugocsl, a única coisa que eu alteraria seria a palavra `cego` por `deficiente visual`, visto que nem todos os usuários que utilizam leitores de tela são necessariamente cegos... mas é claro, isso é apenas uma tecnicalidade que não afeta a qualidade da resposta. Crítica construtiva man..
Comment: @fernandosavio com toda razão!


Recognize classical algebraic varieties as schemes

Question: Let $V$ be a (classical) algebraic variety over an algebraically closed field $k$.
The set $X$, which consists of all the irreducible closed sets of $V$, is a scheme on $k$.
Hartshorne's proof doesn't seem to work unless $k$ is an algebraically closed field.
Is $X$ a scheme when $k$ is a field that is not algebraically closed?
If not, is there a way to recognize $V$ as a scheme?
In Hartshorne's book, the projective space $ \mathbb {P} ^ n_A $ on the ring $ A $ is defined using the fiber product.
I don't know how this complex definition works.
Let $ k $ be a non-algebraically closed field. $ \mathbb {P} ^ n_k $ uses the same symbols as the classic projective space definition, but can they be recognized as the same?

addition.
I came across the following description on wikipedia.
In scheme theory, the $n$-dimensional projective space over $k$ is
$$\mathbb{P}^n_k = \text{Proj} k [x_0, ..., x_n]$$
It is defined using a polynomial ring like this. The entire $k$-value point of $\mathbb{P}^n_k$ matches the classical projective space.
Comment: What source do you have for classical varieties over non-algebraically closed fields? AFAIK, Hartshorne Chapter I assumes $k$ to be algebraically closed.
Answer: I don't think that is easily done if $k$ is not algebraically closed. Consider for example
$$V = \mathbb R$$
as a variety over $\mathbb R$. Then the associated scheme ought to be $X = \mathbb A^1_{\mathbb R} = \operatorname{Spec} \mathbb R[x]$, right? But the maximal ideal
$$(x^2 + 1) \subset \mathbb R[x]$$
does not correspond to any irreducible closed subset of $V$.
Comment: Thank you very much. I agree.

Added related questions.


Item do carrinho de compras não é removido

Question: Tenho um carrinho de compras criado dinamicamente, mas o problema é que quando eu excluo um item ele até remove, mas se eu tentar adicionar outro item  ele pega aquele que foi removido e adiciona novamente no carrinho de compras e mais o item que eu adicionei. O que será que eu de errado? Lembrando que este carrinho somente mostra os produtos para o usuário e remove.
Código de adicionar:
  // JQUERY ADICIONAR AO CARRINHO DE COMPRAS
  $(document).on("click",".btn-adicionar-carrinho",function(e)
  {
    // esvaziar o carrinho de compras
    carrinhovazio = $('.car-vazio').empty();
    // id do produto
    var idProduto = $('.nomeproduto').attr("id");
    // nome do produto
    var nomeproduto =  $('.header-ingrediente').text();
    // valor do produto
    var valorproduto = $('.v-lor').text();
    // quantidade do produto
    var quantidadeproduto = $('.item-qtd-produto').text();
    // imagem do produo
    var imagemproduto =  $('.imagemprodseq').data('id-imagem');

    mostraProdutos += "

"
    mostraProdutos += ""
    mostraProdutos += "

Remover[x]
"+ nomeproduto+ "
"
    mostraProdutos += "
Quantidade:

"+ quantidadeproduto  + "

"+ valorproduto +"

";
    // mostra os dados do carrinho de compras selecionado
    tabelacarrinho.html(mostraProdutos);

    somaValores();
  });

Código do botão excluir:
      // botão para fechar a tela de produtos do carrinho
      $(document).on("click",".excluir-produto-carrinho", function(e)
      {
        e.preventDefault();
        var c = "";

        if (urlselecionada == valorurl1)
        {
          $(this).fadeOut('slow', function(c)
          {
            var quantitem = 1;
            var contTotal = "";
            var quantcarrinho = $('#quantidadecarrinho').text();
            var idStorage = $('#nomeproduto').text();
            var listaquantcarrinho = $('#quantidadecarrinho');

            contTotal += (quantcarrinho - quantitem);

            $(this).closest('.cart-sec').remove();
            listaquantcarrinho.html(contTotal);
            JSON.stringify(excluirProdutoStorage(idStorage));

          });
        }
        else
        {
          var prodmostra = $(".prodmostra").length;
          //remove o produto conforme o total do carrinho
          var precoproduto = $(this).parent().find('.valorprodutocart span').text().replace('R$','');
          precoproduto = precoproduto.replace(',','.');
          // pega o valor total
          var pegatotalcarrinho = $('.total-carrinho').text().replace('R$','');
          pegatotalcarrinho = pegatotalcarrinho.replace(',','.');
          subtotal = subtrairValorTotal(pegatotalcarrinho, precoproduto);
          replacetotal = formataReal(subtotal);
          resultadoitem = Money.format(replacetotal);
          $(".total-carrinho").html(resultadoitem);

          // remove o item no carrinho de compras
          $(this).parent().parent().remove();

          if (prodmostra == 1)
          {
            carrinhovazio = $('.car-vazio').text('Carrinho vazio');
          }
          else
          {
            carrinhovazio = $('.car-vazio').empty();
          }
        }

      });

Comment: parece que você sempre concatena os valores presentes no carrinho pela var mostraProdutos mas não limpa quando é removido o item, e quando adiciona outro já utilizando o +=('mostraProdutos +='), o item anterior já removido vem junto.
Comment: Ele até funcionou, coloquei depois de $(this).parent().parent().remove(); o item mostraProdutos = "", mas na hora de adicionar dai ele apaga o que já está no carrinho. Como posso resover?
Comment: Blz. Voce pode utilizar o find na variavel e remover cada elemento, ou atribuir para uma var mediadora que entrega no final os produtos corretos. Vou te enviar o codigo na resposta.
Comment: Ok. Problema resolvido. Muito obrigado
Answer: Alterado conforme solicitação.
Segue código conforme conversamos.
// JQUERY ADICIONAR AO CARRINHO DE COMPRAS
$(document).on("click",".btn-adicionar-carrinho",function(e) {
    // esvaziar o carrinho de compras
    carrinhovazio = $('.car-vazio').empty();
    // id do produto
    var idProduto = $('.nomeproduto').attr("id");
    // nome do produto
    var nomeproduto =  $('.header-ingrediente').text();
    // valor do produto
    var valorproduto = $('.v-lor').text();
    // quantidade do produto
    var quantidadeproduto = $('.item-qtd-produto').text();
    // imagem do produo
    var imagemproduto =  $('.imagemprodseq').data('id-imagem');

    //Alteração 1!
    //Reload a var com os produtos presentes no carrinho.
    //Aqui vamos buscar os valores atuais que estão no carrinho para unir com o novo produto.
    mostraProdutos = tabelacarrinho.html();

    //Carrega o novo produto

    //Alteração 2!
    //Foi criada uma referência para cada produto, onde podemos com facilidade encontrar este elemento e manipular como quisermos.
    mostraProdutos += "
";

    mostraProdutos += "

"
    mostraProdutos += ""
    mostraProdutos += "

Remover[x]
"+ nomeproduto+ "
"
    mostraProdutos += "
Quantidade:

"+ quantidadeproduto  + "

"+ valorproduto +"

";
    // mostra os dados do carrinho de compras selecionado
    tabelacarrinho.html(mostraProdutos);

    somaValores();
});

Excluir:
// botão para fechar a tela de produtos do carrinho
$(document).on("click",".excluir-produto-carrinho", function(e) {

    e.preventDefault();
    var c = "";

    if (urlselecionada == valorurl1)
    {
      $(this).fadeOut('slow', function(c)
      {
        var quantitem = 1;
        var contTotal = "";
        var quantcarrinho = $('#quantidadecarrinho').text();
        var idStorage = $('#nomeproduto').text();
        var listaquantcarrinho = $('#quantidadecarrinho');

        contTotal += (quantcarrinho - quantitem);

        $(this).closest('.cart-sec').remove();
        listaquantcarrinho.html(contTotal);
        JSON.stringify(excluirProdutoStorage(idStorage));

        //Alteração 3!
        //Remover o produto do carrinho.
        //Busca pela referência que criamos na funcao acima o produto, localiza dentro do html corrente da pagina, e remove o elemento.
        tabelacarrinho.find('#prd_'+ idProduto).remove();

      });
    }
    else
    {
      var prodmostra = $(".prodmostra").length;
      //remove o produto conforme o total do carrinho
      var precoproduto = $(this).parent().find('.valorprodutocart span').text().replace('R$','');
      precoproduto = precoproduto.replace(',','.');
      // pega o valor total
      var pegatotalcarrinho = $('.total-carrinho').text().replace('R$','');
      pegatotalcarrinho = pegatotalcarrinho.replace(',','.');
      subtotal = subtrairValorTotal(pegatotalcarrinho, precoproduto);
      replacetotal = formataReal(subtotal);
      resultadoitem = Money.format(replacetotal);
      $(".total-carrinho").html(resultadoitem);

      // remove o item no carrinho de compras
      $(this).parent().parent().remove();

      if (prodmostra == 1)
      {
        carrinhovazio = $('.car-vazio').text('Carrinho vazio');
      }
      else
      {
        carrinhovazio = $('.car-vazio').empty();
      }
    }

});


What distinguishes a user story/feature from a use case/scenario?

Question: What distinguishes a user story/feature from a use case/scenario? Any pointers would be very much appreciated.
Is it just the granularity, similar to epics and user stories?
Answer: Much has already been written on the meanings of and difference between "use case" and "user story". Google "stackoverflow difference between use case and user story" to see some of it.
Regarding "feature" and "scenario":

In Gherkin, a Scenario is a series of steps that are executed together.

In terms of user stories: it might take one or many Scenarios to specify a user story, depending on the size of the story.
In terms of use cases: Gherkin's use of "scenario" corresponds almost exactly to the meaning of "scenario" in use case analysis: A use case describes a series of interactions between an actor (usually but not always a human user) and the system, in enough detail to make the system's responsibilities clear; a scenario is an instance of that use case with some values and the resulting details of interactions exactly specified, with the purpose of making some fine point of the system's responsibility (often edge case or error handling) clear. Just like a use case scenario, a Gherkin Scenario necessarily specifies all values completely (even if they're hidden in step definitions and not visible in the Gherkin itself), so "Scenario" was a good choice of name for that Gherkin concept.

A Gherkin Feature is just a collection of Scenarios, which you can use however you like.

In terms of user stories: A Feature can be the set of Scenarios that implement a user story or epic. In my experience, after more than one story has been implemented that touches the same part of an application (the same "feature"), one wants to put the Scenarios for those stories in the same Feature so one knows where to find them -- for example, you might want to put all the Scenarios related to user accounts (create an account, upgrade a subscription plan, cancel an account) in the same Feature rather than in different features, even though they implement quite different stories.
In terms of use cases: It usually takes several Scenarios to specify a use case (e.g. one for the happy path and one for an error path). Clearly these belong together in the same Feature file. As with user stories, one often wants to put the Scenarios for use cases that describe the same feature of the software in the same Feature.


How to programmatically create Fields?

Question: How can I approach implementing the following in Drupal 7?
What I need to do is create a module which defines a new fieldable entity, called 'Company'. I have a list of, let's say, 20 fields that need to be filled out by each Company instance. These questions are predefined and some may contain custom validation.
At the moment, I'm at the point when I can add new fields to the Company entity. This works fine at the moment. My issue is that I need all these fields to be there as soon the module is installed, so adding them via the interface isn't an option.
I was wondering how can I approach this? I'd assume it comes down being able to do what can be done using the 'Manage Fields' UI programmatically.
Comment: A few words of guidance. If you want to retain a level of control over the configuration of the fields, use Features to record and apply them. If you want to define them as a one-shot operation and let their configuration be overridden freely in the future, choose a code solution in an .install file.
Comment: If you're doing this in code as opposed to Features, take a look at the Fields Example and the Node Example in the [Examples Project](http://drupal.org/project/examples).
Comment: I'm not clear on the full scope of your needs, but I think this thread will be useful for you: http://drupal.org/node/721552 It shows sample code for creating a custom content type with fields when a module is first installed. You'll probably have to dig through the API to get the exact field settings you need, but this would be a good starting point. Basically, you will need to look into `node_type_set_defaults()` and `node_type_save()`, as well as, of course, `hook_install()`.
Answer: Use field_create_field() to create the field itself and field_create_instance() to have an instance for the given entity bundle.
When creating fields as part of a custom module, you may or may not want to delete the field when the module is uninstalled.  To do so, you can use field_delete_field() if you want to delete the field and all field instance, or if you want to delete specific instances you can use field_delete_instance().
Comment: How we delete the fields we created when uninstall the module ?
Comment: Ashok, I added a clarification for you in an edit I just made to the answer.
Answer: Example on how to programatically add fields to the user profile and how to avail them, or not, into the User Registration form.

function MYMODULE_enable() {
  // Check if our field is not already created.
  if (!field_info_field('field_myField')) {

    // Create the field base.
    $field = array(
      'field_name' => 'field_myField',
      'type' => 'text',
    );
    field_create_field($field);

    // Create the field instance on the bundle.
    $instance = array(
      'field_name' => 'field_myField',
      'entity_type' => 'user',
      'label' => 'My Field Name',
      'bundle' => 'user',
      // If you don't set the "required" property then the field wont be required by default.
      'required' => TRUE,
      'settings' => array(
        // Here you inform either or not you want this field showing up on the registration form.
        'user_register_form' => 1,
      ),
      'widget' => array(
        'type' => 'textfield',
      ),
    );
    field_create_instance($instance);
  }
}

Comment: This should be implemented in the hook_install().
Comment: If all you want to do is to add a new field to an existing content type and from there on continue in the backend, that approach is totally fine. Activate the module, deactivate it, done. The new field is there, editable, the module can be deleted.
Answer: If you need to quickly create / delete fields from an existing Content Type or or Entity, without neither using the UI nor programming, you can use these little-known Drush commands:

drush field-create <bundle(for nodes)>
  <field_name>,<field_type>,[widget_name] --entity_type : Type of
  entity (e.g. node, user, comment). Defaults to node.

E.g.: Create two new fields for Article:
drush field-create article city,text,text_textfield subtitle,text,text_textfield

Other commands:
drush field-delete <field_name> [--bundle] [--entity_type]
drush field-info [field | types]
drush field-update <field_name> Return URL for field editing web page.
drush field-clone <source_field_name> <dst_field_name>

Answer: As pointed by others, you can use the Field API functions from the hook_install() implementation of your module to create fields and their instances for you content type. See the node_example_install() for example usage of the function.
Another solution is to use the Features module. Features can export various site components to code in a module. Content types and fields are amongst these exportable. You can either generate a Features module and override your existing code, Features will then do its best to avoid breaking your code. Or you can generate a dummy module and copy./paste the fields-related code to your module. This requires basics understanding of how Features works.
Answer: In your install file you'll need to define both the 'hook_install' and the 'hook_uninstall'. Example included but read all about the extra keys in the API references (code is untested so might be typo's in there).
In the hook_install you can add the fields using:
field_create_field, This function builds a template for a field.
field_create_instance Can be used after creating the field to add it to content_types (also known as bundles).
NOTE
The names of various field types can be found in the modules that generate them (it's the key of the array item in their hook_field_info). You can find all the core field implementing modules in the modules/field/modules folder.
Settings can also be derived from the field modules. The settings you set in the field_create_field are the site wide ones. The ones you set in field_instance_create are the node_type specific ones
    MY_MODULE_install(){
      // Generate the base for the field
      $field = array(
        'field_name' => 'FIELD_MACHINE_NAME',
        'type' => 'FIELD_TYPE' // See note above for what to put here
      );
      // Instance
      $instance = array(
        'field_name' => 'FIELD_MACHINE_NAME',
        'entity_type' => 'node',
      );

      // Create instances of the field and add them to the content_types
      $node_types = node_type_get_types();
      foreach($node_types as $node_type){
         $instance['bundle'] = $node_type->type;
         field_create_instance($instance);
      }
    }

In the hook_uninstall
field_delete_instance and field_delete_field can be used to remove them again, field_delete_field is called automatically if you delete the last instance (normally).
    MY_MODULE_uninstall(){
      $node_types = node_type_get_types();
      foreach($node_types as $node_type){
        if($instance = field_info_instance('node', 'FIELD_MACHINE_NAME', $node_type->type)) {
          field_delete_instance($instance);
        }
      }
    }

Answer: I recently had a similar need for a project, here is how I approached it, hope it helps somebody.
Basically you will create the fields you need using the fields UI, export them to code and then include them in your custom module. You will need the Devel module enabled.
I also created a  Gist with this info.
Here we go....

Create the fields you need, using the usual Drupal UI.
On the same site, go to example.com/devel/php
Paste the following code into the "PHP code to execute" textbox.
Set the first 3 variables and then click execute
$entity_type = 'node';
$field_name = 'body';
$bundle_name = 'article';

$info_config = field_info_field($field_name);
$info_instance = field_info_instance($entity_type, $field_name, $bundle_name);
unset($info_config['id']);
unset($info_instance['id'], $info_instance['field_id']);
include_once DRUPAL_ROOT . '/includes/utility.inc';
$output = "\$fields['" . $field_name . "'] = " . drupal_var_export($info_config) . ";\n";
$output .= "\$instances['" . $field_name . "'] = " . drupal_var_export($info_instance) . ";";
drupal_set_message("<textarea rows=30 style=\"width: 100%;\">" . $output . '</textarea>');

You will get 2 arrays, something like this, hopefully with all the properties filled in.

$fields['field_some_field'] = array(
  'properties of the field'
);

$instances['field_some_field'] = array(
  'properties of the instance'
);

Now add the following code to your .install file. Replace all instances of mymodule with the actual module name. Paste the code from the devel output into the _mymodule_field_data and _mymodule_instance_data, as noted int the respective functions below.
You can do this for as many fields as you like, just put all the $fields arrays in the _mymodule_field_data function and all the $instances in the _mymodule_instance_data function.

function mymodule_install() {

  // Create all the fields we are adding to our entity type.
  // http://api.drupal.org/api/function/field_create_field/7
  foreach (_mymodule_field_data() as $field) {
    field_create_field($field);
  }

  // Create all the instances for our fields.
  // http://api.drupal.org/api/function/field_create_instance/7
  foreach (_mymodule_instance_data() as $instance) {
    field_create_instance($instance);
  }
}

// Create the array of information about the fields we want to create.
function _mymodule_field_data() {
  $fields = array();
  // Paste $fields data from devel ouput here.
  return $fields;
  }

// Create the array of information about the instances we want to create.
function _mymodule_instance_data() {
  $instances = array();
  // Paste $instances data from devel output here.
  return $instances;
}

Comment: h/t https://steindom.com/articles/exporting-and-creating-field-definitions-drupal-7
Answer: You could also consider using the Features module to create the fields at install time.
As Features generates code for fields, an option is just to use the Feature module to generate the code into a dummy module and then  copy & paste into your module's .install file.
The benefit being is the module doesn't depend on the Features module in your target environment.
Comment: Altough Features is a good way to export fields to code, that's not how to use Features. Features doesn't use the Field API CRUD to create fields from the generated .install.
Answer: You can use the customcompanymodule code given below to programatically create a content type with its various fields.
You can add this code in a .install file of your custom module. It will programmatically add  a content type called "company" and its various types of fields (text, numeric, date (note: you will need to install Date module as Date field is not provided by default), image, list).
I have also added the uninstall code that will remove the content type "company" along with all of its fields and data when you will uninstall your 'customcompanymodule' module.
You can modify/remove these fields as per your needs:
function customcompanymodule_install() {
     $t = get_t();
     node_types_rebuild();
     $company = array(
    'type' => 'company',
    'name' => $t('Company'),
    'base' => 'node_content',
    'module' => 'node',
    'description' => $t('Content type to handle companys.'),
    'body_label' => $t('Company Description'),
    'title_label' => $t('Company Title'),
    'promote' => 0,
    'status' => 1,
    'comment' => 0,
);
$content_type = node_type_set_defaults($company);

node_type_save($content_type);

foreach (_company_installed_fields() as $field) {
    field_create_field($field);
}

foreach (_company_installed_instances() as $instance) {
    $instance['entity_type'] = 'node';
    $instance['bundle'] = 'company';
    field_create_instance($instance);
}

$weight = db_query("SELECT weight FROM {system} WHERE name = :name",    array(':name' => 'categories'))->fetchField();
db_update('system')->fields(array(
            'weight' => $weight + 1,
        ))
        ->condition('name', 'company')
        ->execute();
}

function _company_installed_fields() {
$t = get_t();
$fields = array(
    'company_startdate' => array(
        'field_name' => 'company_startdate',
        'label' => $t('Company Start Date'),
        'cardinality' => 1,
        'type' => 'datetime',
        'module' => 'date',
        'settings' => array(
            'granularity' => array(
                'month' => 'month',
                'day' => 'day',
                'hour' => 'hour',
                'minute' => 'minute',
                'year' => 'year',
                'second' => 0,
            ),
            'tz_handling' => 'site',
            'timezone_db' => 'UTC',
            'cache_enabled' => 0,
            'cache_count' => '4',
            'todate' => 'required',
        ),
    ),
    'company_totalwinners' => array(
        'field_name' => 'company_totalwinners',
        'label' => $t('Maximum Company Winners'),
        'cardinality' => 1,
        'type' => 'number_integer',
        'module' => 'number',
        'settings' => array(
            'max_length' => 10000,
        ),
    ),
    'company_minwinner' => array(
        'field_name' => 'company_minwinner',
        'label' => $t('Minimum Entries for Company to Activate'),
        'cardinality' => 1,
        'type' => 'number_integer',
        'module' => 'number',
        'settings' => array(
            'max_length' => 10000,
        ),
    ),
    'company_totalentries' => array(
        'field_name' => 'company_totalentries',
        'label' => $t('Company Total Entries'),
        'cardinality' => 1,
        'type' => 'number_integer',
        'module' => 'number',
        'settings' => array(
            'max_length' => 10000,
        ),
    ),
    'company_points' => array(
        'field_name' => 'company_points',
        'label' => $t('Company Points'),
        'cardinality' => 1,
        'type' => 'number_integer',
        'module' => 'number',
        'settings' => array(
            'max_length' => 10000,
        ),
    ),
    'company_image' => array(
        'field_name' => 'company_image',
        'label' => $t('Image'),
        'cardinality' => 1,
        'type' => 'image',
        'settings' => array(
            'default_image' => 0,
            'uri_scheme' => 'public',
        ),
    ),
    'company_description' => array(
        'field_name' => 'company_description',
        'label' => $t('Company Description'),
        'cardinality' => 1,
        'type' => 'text',
        'module' => 'text',
        'length' => '255'
    ),
    'company_winner' => array(
        'field_name' => 'company_winner',
        'label' => $t('Company Description'),
        'cardinality' => 1,
        'type' => 'text',
        'module' => 'text',
        'length' => '255'
    ),
    'field_autowinnerselection' => array(
        'field_name' => 'field_autowinnerselection',
        'label' => $t('Auto Company Winner Selection'),
        'type' => 'list_boolean',
        'module' => 'list',
        'active' => '1',
        'locked' => '0',
        'cardinality' => '1',
        'deleted' => '0'
    ),
);
return $fields;
}

function _company_installed_instances() {
$t = get_t();
$instances = array(
    'company_startdate' => array(
        'field_name' => 'company_startdate',
        'label' => $t('Company Lifespan'),
        'cardinality' => 1,
        'widget' => array(
            'type' => 'date_popup',
            'module' => 'date',
            'settings' => array(
                'input_format' => 'm/d/Y - H:i:s',
                'input_format_custom' => '',
                'year_range' => '-3:+3',
                'increment' => '15',
                'label_position' => 'above',
                'text_parts' => array(),
            ),
        ),
    ),
    'company_totalwinners' => array(
        'field_name' => 'company_totalwinners',
        'label' => $t('Maximum Company Winners'),
        'cardinality' => 1,
        'widget' => array(
            'type' => 'number',
            'module' => 'number',
            'settings' => array('size' => 60),
        ),
    ),
    'company_minwinner' => array(
        'field_name' => 'company_minwinner',
        'label' => $t('Minimum Number of Entries for Company to Activate'),
        'cardinality' => 1,
        'required' => 1,
        'widget' => array(
            'type' => 'number',
            'module' => 'number',
            'settings' => array('size' => 60),
        ),
    ),
    'company_totalentries' => array(
        'field_name' => 'company_totalentries',
        'label' => $t('Company Total Entries'),
        'cardinality' => 1,
        'required' => 1,
        'widget' => array(
            'type' => 'number',
            'module' => 'number',
            'settings' => array('size' => 60),
        ),
    ),
    'company_points' => array(
        'field_name' => 'company_points',
        'label' => $t('Company Points'),
        'cardinality' => 1,
        'required' => 1,
        'widget' => array(
            'type' => 'number',
            'module' => 'number',
            'settings' => array('size' => 60),
        ),
    ),
    'company_image' => array(
        'field_name' => 'company_image',
        'label' => $t('Image'),
        'cardinality' => 1,
        'required' => 1,
        'type' => 'company_image',
        'settings' => array(
            'max_filesize' => '',
            'max_resolution' => '213x140',
            'min_resolution' => '213x140',
            'alt_field' => 1,
            'default_image' => 0
        ),
        'widget' => array(
            'settings' => array(
                'preview_image_style' => 'thumbnail',
                'progress_indicator' => 'throbber',
            ),
        ),
        'display' => array(
            'default' => array(
                'label' => 'hidden',
                'type' => 'image',
                'settings' => array('image_style' => 'medium', 'image_link' => ''),
                'weight' => -1,
            ),
            'teaser' => array(
                'label' => 'hidden',
                'type' => 'image',
                'settings' => array('image_style' => 'thumbnail', 'image_link' => 'content'),
                'weight' => -1,
            ),
        ),
    ),
    'company_description' => array(
        'field_name' => 'company_description',
        'label' => $t('Company Description'),
        'cardinality' => 1,
        'widget' => array(
            'weight' => '-3',
            'type' => 'text_textfield',
            'module' => 'text',
            'active' => 1,
            'settings' => array(
                'size' => '1000',
            ),
        ),
    ),
    'company_winner' => array(
        'field_name' => 'company_winner',
        'label' => $t('Company Winner'),
        'cardinality' => 1,
        'widget' => array(
            'weight' => '-3',
            'type' => 'text_textfield',
            'module' => 'text',
            'active' => 1,
            'settings' => array(
                'size' => '60',
            ),
        ),
    ),
    'field_autowinnerselection' => array(
        'field_name' => 'field_autowinnerselection',
        'required' => 1,
        'label' => $t('Auto Company Winner Selection'),
        'widget' => array(
            'weight' => '-3',
            'type' => 'options_buttons',
            'module' => 'options',
            'active' => 1,
            'settings' => array(),
        ),
    ),
);
return $instances;
}

function customcompanymodule_uninstall() {
$content_types = array(
    'name1' => 'company',
);
$sql = 'SELECT nid FROM {node} n WHERE n.type = :type1';
$result = db_query($sql, array(':type1' => $content_types['name1']));
$nids = array();
foreach ($result as $row) {
    $nids[] = $row->nid;
}
node_delete_multiple($nids);
node_type_delete($content_types['name1']);
field_purge_batch(1000);
}

Answer: Try this module if you're looking for a way to create fields programmatically in Drupal 8: https://drupal.org/project/field_create

How to programmatically create Fields?


Wondering what type of covariance correction for standard errors is better: Hansen-Hodrick or Newey-West?

Question: I am wondering what type of covariance correction for standard errors is better: Hansen-Hodrick or Newey-West?
Also, does someone know if StatsModels package that uses "HAC" for robust covariance uses Hansen-Hodrick or Newey-West correction for standard errors?
statsmodels.regression.linear_model.RegressionResults.get_robustcov_results

Answer: statsmodels uses by default Newey-West corrected standard errors with the usual Bartlett window.
There is a 'weights_func' or 'kernel' option to choose a different window than Bartlett, eg. uniform.
However, statsmodels has no other options for HAC robust standard errors like pre-whitening or automatic lag selection, or autocorrelation robust standard errors without heteroscedasticity robustness (i.e. only 'HAC', but no AC).
If I remember correctly Hansen-Hodrick is 'AC' with uniform kernel.