What are modes and how are they useful?

Question: I'm a self-taught guitarist with zero to little experience in music theory. After marrying a violinist I found that there's a lot to music theory that I know tribally but don't understand the technical or formal definitions for.
I'd like for someone to briefly explain to me what the different modes are (Dorian, Phrygian, Lydian, Mixolydian, Aeolian, Locrian, and Ionian) and how they are used.
Comment: One of the most interesting and useful classes I took was music theory, tied to its corollary, ear training. Theory gives you the knowledge to understand song structure and how to wind a melody through chord changes. It's important to hear it as we're playing but the knowledge lets you think ahead based on knowing the changes and hear/see it in advance and decide on tonally safe vs. adventurous lines.
Comment: For *Too Much Info*, see [this ebook](http://www.hellenicaworld.com/Greece/Literature/DavidBinningMonro/en/TheModesOfAncientGreekMusic.html).
Comment: http://music.stackexchange.com/questions/1136/what-is-a-circle-of-fourths-chord-progression Have you seen this?
Comment: @DRL Yeah, I like the conciseness of your answer on there. There's also some good answers on this post too, so I'm keeping it open :).
Answer: Let's take the C Major scale which consists of C, D, E, F, G, A, B and back to C.
The bare basic way to think about the modes is: play the scale starting at another note.
So, the C major scale can be played starting at C:

C D E F G A B C (C - C)

Or starting at D:

D E F G A B C D (D - D)

It's the same exact notes of the scale, you just start at the 2nd note & end at the second note. This is Dorian.
You can keep on going; start & end on the 3rd, then the 4th, then the 5th, then the 6th and finally the 7th.
Putting it all together we get 7 modes of C Major

Ionian (CDEFGABC)
Dorian (DEFGABCD)
Phrygian (EFGABCDE)
Lydian (FGABCDEF)
Mixolydian (GABCDEFG)
Aeolian (ABCDEFGA)
Locrian (BCDEFGAB)

Again, you take the basic C Major scale & start & end it on another note. Same pattern, same intervals, just change where you start & end it.
The usage is 100% up to you. If you want to play in C Major (continuing the example) you now have more options. You can play any of the 7 C Major modes and still be in key.
Another great use of the modes is that you can string them together. Since each note shows up on the fretboard many times you can string modes together to help you move up and down the board. Play C Ionian on the 8th fret of the 6th string; when you get to the C octave on 4th string fret 10 you can then slide into 12th fret of the 4th string (which is a D) and start Dorian. Now you're starting up an octave and you've moved hand position up the board by a couple frets. You can continue that all the way thru.
Creatively string together the modes (you don't have to play all the notes each time) yield you a decent little solo =).
Comment: You did use them, you just didn't know that was what they were called. Musicians who solo eventually learn about modes as they learn new styles of music, especially if you want to play styles that have a Latin feel, or are more jazz-like. You can claim they are useless and learn over time, or go after the knowledge and really learn; It just depends on what you want to do with your music and how successful you want to be.
Comment: Arin's answer has some truth but potentially confuses things.  Playing the notes in the C scale starting on D is in fact a Dorian scale but it is the D Dorian.  Playing the same notes found in the C Ionian scale but starting on E gives you E Phrygian (not C Phrygian as some may interpret Arin to mean).  The various modes are differentiated by the relative intervals between the notes (whole or semi tone).  Neil Meyer's answer below gives further explanation of the notes in the other modes of C (all of which contain notes unique to their respective modes).
Comment: Hmz... I'm playing for 7 years already, and haven't used modes even once... Why you need to start from D note if D chord is playing? What about your ears? I'm ALWAY have good accents while soloing on any chord changes! It's there's any other usefulness of modes beside "right note in a right place" ? This is completely useless for me. I know major/minor scales, and that's it.. Intuition shows the way while playing... tell me more please =)
Comment: @holms John Lennon felt the same way! http://en.wikipedia.org/wiki/Not_a_Second_Time
Comment: An analogy would be like throwing a ball at a coconut shy: You can either do the maths which calculates the trajectory, force required, etc and stand a good chance of hitting the coconut each time, or you can just chuck the ball and learn from how it 'felt'. Some like the maths & prediction, but 'play by ear' types (myself included) find it more fun to just throw the ball. Then you find out there's mathematics/theory which describe what you just did, which you feel you don't need to keep on doing it.
Comment: That is incorrect. It all depends on context. If your tonic is C and you play what you believe is D Dorian (for example: ascending scale), you are actually just playing in C Ionian a 2-3-4-5-6-7-1-2. Anything that depends on visualizing and not on hearing should not be discussed regarding to music, only regarding techniques and specific instruments.
Comment: Those are not the the 7 C Major modes. Those are C ioninan, D dorian, E phrygian, F lydian, G mixolydian, A aeolian and B locrian. Modes in C other than ionian must have sharp or flattened notes.
Answer: The thing to realize about modes is that they are simply emphasizing different key notes in the same collection of notes.
Take a scale, any scale, and "emphasize" one note. This will make the "Scale" sound different than if you emphasize a different note.
C ionian and A aeolian are the common major and minor(almost) type of sound. But they are exactly the same notes. What makes them different is that in one you are emphasizing C and in the other you are emphasizing A.
That is, you somehow make the listener lock onto the root note. You do this by setting it up using certain techniques.
For example, if you take CDEFGAB as your "scale" but play a B in the background underneath everything you play, B will seem to dominate(not to be confused with the dominant) the sound. Every note will be heard against that B and your ears will treat B as the "king" of all the notes.
This means if you play a C against it you are playing a min2nd interval. This interval doesn't exist in C major because the 2nd interval is D which gives a maj2nd. The other intervals are just as important and all of them relative to B give that characteristic sound which we call locrian.
Now if you simply changed the root emphasis to A then that C is not a min3rd which gives a minor sound.
For example, if you take a single line solo and play a B bass note it will transform the sound than if you played any other note.
In fact, if you want, you can think of all modes, scales, keys, etc... as simply coming from the chromatic scale and whats makes all the different sounds different is simply emphasis. C major is the emphasis of the note C followed by others less and less so. A min is the exact same 12 notes but we emphasis an A as the root and others less and less.
Effectively we create a hierarchy. You have your king, queen, prince, servants, etc...
The best way to really hear it is to simply play the same scale over a minor chord and then a major chord and hear how it changes.
Note that it is possible to play sort of major sound over a minor chord by emphasizing that a major sound in your solo. e.g., if you play C E G over a static A bass note you'll have a more major sound than if you play C E A. This is because you are outlining(called an arpeggio) the C major chord even when the bass note is A. Depending on a lot of factors you might hear the A note as being part of the C chord rather than the G being part of the Am chord. i.e., you have the notes A - C E G. But depending on context your brain might group them as A C E - G or A - C E G. The first case we have a sort of Am called Am7 and the second we have a sort of Cmaj6. Most likely you will hear Am7 unless the A bass note is very quite.
In any case it doesn't matter much about the theory as you need to learn the sounds. You just have to know that it all depends on emphasis and context. If I'm playing a Cmaj chord and you're playing a Dbmaj chord at the same time what key are we in(assuming that's all we are playing)? what mode? The answer is that it depends.
Comment: Regarding your example: When you are playing C E G over a static A bass note, you're playing the b3-5-b7 of the scale, which is a bIIImaj chord. When playing C E A, you're playing b3-5-1, which is a first inversion i minor chord.
Most of your answer doesn't regard modes, but the pedal tone (static bass) is absolutely correct.
Regarding your ending - "The answer is that it depends" - Depends on what? Depends on what you perceive as the tonic, if you even perceive a tonic.
Answer: They are part of our western music tradition. They are no harder to learn that any other scale. You can use them to give a feeling to the music that is unlike anything else. To there uses I can say there use is the same as any other scale. They are used to make music.
This explanation relies on you at the very least knowing what semi tones and whole tones are. I do not think it is possible to learn them without you knowing that. I strongly you urge you to ask a theory teacher to explain that to you.
You should not see them as any different to any other scale. A scale at its core is a set of notes with the semi tones and whole tones in specific places. The Major scale has its semi tones between the third and fourth note in the scale and the seventh and eight. Seeing as the various modes have semi tones in different places they do not sound alike. If you play any popular type music training your ears to recognize the modes would be a tremendous skill to possess.
No matter on which note you start from if the semi tones are at that spot you have a major scale. This is why to a certain degree all major scales sound alike. The distance between the notes are the same in all the Major Keys.
So with that in mind we can use our knowledge of semi tones and whole tones on the Modes so we can learn what notes they have regardless on what note we start from. I will start on C for each example.
DORIAN:
Has its semitones between 2 / 3 and 6/7.
C Dorian hence has C-D-Eb-F-G-A-Bb-C
PHRYGIAN:
Has its semitones between 1/2 and 5/6
C Phrygian therefore has C-Db-Eb-F-G-AB-Bb-C
LYDIAN:
Has it semitones 4/5 and 7/8
C lydian has C-D-E-F#-G-A-B-C
MIXOLYDIAN:
Has its semi tones between 3/4 and 6/7
C Mixolydian hence has the notes C-D-E-F-G-A-Bb-C
AEOLIAN:
Has its semi tones between 2/3 and 5/6
C aeolian has C-D-Eb-F-G-Ab-Bb-c
IONIAN:
Has its semi tones between 3/4 and 7/8
C ionian therefore has C-D-E-F-G-A-B-C
If the last two looks familiar it is because they are our good old Major and Minor Keys.
SOURCE:
Oxford Companion To Music 10th Edition
Comment: A scale is not a stack of seconds - it is a collection of scale degrees, and the seconds between the notes is just something that happens whenever there's more than two notes involved. Classical students learn scales using intervals because that's how they were taught, however, that doesn't mean that it's correct.
For example: The intervals of the Ddorian scale (WHWWWHW) appear when playing C major from D. That doesn't mean that you just played a D dorian. The only thing that matters is the tonic, and what the relations of all the notes played to it.
Answer: Modes are just an indication of where you start in the scale.  I think far too much emphasis is placed on modes - I've never found any good practical application for them.  They are, in my experience, an unnecessary abstraction. Know your current/active chord; know the tones in the chord; play notes related to those tones; know what chord comes next and tie those mutts together.  Bam.
Comment: Modes give us ways to understand, create and or work with harmonies that are not strictly conforming to a major or minor tonal center or feel. They are heavily used in Jazz, often times shifting from one modal center to another multiple times in a given work. Know how the modes relate to harmonies and you have a strong insight into the complicated harmonic and melodic world of jazz.
Comment: @Tulkinghorn. What you describe (know the active chord, the tones in the chord and notes related to those tones) is tantamount to knowing your modes. You're basically saying you need to know your modes, just maybe not their names. It's like saying you don't need to know the C major scale as long as you now the C major chord, the tones in the C major chord and the notes related to those tones...So you don't need to know the C major scale, but you need to know the C major scale. How does this help?...
Comment: This answer ignores a broad swath of music that does not rely on chords but is distinctly modal. The overlap between such music and more chordal music is considerable, for example areas in songs or charts where a musician is allowed or required to expand on a single tonal center. Not all tonal centers are chord-based.
Also, chords do not tell you all there is to know. Consider a minor 7 chord on C. If the mode is phrygian, the neighbor tones are different from dorian. In the first, one would play D-flat and A-flat, in the second one would play D- or A-natural. Very different character.
Comment: This answer is incorrect. Modes refer not to the current chord (however many people do not understand that - even in higher levels of education) but to the tonal center.
Answer: ignore or forget anything that says "starts from".
melodies that start on C might not be in the key of C or any of it's modes.  it could be Em phygian or A minor!  "pickup notes" can be misleading.  I would've understood modes 6 years earlier if every book or article didn't say "STARTS FROM".  It's incredibly misleading and just plain wrong.
situation #2 - chord progressions may produce modal change
key of C feels different with F in the bass vs B in the bass.  What you're actually hearing is the difference between F Lydian and B locrian!!!  This is the key to begin understanding.  It's still key of C, but when the bass changes it's a new CONTEXT.
modal changes are better thought of as CONTEXTUAL changes.
situation #3 - key changes may produce modal change
Play key of Ab with a C in the bass.  Then move to key of G with a C in the bass.  Bass note stays the same but what you're really hearing is Cm Phrygian moving to C Lydian.  It's a key change AND a modal change!
rule #1 - modes are directly related to style and emotion.  modal changes are better thought of as "Mood Changes"
you will never understand modes until you learn to quantify the emotional experience resulting from a mode and associating it accordingly.  MAKE A LIST on your own of the MOOD you feel when you hear each mode.  Do not listen to other's opinions of whether the mixolydian mode sounds "bluesy" or "jam band" or if the locian mode sounds like "the circus" or "sludge metal" or if Ionian sounds like "lullaby" or "pop punk".  Only you can match your subjective reactions to the modes with the mode names themselves.
Afterwords, you can use modes at will to cause an emotional effect.  (I want to sound happy, now I want to sound evil) for example.  For some styles of music, some modes (moods) are inappropriate and will never be used.  Death Metal crowds do not want to hear Ionian.  Pop Punk crowds do not want to hear Locrian.  Hippes LOVE mixolydian.   Goth kids scream in terror when they hear mixolydian, unless if maybe it's coming from The Doors
Comment: This is built on the axiom that you're hearing the scale - that is incorrect. You are hearing the scale degrees and the clashes between them. For example: In the mixolydian scale, there's a tritone between 3 and b7, which gives you a somewhat bluesy feel, as we've heared that so many times in blues.
Answer: The noteworthy differences of the various modes of a scale are the variations in the tonal centers. For example, D-Dorian mode (in C-major scale) will have the tonal center of D instead of C. Depending on the mode, it can be of major & minor tonal center. Ionian, Lydian & Mixolydian are of major tones, whereas Dorian, Phrygian & Aeolian are of minor tones. Locrian mode shows diminished tone, which in principle is also a minor tone.
Comment: Your answer is really close to the truth - correct regarding the tonal center and not the intervals, however, the tonal center is a single note, not a chord.
The reason why we hear something as major or minor is because we've been spoonfed only ionian and aeolian music, which have 3 different scale degrees, thus are very audible. However, the third really doesn't matter
Higher level education tends to build on the that incorrect axiom that major and minor are entirely different things - and on the incorrect axiom that scales are audible (they are just theoretical guidelines).
Answer: Modes are scales which share the same notes, but the context is different. That's it. What does that mean? Context means tonality - If we play C with the tonic (tonal center, most stable note) being C, it will sound a lot different than if the tonic was B (making C the b2, a very unstable note).
For example: The modes of the C Major Scale are C Ionian, D Dorian, E Phrygian, F Lydian, G Mixolydian, A Aeolian and B Locrian. The letter refers to the tonic and the name of the mode refers to the scale degrees:

Ionian: 1-2-3-4-5-6-7
Dorian: 1-2-b3-4-5-6-b7
Phrygian: 1-b2-b3-4-5-b6-b7
Lydian: 1-2-3-#4-5-6-7
Mixolydian: 1-2-3-4-5-6-b7
Aeolian: 1-2-b3-4-5-b6-b7
Locrian: 1-b2-b3-4-b5-b6-b7

So how do we make a certain note the tonic? A mistake made by a lot of people is to think that a you get a mode when you start on a different note. The tonic is the most used note, or the note of resolution for a phrase. If you play the notes of the C major scale from F, you are not going to get F Lydian, but C major. However, if you play the notes of the C major scale (doesn't matter on which note you start) with F in the bass, or always return to F (or F in the harmony, so on...), you will hear F Lydian.
However, there is nothing mystical about modes and scales. You absolutely do not hear the sound of the mode, but the sound of its "characteristic" scale degrees - Scale degrees that are different from what you're so used to hearing: In the Lydian scale, you hear the #4. In Mixolydian, you hear the b7 and so on...
This is why scales are useless for ear training and for ear based improvisation - only serving as "training wheels" for people without a strong ear, and a poor excuse for people to not do ear training.
Answer: Modes are one way of thinking about melodic lines and their underlying harmonies. They can be used as a compliment or an alternative to the "usual" diatonic way of thinking about harmony. They predate diatonic harmony by a number of centuries in Western music. The diatonic system grew out of the modal system as it became more polyphonic because certain melodic tensions seemed to always resolve in certain ways (See this question for an excellent overview of the details). As other posters have pointed out, modes are basically just inversions of a scale. As such, all scales have modes, though symmetric scales, like the diminished, have fewer modes than they have scale degrees (the diminished has eight degrees but only two modes). That said, the modes that have specific names are primarily inversions of the major scale.
In modern usage modes are often a way to build more interesting melodies that fit with certain chords, or sometimes to generate chord changes themselves.
An example of the first type of usage would be playing C Lydian over a C Major triad. The F sharp (a raised 4th) from the lydian doesn't clash with any of the chord tones, so if a composer or improviser wants a fourth in the melody she can chose to use the sharp four from the lydian. That may not be the only or best way of thinking about or analyzing this note (maybe it's meant to imply a secondary dominant, for example), but this sort of chord/scale relational thinking is common especially in jazz.
Also, each mode has a characteristic "flavor." So for the most common modes of the major scale: lydian sounds playful and whimsical (e.g. Saria's song from the game Ocarina of Time), Mixolydian sounds like a primeval version of the major scale with it's flat leading tone ("In a Silent Way" is the best example I can think of), Phrygian is oftern associated with flamenco music (more stereotypically, you'll have heard it in the "bull fighter" music that's played at sporting events), and Dorian has a funky, tongue in cheek sound (Herbie Hancock's "Chameleon", Miles Davis' "So What"). From these examples you can probably infer that these characteristic sounds emerge most clearly when the harmony remains relatively static or vamps between two or three chords.
Comment: Who's to say that an F# over  major isn't using Blues rather than Lydian? Could be construed as b5.
Comment: Tim, what you're hearing is the note a tritone above the tonic. Doesn't matter at all what's the scale, as you do not hear the scale, only the scale degrees.
Answer: Modes are used in many types of music; from sacred music to jazz to rock. Composers use it to add "flavor" to their compositions in order to avoid predictability. It is formed by naming a different note as the root (1st) instead of the original root of the scale. Thus, in a way, modes can be defined as displaced major scales.
Types of Modal Scales
Ionian - Also known as the major scale; follows the pattern W-W-H-W-W-W-H.
Dorian - Constructed from the second note of a major scale; follows the pattern W-H-W-W-W-H-W.
Phrygian - Constructed from the third note of a major scale; follows the pattern H-W-W-W-H-W-W.
Lydian - Constructed from the fourth note of a major scale; follows the pattern W-W-W-H-W-W-H.
Mixolydian - Also known as "mixo," is constructed from the fifth note of a major scale and follows the pattern W-W-H-W-W-H-W.
Aeolian - Also known as the natural minor scale, is constructed from the sixth note of a major scale and follows the pattern W-H-W-W-H-W-W.
Locrian - Constructed from the seventh note of a major scale; follows the pattern H-W-W-H-W-W-W.
Each mode has a very distinct sound; for example, the Phrygian mode sounds melancholy and reflects the music of Spain. The Lydian mode sounds happy and is often used in jazz and rock music. The Mixolydian mode evokes a bluesy sound and can often be heard in jazz, blues and rock music. The Locrian mode, on the other hand, has a very strange sound but is rarely used.

What are modes and how are they useful?


Update Drush in Docker4drupal

Question: I have to upgrade Drupal core to 8.4 with Composer. Drush version is 8.1.12.
I use Docker4drupal for my local development environment. How do I update Drush to 9 within my docker container setup, so I can upgrade Drupal to 8.4?
Answer: Assuming you're using the default wodby/drupal:8-7.1-3.3.2 container image, the drush launcher is already installed in the PHP FPM container.
Thus, your next step is to update your Drupal 8 composer drush dependency via composer:
composer require 'drush/drush:^9'
When drush launcher executes under your web root it will pick up the version 9 that's installed in your vendor dir.

Update Drush in Docker4drupal


Source files in a bash script

Question: I am using two versions of ROS next to each other. To use one I have to source some environment variables for the specific version. I would like to create a script that does this.
But if I create a script like below the variables are not set, they are probably set in a subshell. How can I source the files to the main terminal shell?
source.sh:
source /opt/ros/fuerte/setup.bash;
source  ~/fuerte_workspace/setup.bash;

Here is how i am calling source.sh:
./source.sh
# This does not echo anything, but I expect it should
echo $ros_config

Update: By sourcing source.sh as suggested in the answer, I can now see the variables being set.
source ./source.sh
# This works now
echo $ros_config

Comment: What about sourcing `source.sh`? This can help you: http://stackoverflow.com/a/15968190/1983854
Answer: Execute Shell Script Using .  ./ (dot space dot slash)
While executing the shell script using "dot space dot slash", as shown below, it will execute the script in the current shell without forking a sub shell.
$ . ./setup.bash

In other words, this executes the commands specified in the setup.bash in the current shell, and prepares the environment for you.
Comment: Note: `source` is an alias for `.` not the other way around.
Comment: `.` is an alias for `source` when delimited by space (like all bash commands). If you `source` you'll avoid common bugs associated with using `.` to mean the current directory, etc.
Comment: This works fine for scripts in the current directory - but what do you do if the script is in /usr/local/bin? That's more convenient because then you don't have to cd to the directory where your script is located.
Answer: Use dot notation to source in the script file in the current shell i.e. without creating a sub-shell:
. /opt/ros/fuerte/setup.bash
. ~/fuerte_workspace/setup.bash

Comment: "`./source.sh`" executes the script (in its own shell). "`. ./source.sh`" (or "`source ./source.sh`", or "`source source.sh`") will apply the script contents to the current shell environment.
Comment: SInce you're providing full path to setup.sh here therefore `./` is not needed since `./` means current dir.
Comment: If I use ./source.sh it does not work. source source.sh does work!

Source files in a bash script


Print field with an array

Question: I have a field name called 'field_serial_571' that contains an array. I've tried several ways to display it and it only display blank space. I ran a debugging script (printr) and this is what I found:
<code>[field_serial_571] => Array ([und] => Array ([0] => Array ([value] => 1)))
</code>
Does anyone know how to display results from a field with an array?
Answer: Assuming this is an entity (it looks like it probably is), you might want to look into using entity metadata wrappers. It'll make your theming life much easier.
Otherwise, to immediately display your field, this would work (assuming a node):
echo $node->[field_serial_571][LANGUAGE_NONE][0]['value'];
LANGUAGE_NONE is a constant for an undefined language, in your example the 'und' part of the array. It's best practice to use that constant rather than using 'und'.
Read even more about entity metadata wrappers here: http://www.pixelite.co.nz/article/how-use-entity-metadata-wrappers-drupal-7/
Comment: I got it to work using this:
'hidden')));
?>

Print field with an array


JSON response will not append to div

Question: I have this

JSON response will not append to div


Multivariate Brownian motion definition and properties

Question: The following is the definition of a Wiener process that I am following:

I am confused regarding the multivariate Brownian motion which is defined as follows:

My question is, does $\mathbf{W}_t$ follow the same conditions 1-4 for a univariate Wiener process? Obviously, condition 1. is satisfied since $\mathbf{W}_0 = (W_0^1, \cdots, W_0^k)^T = (0, \cdots, 0)^T$, but how are conditions 2. and 3. satisfied? For example, is it true that $\mathbf{W}_u - \mathbf{W}_t$ independent of $\mathbf{W}_s - \mathbf{W}_r$? If i write out the vectors, I get:
$\mathbf{W}_u - \mathbf{W}_t =\begin{bmatrix} W_u^1 - W_t^1 \\ \vdots \\ W_u^k - W_t^k \end{bmatrix}$ and $\mathbf{W}_s - \mathbf{W}_r =\begin{bmatrix} W_s^1 - W_r^1 \\ \vdots \\ W_s^k - W_r^k \end{bmatrix}$. How are these two vectors independent of each other? If we look element by element, clearly $W_u^1 - W_t^1$ is independent of $W_s^1 - W_r^1$ but is $W_u^1 - W_t^1$ independent of $W_s^2 - W_r^2$?
Finally, for condition 3. is it true that $\mathbf{W}_t - \mathbf{W}_s \sim N(0, (t-s)\mathbf{I}_k)$ where $\mathbf{I}_k$ is the $k \times k$ identity  matrix? If so, why is it true?
Answer: your first definition is the definition of a standard one-dimensional Brownian motion.
The second definition is of a non-standard $k$-dimensional Brownian motion.
In particular
$$
Z_t - Z_s \sim N(\mu (t-s), (t-s) \Sigma ).
$$
Therefore, if you set $\mu = 0$ and $\sigma = I_k$, then $\Sigma = I_k$ and
$$
W_t - W_s \sim N(0, (t-s)I_k).
$$
Note that in the second definition it is assumed that $W_t$ consists of $k$ independent (one dimensional) standard Brownian motion. In particular,
$W^2_u$ is independent of $W^1_t$ for all $u$ and $t$.


how can i use SVM for text classification without using Tf-idf

Question: I want to classify every entry message.I work with Persian texts. I already implemented a text classifier with Naive Bayes. i did not use Tf-idf because every single feature is important for me. But i did some tricks to delete stop-words and pouncs to have a better accuracy.
I want to implement a text classifier with SVM  but i searched a lot. all I found is related to using pipeline function with the use of Tf-idf. like below:
model = Pipeline([('vectorizer', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', OneVsRestClassifier(LinearSVC(class_weight="balanced")))])

now, how can i use SVM without Tf-idf?
thanks
Comment: could you provide more about the model you are trying to build? what are your features? words? are you using bag of words of the massage as your data?
Comment: my model include body as text of messages and label. i have 6 labels. yes my data is made of some words that make sentences.@thebeancounter
Answer: See here for the sklearn page about SVM, there you have a section for multiclass classification using SVM. You first have to convert your texts into a feature vector (numeric, if you wish you use SVM) If you should like to use bag of words you could use this SO question and this manual page of sklearn
You can use pre written python code to create BOW from you texts doing something like that - mind you, i geathered the relevant information for the OP - that was unclear and not compatible with SO strandarts, so you might need to work the code a bit for it to fit your excact usage.
>>> from sklearn.feature_extraction.text import CountVectorizer

>>> vectorizer = CountVectorizer()
>>> vectorizer
CountVectorizer(analyzer=...'word', binary=False, decode_error=...'strict',
        dtype=<... 'numpy.int64'>, encoding=...'utf-8', input=...'content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), preprocessor=None, stop_words=None,
        strip_accents=None, token_pattern=...'(?u)\\b\\w\\w+\\b',
        tokenizer=None, vocabulary=None)

>>> corpus = [
...     'This is the first document.',
...     'This is the second second document.',
...     'And the third one.',
...     'Is this the first document?',
... ]
>>> X = vectorizer.fit_transform(corpus)
>>> X
<4x9 sparse matrix of type '<... 'numpy.int64'>'
    with 19 stored elements in Compressed Sparse ... format>

Then you might need to convert x into a dense matrix (depends on sklearn version)
Then you could feed x into SVM model you can create like so
>>>>from sklearn import svm
>>> X = [[0], [1], [2], [3]]
>>> Y = [0, 1, 2, 3]
>>> clf = svm.SVC(gamma='scale', decision_function_shape='ovo')
>>> clf.fit(X, Y)
SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovo', degree=3, gamma='scale', kernel='rbf',
    max_iter=-1, probability=False, random_state=None, shrinking=True,
    tol=0.001, verbose=False)
>>> dec = clf.decision_function([[1]])
>>> dec.shape[1] # 4 classes: 4*3/2 = 6
6
>>> clf.decision_function_shape = "ovr"
>>> dec = clf.decision_function([[1]])
>>> dec.shape[1] # 4 classes


Why 1 is appended in last in the substraction in double format in java? How to resolve it?

Question: 2.333333333333333 - 1.666666666666667 It appends 1 in the last, increasing one bit. How to resolve it.
In the answer its giving, 0.6666666....1, but it should be 0.66666....6
Comment: give explanation to your question.. make it clear to us.
Comment: Welcome to Stack Overflow. Your question doesn't give us enough detail in order for us to give you an answer. At a minimum, you should post a relevant sample of your code and the exact result you are getting. I encourage you to take the [tour](http://stackoverflow.com/tour) and to visit the [help center](http://stackoverflow.com/help/asking) for guidance on asking good questions on this site.
Comment: Please read [What Every Computer Scientist Should Know About Floating-Point Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html).  The computer cannot represent numbers with arbitrary precision when using `float` or `double`.
Comment: In the answer its giving, 0.6666666....1, but it should be 0.66666....6
Answer: I am going to guess you meant.6666...7 but it should be .6666...6 if you plan on programming I would strongly recommend that you learn how software interacts with the hardware. When a computer stores memory it cannot save the same amount of positive numbers as negative numbers. The reason being that the computer uses somthing of the format like,
10000001111

For (this is in binary) negative numbers where the one in front indicates that it is negative while,
00000001111

is the positive version of the same number (the absolute value for all you math nerds). Seeing this you should realize that you can store more positive numbers than negative numbers, so you need to be careful with whatever type you are using. However since your question is not 100% clear I cannot guarantee this will 100% answer your question. Good luck...


Why for simple roots in Lie algebras the master formula reduces to one integer?

Question: The master formula for two generic weights (roots) is
$$ 2 \frac{\vec{a} \cdot \vec{b} }{\vec{a} \cdot \vec{a} }=q-p $$
but if we require that the roots are simple then this reduces to
$$ 2 \frac{\vec{a} \cdot \vec{b} }{\vec{a} \cdot \vec{a} }=q \neq 0 $$
i.e. $p=0$. Why is this and why $q\neq0$?

P.S. I understand why  $q\neq0$ since it comes straight out from the requirement that the roots are simple.
Comment: Yes, I copied it wrong. But still I cannot understand the main part of the question.
Answer: The answer to your doubt can be found in chapter $8$ of Lie algebras and particle physics by Georgi.
In particular, a simple root is a root that cannot be written as a sum of positive roots.
Given two simple roots $\vec{a}, \vec{b}$ then $\vec{a}- \vec{b}$ is not a root, otherwise - say $\vec{b}$ is larger - $\vec{b} = \vec{a} + (\vec{b}-\vec{a})$ and $\vec{b}$ would be not simple.
So, since $\vec{a}- \vec{b}$ is not a root you have
$$ 2 \frac{\vec{a}\cdot \vec{b}}{\vec{a}\cdot\vec{a}} = q-p$$
with $p = 0$, just because the combination of roots you have chosen is not a root. Recall that $p$ is the number of times you apply a (negative) root to your weight representation state by $E_{-a}$.
Comment: In any case, do you remeber how $p$ is defined? given the $SU(2)$ algebra defined by a root $\vec{a}$, $p$ is the maximum (non-negative) integer such that $(E_{-a})^p\vert \mu, R\rangle \neq 0$. For a simple root applied to the state (in the adjoint rep) $\vert E_{b} \rangle$ such integer is zero, since $\vec{a}-\vec{b}$ is not a root. (I am very sorry, my notation had $p$ and $q$ exchanged, now I fixed it)
Comment: Ah, so I guess this does not apply to $SU(2)$ because it only has one simple root, right? I need bigger ranks to see this working.
Comment: Yes, exactly. $SU(2)$ is to small to see this working, Cartan sub-algebra is one dimensional, etc.
Comment: thanks. I do not have that particular book but I uploaded a pic from the one I have. I agree with what you say but I do not understand yet why $p=0$! If $p$ is that then what is $q$ (the equivalent for the "lowest weight state"?)
Comment: if you click on the link of the book you get a pdf
Comment: Yes, I remember, I agree. I am confused with this when we are going to simple roots. $p$ tells me how many times I can apply the "raising operator" before I get zero, this is the maximum number of $q$ characterizes the maximum root or weight. But if $p=0$ then for a simple root it means I cannot have a raising operator?
Comment: No, what is zero is the number related to the lowering operator. Meaning that starting from a state given by a simple root, you cannot descend along the direction of another simple root.
Comment: The book you sent me seems to have the clearest of the explanations. Thanks.


Ehresmann's lemma equivalent etale topology

Question: In complex algebraic geometry we have Ehresmann's lemma which states that given $f:X \to Y$ a smooth surjective proper map between complex manifold this is locally a fibration in the analytic topology.
I was wondering whether there was a generalization of that in the context of etale topology. Lots of statement which are true analytic locally in the context of $\mathbb{C}$ can also be carried in the etale topology for smooth varieties over any algebraically closed field $k$.
In the (beautiful) notes by Conrad https://www.math.ru.nl/personal/bmoonen/Seminars/EtCohConrad.pdf at page 53, he states what he calls etale Ehresmann's lemma: given $f:X \to S$ smooth and proper, the sheaf $R^if_*F$ are locally free whenever $F$ is a torsion etale sheaf on $X$.
The hypothesis are however very general:assuming $X,S$ to be both smooth varieties over an algebraically closed field, it is true the stronger statement that $f$ is a fibration etale locally?
Answer: No. I think you might be misinterpretting what 'locally trivial fibration' means.
Namely, what Ehresmann's theorem implies is the following:

Corollary to Ehresmann's theorem: Let $f:X\to Y$ be a smooth proper surjection of complex manifolds. Then, there is an open cover
$\{U_i\}$ of $Y$ such that $f^{-1}(U_i)$ is diffeomorphic (as a manifold over $U_i$) to
$U_i\times B$ for some smooth manifold $B$.

One cannot replace diffeomorphic by 'biholomorphic' or, if $X$ and $Y$ are algebraic and the $U_i$ is a Zariski cover, 'algebraic'.
Here's a silly counterexample to your literal claim.
Consider $D(\Delta)\subseteq \mathbb{A}^2_k=\mathrm{Spec}(k[a,b])$ where, for simplicity, $k$ is an algebraically closed field of characteristic greater than $3$ where
$$\Delta(a,b)=-16(4a^3+27b^2)$$
We then have the universal family of Weierstrass equations living over $D(\Delta)$
$$\mathcal{W}^\text{univ}=V(y^3-axz^2-bz^3)\subseteq \mathbb{P}^2_k\times D(\Delta)$$
We then note that
$$\mathcal{W}^\text{univ}\to D(\Delta)$$
is a family of elliptic curves and so, in particular, smooth proper and surjective. Moreover, each $\mathcal{W}^\text{univ}$ and $D(\Delta)$ is smooth.
That said it's not an etale local fibration since this, in particular, would imply that the fibers of $\mathcal{W}^\text{univ}\to D(\Delta)$ are Zariski locally isomorphic. Indeed, let $\{U_i\to D(\Delta)\}$ be an etale cover such that $\mathcal{W}^\text{univ}\times_{D(\Delta)}U_i\cong U_i\times B_i$ for some $k$-space $B_i$. Then, in particular we see that for all $x\in V_i:=\text{im}(U_i\to D(\Delta))$ (this is an open set) we have that
$$\mathcal{W}^\text{univ}\times_{D(\Delta)}U_i\times_{V_i}x\cong \bigsqcup_{y\mapsto x}B_i$$
which since $k(x)\to k(y)$ is an isomorphism implies that $\mathcal{W}^\text{univ}_x\cong B_i$. Thus, $\mathcal{W}^\text{univ}_x\cong B_i$ for all $x\in V_i$.
Note that we have an algebraic map $j:D(\Delta)\to \mathbb{A}^1_k$, the $j$-function map, which sends, in particular, $x$ to the $j$-invariant $\mathcal{W}^\text{univ}_x$. Since $\mathcal{W}^\text{univ}_x$ is locally isomorphic this implies that $j$ is locally constant which implies, since $D(\Delta)$ is connected, that $j$ is constant and thus $\mathcal{W}^\text{univ}_x$ is actually constant for $x$ in $D(\Delta)$.
Of course, this is ludicrous since $\mathcal{W}^\text{univ}_x$ contains isomorphism classes of every elliptic curve over $k$.
Of course, it is true that the fibration $W^\text{univ}\to D(\Delta)$, when $k=\mathbb{C}$, is analytically locally on the target DIFFEOMORPHIC to the trivial fiber bundle. The operative point being that any two elliptic curves are diffeomorphic but are rarely isomorphic/biholomorphic.
Comment: Great answer! Thank you very much this example was superuseful!It is probably well known but the basic case of smooth of relative dimension =0 (i.e finite and etale) is actually true. A good reference is  proposition 16.25 of these notes (http://people.dm.unipi.it/tdn/CoomologiaEtale/Note.pdf which are actually hyperwritten in my opinion)
Comment: *super well written indeed.
Comment: @TommasoScognamiglio Sure but this is the whole reason, in some sense, that finiteness criterions are needed in 'naive' definitions of etale cohomology (e.g. why l-adic cohomology is defined using a limit).


drupal 7: how to add a field that displays a message when user is not logged in?

Question: I need a way to set up a field available only for logged in users. When anonymous user reach the page that contains this field, drupal must display a message that request to hte user to login or register to display this field. This is the same that appens for comments. There is a module that impelments this for single fields?
Comment: What about showing a block only to anonymous users on pages that contain the field?  Would that work?
Comment: You need a normal entity field that only displays for logged in users and if the user is not logged in it should display a message instead, right?
Answer: You can try the following 2 modules: Field Permissions and Markup.

Define the field.
Directly behind, define the markup field that shows the message to log in.
Use field permissions to allow access to the target field for authenticated users only.
Use field permissions to allow access to the markup field for anonymous users only.

That workaround should solve you issue.


Failing to save attributes of associated object in rails application

Question: I have three models defined as follows
class Student < ActiveRecord::Base
  belongs_to :user
  has_many :placements
  has_many :companys , through: :placements
end

class Company < ActiveRecord::Base
    has_many :placements
    has_many :students , through: :placements
end

class Placement < ActiveRecord::Base
  belongs_to :student
  belongs_to :company

  before_save  :set_placed

  def set_placed
    s = self.student
    s.is_placed = true
    s.save
  end
end

Each time i add data for placement object i want to update a field in its corresponding student object. But when i use rails_admin to add data , i am getting the error Placement failed to be created .
When i remove the before_save call , data can be added.
I am using better_errors gem for debugging. I am getting the following from it
@_already_called

{[:autosave_associated_records_for_student, :student]=>false,
 [:autosave_associated_records_for_company, :company]=>false}

i am hoping this could be the reason for error.
How can i solve this error??
Answer: You have a s.save in your set_placed callback. You don't save an ActiveRecord object in a callback, and especially not in a before_save callback.
Comment: then how can i do the same?
Comment: Try to remove the explicit `s.save` line from the `set_placed` callback. It looks like the associated records are auto-saved, anyway.
Answer: try this,
def set_placed
  self.student.is_placed = true
end


handling file upload for drupal 6

Question: I have a question about handling file uploads.  I am trying to implement a form that will allow the user to upload a file.  I do not need to store the file, though.  I just need to take the uploaded file, where its location happens to be, and pass it to an outside API, which uploads it to its server (its Jira).  Is there a way to do this?  The file has to be publicly available to be consumed by Jira's API (in other words, Jira won't be able to provide a username/password to validate against drupal).  This is what I have, so far:
in my _add_form hook:
$form['file'] = array(
                '#type' => 'file',
                '#title' => t('Document'),
                '#description' => t('Upload a file, allowed file types: pdf, images, documents'),
            );

in my _add_form_validate hook:
$validators =  array(
                'file_validate_extensions' => array( 'jpg jpeg gif png bmp txt doc docx xls xlsx pdf ppt pptx ' )
            );

            $file = file_save_upload('file', $validators);

            if (isset($file->filename)) {
                $form_state['storage']['file'] = $file;
            }

in my _add_form_submit hook:
i can see the object form_state->storage->file but I don't see the full path to where the file is residing.  All i see, thats close, is:
 file_name = "myfile.jpeg"
 file_path = "/tmp/myfile.jpeg"
 destination = "/tmp/myfile.jpeg"

but, as I mentioned, I need to full path to pass to another, external, API so that the file can be consumed.  I do not need to store or save the file afterwards.
admin/settings/file-system settings:
File System Path:  sites/mySite.net/files
Temporary Dir: /tmp
Download Method: Private

Comment: I think you would have to actually save the file and then delete it on the success to the API call to Jira since the /tmp area is probably not going to be publicly available. Well at least I would think it wouldn't be. Either way I think setting it up as a normal file upload into the public directory (and setting a proper directory for the image) running through the API calls on the submit success and then deleting the file on the API success would work out. If you would like an overview in code I can post an answer or maybe I am misunderstanding what you are trying to accomplish?
Comment: Thanks for your response.  That might be a way to do it.  Can you explain how to upload to a public directory?  I'm new to working in drupal.
Comment: Yes I will add an answer shortly stepping through a high-level solution for this.
Answer: Ok here is high level overview of the file uploading. I took most of this from the Drupal 6 Developer modules but made some changes that should get you started on how to achieve your final goal here. I left all the original comments and denoted any new comments with // ** Comment Text so you should be able to skim through this and get the general idea.
<?php

//////////////// Tutorial Example 11 //////////////////////

// ** Will denote any changes or anything that I am pointing out below.

/**
 * Example 11: A form with a file upload field.
 *
 * This example allows the user to upload a file to Drupal which is stored
 * physically and with a reference in the database. In this example, only
 * png, gif, jpg, and jpeg files are allowed.
 *
 * @see form_example_tutorial_11_validate()
 * @see form_example_tutorial_11_submit()
 * @ingroup form_example
 *
 */
function form_example_tutorial_11($form_state) {
  // This is required to upload files.
  // enctype="multipart/form-data" required by browsers to handle files.
  $form = array(
    '#attributes' => array('enctype' => "multipart/form-data"),
  );

  $form['file'] = array(
    '#type' => 'file',
    '#title' => t('Image'),
    '#description' => t('Upload a file, allowed extensions: jpg, jpeg, png, gif'),
  );

  $form['submit'] = array(
    '#type' => 'submit',
    '#value' => t('Submit'),
  );

  return $form;
}

/**
 * Validate handler for form_example_tutorial_11().
 * Verify the valid extensions, and verify content is an image also.
 */
function form_example_tutorial_11_validate($form, &$form_state) {

  // ** I just left this as image but you can change the ext array if needed.
  $file = file_save_upload('file', array(
    'file_validate_extensions' => array('png gif jpg jpeg'),
    'file_validate_is_image' => array(),
  ));

  // If the file passed validation: ** File is in /tmp folder now
  if (isset($file->filename)) {
    // Move the file, into the Drupal file system
    // ** Notice the 'jira-upload/', this would be where your file is placed,
    // ** It will be in your /files folder, but you need to make this directory,
    // ** You can do it in code but for simplicity I just left it as is for now.
    if (file_move($file, 'jira-upload/' . $file->filename)) {
      // Update the new file location in the database.
      // ** Change from /tmp/filename.ext to,
      // ** /sites/default/files/jira-upload/filename.ext in database.
      // ** This will be important for Drupal later.
      drupal_write_record('files', $file, 'fid');
      // Save the file for use in the submit handler.
      $form_state['storage']['file'] = $file;
    }
    else {
      form_set_error('file', t('Failed to write the uploaded file the site\'s file folder.'));
    }
  }
  else {
    form_set_error('file', t('Invalid file, only images with the extension png, gif, jpg, jpeg are allowed'));
  }
}

/**
 * Submit handler for form_example_tutorial_11().
 */
function form_example_tutorial_11_submit($form, &$form_state) {
  $file = $form_state['storage']['file'];
  // We are done with the file, remove it from storage.
  unset($form_state['storage']['file']);
  // Make the storage of the file permanent/temp
  // ** This will actually make it where you don't have to delete,
  // ** You can let Drupal clean it up later on cron runs after a certain,
  // ** Amount of time. This will be done automatically.
  file_set_status($file, FILE_STATUS_TEMPORARY);
  // ** This is what you wanted, was the public file that can be navigated to,
  // ** This should work pretty well, but there are other ways.
  // ** Since validation is already passed I would run the API call here as,
  // ** Well. You could pass the $file object to another function or your pref.
  $fileurl = file_create_url($file->filepath);
  // Set a response to the user.
  // ** I added the $fileurl so you could see it on the frontend,
  // ** Without having to know if devel is there.
  drupal_set_message(t('The form has been submitted and the image has been saved, fileurl: @fileurl. filename: @filename.', array('@filename' => $file->filename, '@fileurl' => $fileurl)));
}

?>

Note: For this to work with the way you are wanting to curl the image and send it you will need to set your file system to public. I'm placing a screenshot of that page with configuration.

Additional Hook:
If your file system is private you will have to run an additional hook in order to allow access to the file through Drupal without having it attached to a node or anything. I tested this with making my file-system private and clearing out all previous images, then along with the code that I edited from the Drupal 6 Examples module I placed the code below in the main module file (just incase anyone needs to know how to test this out without knowing all the code). This should give you the functionality that you need without having to change the file-system.
<?php

/**
 * Implements hook_file_download().
 */
function form_example_file_download($file) {
  //dpm($file);
  // Used jira- as indicator but it needs to match your folder where you are storing your files.
  if (strpos($file, 'jira-') === 0) {
    $info = image_get_info(file_create_path($file));
    return array('Content-type: ' . $info['mime_type']);
  }
}

?>

Comment: Thank you for your very in depth response.  The problem, though, is that when my Curl object tries to open the file, I get this error :couldn't open file "http://localhost/mySite/web/system/files/jira_attachments/DSC_0344_3.JPG"
Comment: Is this due to Drupal permissions or something with Curl?
Comment: oh, and if it helps for context, the reason I need to get this to work is for another issue I'm having, here : http://stackoverflow.com/questions/28030564/adding-attachment-to-jiras-api
Comment: So when you submit your form you are actually getting the mySite/web/system/files...? Is that where you setup your public files for Drupal to be served from? You can check that config by going to this page /admin/settings/file-system Also can you get to the image in the browser?
Comment: Yes, I can see the file in my folder explorer being uploaded. I can then navigate in my browser to that file, using the url that's created by file_create_url.  As for permissions in admin/settings/file-system, all I see are:  the option is Download Method: private.  Also, if I use the absolute path (*id: c:/windows/wamp/...") it works fine
Comment: Yea based on the absolute URL that you placed in the other question you will need to change the file upload config. Make that setting public, and set the file path to sites/default/files or sites/mysitename.com/files (whatever the path Drupal will need to be able to write to this).
Comment: I've amended my question to show the settings I have in admin/settings-file-system.  Do you mean change Download Method to public?  Would this have a global affect?  My module is just one small module in a larger app.  I can't make changes that might have affects on other modules/site.  is there another way to deal with this?
Comment: Yes I will update my answer with the new code after lunch
Comment: Great, thanks!  If I can get this to work I certainly won't mind donating some rep points!
Comment: hmm.. this seems to work:  $fileurl =$_SERVER['DOCUMENT_ROOT'] .  $GLOBALS['base_path'] . file_create_path($file->filepath);  Is there any reason I couldnt use that instead of file_create_url?
Comment: What's the URL that comes out? I'm sure you possible could use that but only locally, you are skipping drupal that way I believe.
Comment: It's not a url.  It's the full path on the machine, where the file lives.  Yeah, this might only work locally, though
Comment: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/20364/discussion-between-burnsjeremy-and-jason).

handling file upload for drupal 6


Query DELETE demorando muito para ser carregado

Question: Estou tendo um problema com uma query, ela tem a função de excluir alguns registros duplicados existentes em uma tabela.
Segue:
DELETE t1
FROM bid_account t1
    INNER JOIN bid_account t2
WHERE t1.id > t2.id AND
    t1.bidding_price = t2.bidding_price AND
    t1.auction_id = '" . $obj->auction_id . "' AND
    t1.bidding_type = 's' AND
    t1.bid_flag = 'd' AND
    DATE(t1.bidpack_buy_date) >= DATE(NOW())

O problema é que ao executar ela, demora muito para concluir e não resolve nada, parece não funcionar.
Eu fiz um select com essas mesmas condições, e algo absurdo aconteceu, ele repetiu vários registros, que não existem, por ex para o leilão com ID 15 tem 100 registros, ao consultar usando essas condições ele retorna mais de 80 mil registros.
Como posso fazer essa query funcionar corretamente?
Comment: Parece haver um **erro de lógica** no seu `SQL`. Se na consulta retorna mais de 80.000 registros é porque seus relacionamentos estão incorretos. Ou talvez o resultado seja esse mesmo.
De qualquer forma, eu evitaria fazer esse `JOIN` que vc fez.... talvez seja melhor fazer um `Exists`, para evitar essas duplicidades.
Comment: tambem acho que você precisa melhorar seu sql.. tem muita clausula alem do join. Se no select já retorna muitos registros, vai demorar mesmo no delete. Tente simplificar
Answer: Antes de responder, vamos colocar alguns pontos nos is:

Eu fiz um select com essas mesmas condições, e algo absurdo aconteceu, ele repetiu vários registros, que não existem, por ex para o leilão com ID 15 tem 100 registros, ao consultar usando essas condições ele retorna mais de 80 mil registros.

Nada de absurdo aconteceu. Esse resultado é um produto cartesiano que foi provocado pelo fato de você ter declarado um JOIN da tabela bid_account com ela mesmo sem qualquer restrição. Então, supondo que você tivesse 10 registros nela. O resultado seria 10 * 10 = 100 registros no resultado inicial. Na sequencia é que seriam aplicadas as restrições que você declarou no WHERE, o que poderia diminuir esse número (mas não muito).

Ao executar ela, demora muito para concluir e não resolve nada, parece não funcionar.

Demora! E para cada novo registro que for inserido nela, a demora aumenta exponencialmente. Isso certamente é pelo mesmo motivo do ponto acima.
Versão TLDR;
Se eu entendi corretamente, você quer eliminar registros que estão completamente repetidos em bid_account e manter somente o que tem o menor id. Nesse caso, sua consulta deveria ser assim*:
DELETE t1
FROM bid_account t1
WHERE EXISTS ( SELECT 1
               FROM bid_account t2
               WHERE t2.bidding_price = t1.bidding_price AND
                    t2.auction_id = t1.auction_id AND
                    t2.bidding_type  = t1.bidding_type AND
                    t2.bid_flag = t1.bid_flag AND
                    DATE(t2.bidpack_buy_date) = DATE(t1.bidpack_buy_date) AND
                    t2.id < t1.id) AND
    t1.auction_id = '" . $obj->auction_id . "' AND
    t1.bidding_type = 's' AND
    t1.bid_flag = 'd' AND
    DATE(t1.bidpack_buy_date) >= DATE(NOW())

* Nota: Você deverá avaliar se as regras que definem a sua "duplicidade" são essas mesmo que eu declarei no subselect. É ali que você deve identificá-las. No WHERE você estabelece seus critérios para exclusão (exceto o que identifica a duplicidade). Assim sua query fica mais limpa e mais fácil de fazer ajustes como mudar os critérios
Um pouco mais sobre o que aconteceu
Na sua query, com o que foi declarado como restrições no where e supondo que a tabela tenha o conteúdo abaixo, teríamos o seguinte:

----------------------------------------------------------------------------------------------
| id | bidding_price |        auction_id        | bidding_type | bid_flag | bidpack_buy_date |
----------------------------------------------------------------------------------------------
|  1 |           2.0 | " . $obj->auction_id . " | s            | d        |    05/05/2018    | // Seria eliminado de t1 porque o id não é maior do que nenhum de t2
----------------------------------------------------------------------------------------------
|  2 |           2.0 | " . $obj->auction_id . " | s            | d        |    13/08/2018    |
----------------------------------------------------------------------------------------------
|  3 |           2.0 | " . $obj->auction_id . " | s            | d        |    17/10/2019    |
----------------------------------------------------------------------------------------------
|  4 |           1.2 | " . $obj->auction_id . " | s            | a        |    22/02/2019    | // Seria eliminado de t1 pelo flag 'a'
----------------------------------------------------------------------------------------------
|  5 |           1.2 | " . $obj->auction_id . " | t            | d        |    30/09/2018    | // Seria eliminado de t1 pelo tipo 't'
----------------------------------------------------------------------------------------------
|  6 |           1.2 | " . $obj->auction_id . " | s            | d        |    01/04/2017    | // Seria eliminado de t1 pela data menor que hoje
----------------------------------------------------------------------------------------------

Então, sobraram 2 registros de t1 e todos os de t2 pois pelo menos um registro de t2 tem id menor do que cada um id de t1 e possui bidding_price equivalente também.
Sabendo que o processamento já seria relativamente lento pelo produto cartesiano, se fizessemos um Select t1.* ... teríamos como resultado final o seguinte (2 * 6 = 12):

----------------------------------------------------------------------------------------------
| id | bidding_price |        auction_id        | bidding_type | bid_flag | bidpack_buy_date |
----------------------------------------------------------------------------------------------
|  2 |           2.0 | " . $obj->auction_id . " | s            | d        |    13/08/2018    |
----------------------------------------------------------------------------------------------
|  3 |           2.0 | " . $obj->auction_id . " | s            | d        |    17/10/2019    |
----------------------------------------------------------------------------------------------
|  2 |           2.0 | " . $obj->auction_id . " | s            | d        |    13/08/2018    |
----------------------------------------------------------------------------------------------
|  3 |           2.0 | " . $obj->auction_id . " | s            | d        |    17/10/2019    |
----------------------------------------------------------------------------------------------
|  2 |           2.0 | " . $obj->auction_id . " | s            | d        |    13/08/2018    |
----------------------------------------------------------------------------------------------
|  3 |           2.0 | " . $obj->auction_id . " | s            | d        |    17/10/2019    |
----------------------------------------------------------------------------------------------
|  2 |           2.0 | " . $obj->auction_id . " | s            | d        |    13/08/2018    |
----------------------------------------------------------------------------------------------
|  3 |           2.0 | " . $obj->auction_id . " | s            | d        |    17/10/2019    |
----------------------------------------------------------------------------------------------
|  2 |           2.0 | " . $obj->auction_id . " | s            | d        |    13/08/2018    |
----------------------------------------------------------------------------------------------
|  3 |           2.0 | " . $obj->auction_id . " | s            | d        |    17/10/2019    |
----------------------------------------------------------------------------------------------
|  2 |           2.0 | " . $obj->auction_id . " | s            | d        |    13/08/2018    |
----------------------------------------------------------------------------------------------
|  3 |           2.0 | " . $obj->auction_id . " | s            | d        |    17/10/2019    |
----------------------------------------------------------------------------------------------

Espero que isso ajude.
Comment: Óptima explicação. espero que ajude o OP.
Comment: @DiegoRafaelSouza [Quais são as abreviações/gírias mais comuns usadas no Stack Overflow em Português?](https://pt.meta.stackoverflow.com/questions/6311/quais-s%C3%A3o-as-abrevia%C3%A7%C3%B5es-g%C3%ADrias-mais-comuns-usadas-no-stack-overflow-em-portugu%C3%AA/6312#6312) :D
Answer: A melhor solução que encontro é estabeleceres um limite para apagar.
Faz o delete na mesma acrescentando, por exemplo, LIMIT 10000 no final da query. Apaga os registos na mesma mas por parcelas.
Um ciclo ou uma rotina para ir executando essa query e penso que o problema fica resolvido.


Adding a "dynamic" route manually for troubleshooting

Question: Is there a way to add a route in Linux with the Dynamic flag set?
The reason I want to do this is to troubleshoot an issue where there exist identically static and dynamic routes, and what happen if I try to delete the static route.
We suspect that the dynamic route was removed, and not the static route.
I have tried:
route add -net 192.168.100.0/24 gw 192.168.0.1 dyn

But route -n only show the flags UG.
Answer: Try:
ip route add 192.168.100.0/24 via 192.168.0.1

could be written as
ip r a 192.168.100.0/24 via 192.168.0.1

there's some trace found in my history.
ip route add 192.168.100.0/24 via 192.168.0.1 scope global dev eth0

You could even add a secondary ip address to same link;
ip a a 192.168.100.210/24 dev eth0 label eth0:test

ip is a full featured iproute2 tool. But as low level, you may have to ip address add before ip route add... So please, RTFM ;-)
man -Pless\ +/^ip.route ip

Comment: I think you misunderstand `dynamic route` meaning. If I rightly understand you ask, you can't. Did you already browse `http://lartc.org` the *linux advanced routing & traffic control* documentation?
Comment: Thanks for reply, but I do not see how this command add the route with the D flag.


Função para limpar html

Question: Como fazer uma função que limpe o HTML? Tenho um campo que vem de um WS e preciso, antes de exibir na página, que limpe o HTML.
Me enviaram isso:
String result = Regex.Replace(htmlDocument, @"<[^>]*>", String.Empty);

Eu preciso colocar esse código numa função jquery, exatamente aqui e o this.ShortDescription é o cara que eu quero limpar. Como eu faço?
str += '
' + this.ShortDescription + '
';

O código acima é parte de uma função jQuery.
Comment: Como assim limpar o html?
Comment: @pnet, por exemplo sua entrada é `algum texto.....` e vc quer que alguma função retorne apenas `algum texto` é isso?
Answer: Em javascript você pode usar:
document.body.innerHTML = '';

Comment: sim, mas como assim Description?
Comment: Eu tenho dificuldade com jquery e javascript. Eu tenho uma jquery que pega de um WS o texto, assim: str = str + this.Description; Depois descarrego a variável str na pagina e é montado tudo. Isso tá funcionando. Como eu agora removo as tags html do Description, conforme exemplo passado?
Comment: é o campo que eu pego na jquery do WS assim: this.Description; esse campo é que possui os textos que eu colocar na página. Como eu removo as tags dele?
Comment: @pnet tenta `this.Description.innerHTML = '';`
Comment: Não deu certo. Ficou undefined =
Comment: @pnet tenta postar mais partes do seu código para podermos entender melhor o seu problema...
Answer: Eu uso o seguinte código vb.net:
Public Function limpaHTML_Recursiva(ByVal sTexto As String, ByVal j As Integer) As String
    If j = 0 Then
        Return sTexto
    End If
    Dim i As Integer = 0
    i = InStr(sTexto, "<")
    j = InStr(sTexto, ">")
    If i > 0 Then
        i -= 1
    End If
    sTexto = Left(sTexto, i) & Right(sTexto, sTexto.Length - j)
    Return (limpaHTML_Recursiva(sTexto, j))
End Function

Chama com :sTexto = limpaHTML_Recursiva(sTexto, 1)

Função para limpar html


Geomesa: Caused by: java.lang.ArrayIndexOutOfBoundsException

Question: Trying to ingest shapefiles with over 600 columns in Geomesa I get the following error.
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter.writeFeature(GeoMesaFeatureWriter.scala:56)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter.writeFeature$(GeoMesaFeatureWriter.scala:46)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$TableFeatureWriter.writeFeature(GeoMesaFeatureWriter.scala:151)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$GeoMesaAppendFeatureWriter.write(GeoMesaFeatureWriter.scala:239)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$GeoMesaAppendFeatureWriter.write$(GeoMesaFeatureWriter.scala:235)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$$anon$3.write(GeoMesaFeatureWriter.scala:111)
        at org.locationtech.geomesa.utils.geotools.FeatureUtils$.write(FeatureUtils.scala:147)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$8(LocalConverterIngest.scala:181)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$8$adapted(LocalConverterIngest.scala:179)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
        at org.locationtech.geomesa.utils.collection.CloseableIterator$FlatMapCloseableIterator.foreach(CloseableIterator.scala:132)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$7(LocalConverterIngest.scala:179)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$7$adapted(LocalConverterIngest.scala:173)
        at org.locationtech.geomesa.utils.io.CloseablePool$CommonsPoolPool.borrow(CloseablePool.scala:68)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$6(LocalConverterIngest.scala:173)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$6$adapted(LocalConverterIngest.scala:172)
        at org.locationtech.geomesa.utils.io.package$WithClose$.apply(package.scala:64)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$4(LocalConverterIngest.scala:172)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$4$adapted(LocalConverterIngest.scala:168)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
        at org.locationtech.geomesa.utils.collection.CloseableIterator$CloseableSingleIterator.foreach(CloseableIterator.scala:85)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$3(LocalConverterIngest.scala:168)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$3$adapted(LocalConverterIngest.scala:167)
        at org.locationtech.geomesa.utils.io.package$WithClose$.apply(package.scala:64)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$2(LocalConverterIngest.scala:167)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$2$adapted(LocalConverterIngest.scala:166)
        at org.locationtech.geomesa.utils.io.CloseablePool$CommonsPoolPool.borrow(CloseablePool.scala:68)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.run(LocalConverterIngest.scala:166)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 2136 out of bounds for length 1024
        at com.esotericsoftware.kryo.io.Output.writeByte(Output.java:226)
        at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.locationtech.geomesa.features.serialization.WkbSerialization.serializeWkb(WkbSerialization.scala:44)
        at org.locationtech.geomesa.features.serialization.WkbSerialization.serializeWkb$(WkbSerialization.scala:42)
        at org.locationtech.geomesa.features.kryo.serialization.KryoGeometrySerialization$.serializeWkb(KryoGeometrySerialization.scala:14)
        at org.locationtech.geomesa.features.kryo.impl.KryoFeatureSerialization$KryoGeometryWkbWriter$.apply(KryoFeatureSerialization.scala:229)
        at org.locationtech.geomesa.features.kryo.impl.KryoFeatureSerialization.writeFeature(KryoFeatureSerialization.scala:71)
        at org.locationtech.geomesa.features.kryo.impl.KryoFeatureSerialization.serialize(KryoFeatureSerialization.scala:43)
        at org.locationtech.geomesa.features.kryo.impl.KryoFeatureSerialization.serialize$(KryoFeatureSerialization.scala:41)
        at org.locationtech.geomesa.features.kryo.KryoFeatureSerializer$MutableActiveSerializer.serialize(KryoFeatureSerializer.scala:75)
        at org.locationtech.geomesa.index.api.WritableFeature$FeatureLevelWritableFeature.$anonfun$values$2(WritableFeature.scala:153)
        at org.locationtech.geomesa.index.api.package$KeyValue.value$lzycompute(package.scala:183)
        at org.locationtech.geomesa.index.api.package$KeyValue.value(package.scala:183)
        at org.locationtech.geomesa.accumulo.data.AccumuloIndexAdapter$AccumuloIndexWriter.$anonfun$write$1(AccumuloIndexAdapter.scala:397)
        at org.locationtech.geomesa.accumulo.data.AccumuloIndexAdapter$AccumuloIndexWriter.$anonfun$write$1$adapted(AccumuloIndexAdapter.scala:396)
        at scala.collection.immutable.Vector.foreach(Vector.scala:1895)
        at org.locationtech.geomesa.accumulo.data.AccumuloIndexAdapter$AccumuloIndexWriter.write(AccumuloIndexAdapter.scala:396)
        at org.locationtech.geomesa.index.api.IndexAdapter$BaseIndexWriter.write(IndexAdapter.scala:153)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter.writeFeature(GeoMesaFeatureWriter.scala:50)
        ... 34 more

The error occurs on both older and newer versions of Geomesa. I have tried installing the latest versions of some key libraries, such as GeoTools and Kryo, but the problem persists.
Could it make sense to change the way the data is serialized? If so, can it be done without recompiling the code?
Comment: I've created a ticket to track the issue here: https://geomesa.atlassian.net/browse/GEOMESA-3289
Comment: Here is an example shapefile that throws the error https://files.fm/u/raeahqpku
Comment: GeoTools opens that file with no issues
Comment: @IanTurton The error is triggered when you try to ingest it with the command geomesa-accumulo ingest --force -i accumulo -z zookeeper -u username -p myPassword -c myCatalog -f myFeature  /path/to/shapefile
Answer: The issue is a bug in GeoMesa affecting schemas with > 479 attributes. It has been fixed in main, and will be available in the next release (4.1.0). In the meantime, you can build from source if you need the fix immediately.


Finding a generalized inverse

Question: I am trying to find a generalized inverse for all real $n \times n$ matrices such that it is 1 on the diagonal as well as 1 on the two entries directly above the diagonal, but 0 otherwise. Is there a "slick" way of doing it or does it amount to calculating until a pattern if found?
Answer: Yes, denote $J$ the matrix with 1 above the diagonal and $0$ otherwise.
$$X=I+J+J^2$$ $$X^{-1}=(I+J+J^2)^{-1}=(I-J)(I-J^3)^{-1}=(I-J)(I+J^3+J^6+\cdots)=I-J+J^3-J^4+J^6-J^7+\cdots$$
Note that $J^n=0$. The sum above has only finite terms which is not $0$.
Wish it helps.
Comment: $I=J^0$ and $J$ communicate with its power. Therefore, it's just derived from $(x^3-1)=(x-1)(x^2+x+1)$ and take $x=J$.
Comment: How do you justify $(I+J+J^2)^{-1}=(I-J)(I-J^3)^{-1}=(I-J)(I+J^3+J^6+\cdots)$?
Comment: Thank you for all the help!
Answer: You can use the classical method consisting in writing $A=I+N$ where $I$ stands for the identity and $N$ is nilpotent or order $n$ (check it, $N^{n}=0$). Then, Use Newton's binomial expansion on $A^{n}$ to obtain a formula for the inverse by gathering all powers of $A$ on one side and identity on the other side.

Finding a generalized inverse


Distribution of minimum of two uniforms given the maximum

Question: Let $X_1$ and $X_2$ be two random variables uniformly distributed on $(0, 1)$. It is easy to calculate the distribution of minimum and maximum of these two numbers:
$$
P[\max(X_1, X_2)<x] = x^2
$$
$$
P[\min(X_1, X_2)<x] = 1 - P[\min(X_1, X_2)>x] \\
= 1 - (1-x)^2 = 2x-x^2
$$
But what would be the distribution of the minimum, given that the maximum is given and equal to some number $z$, i.e.
$$
P[\min(X_1, X_2)<x | \max(X_1, X_2)=z] = ?
$$
Comment: @Alex, I added the derivation that I think is correct. In your case, didn't you miss the case that both of the random variables are smaller than $x$?
Comment: why is the first result $2x - x^2$? I keep getting $x(1-x)+x(1-x) = 2x - 2x^2$
Comment: OK thanks I got it
Answer: Good question, this had me stumped at first.  The minimum is actually the uniform distribution on [0,z].
Without loss of generality, let $X_1$ be at the maximum value z.
Then $P(X_2 > z) = 0$ since z is the max, and $P(X_2 \le z) = 1$.
Let Y be the event that $X_2 \le z$.  Then by Bayes rule.
$$
f_{X_2}(x|Y) = P(Y|X_2=x)f_{X_2}(x)/P(Y) \\
= 0 \quad for \quad x > z \\
= 1(1)/z \quad  for \quad x \le z \\
$$
Therefore the minimum is uniform on [0,z]
Comment: Thanks for that! Does it hold in the more general case where we have $N$ random variables $X_1$, ..., $X_N$, and we condition the distribution of one of them (which is not maximal) on the maximum of all these random variables being $z$?
Comment: New problem $\implies$ New question.


Interpreting a list traversal with condition?

Question: Can you please explain what does this statement mean?
  NewList = [{{U, S, R}, I} || {{U, S, R}, I} <- Users, U =:= User],

where USERS is a list.
Answer: It returns all elements of Users which 1) match the pattern {{U, S, R}, I} (i.e. are 2-tuples where the first element is a 3-tuple); 2) U in that pattern is equal to the value of User.
See the manual for details.
Comment: @user2384694: Just efficiency advice, use `NewList = [X || {{U, _, _}, _} = X <- Users, U =:= User],` which makes a lot less garbage on the heap and do less work.


Custom template for fields in user profiles

Question: I'd like to customize the fields in my user profiles. So far I've created field.tpl.php which of course applies to all fields on my whole website and not only to the fields in the user profiles.
How do I have to name the field.tpl.php? Is it field--user-profiles.tpl.php? That didn't work.
Answer: According to Drupal's theme hook suggestions for Drupal 7 the following suggested template names are available:

field--field-name--content-type.tpl.php
field--content-type.tpl.php
field--field-name.tpl.php
field--field-type.tpl.php

The content-type in this case is user (I guess it means entity type really), so to theme fields just for user profile pages, your template should be called:
field--user.tpl.php

I've just tested that on a dev site and it works perfectly.
Answer: You may write a template for whole form by altering form
I assume that you didnt use profile2 module. Your user profile(maybe account) page is at user/1/edit
first of all you need write alter function
function MYMODULE_form_alter(&$form, &$form_state, $form_id) {
  switch($form_id){
    case 'user_profile_form':
      //load an entity object
      $form['#theme'] = 'custom_profile_form';
    break;
}}

then you have to add own template for this form item
like
function MYMODULE_theme($existing, $type, $theme, $path){
  return array(
    // user profile 2 form
    'custom_profile_form'=>array(
      'render element'=>'form',
      'template'=>'custom-profile-form'
    ),

then you have to create custom-profile-form.tpl.php in your modules folder
in this template file you have to render some form specific variables
<form action="<?php print $form['#action']; ?>" method="<?php print $form['#method']; ?>"
        id="<?php print $form['#id']; ?>" accept-charset="UTF-8">
    <?php print drupal_render($form['form_build_id']); ?>
    <?php print drupal_render($form['form_token']); ?>
    <?php print drupal_render($form['form_id']); ?>
//here you have to render own elements

</form>

rendering fields like this will give field form elements without waste divs that core attaches
<?php print drupal_render($form['profile_profile']['field_status']['und'][0])?>

templates like field.tpl.php will effect only to field's formatter. not to widget
Comment: oh that looks good, but i don't know if that's necessary in my case. the above worked perfectly. but if i come to limitations (who knows), i will give your solution a try. thanks! :)


For a 10 year old learning piano what does music pedagogy says about using metronome?

Question: I am teaching piano to a 10 year old using the Suzuki 1 book.  This child has no problem maintaining tempo when singing.  Is it better for the child to use metronome to keep the tempo during practicing (she slows down during more complicated passage) or will using metronome can be counter productive to developing inner beat?
I would like references from music pedagogical literature on similar age group kid learning an instrument in the first few years.
Addendum: I should clarify that she has no problem playing the more complicated passages (having practiced them for weeks), but she still slows down and is not aware she does that until I points it out to her.  I also tried playing together with her, which helps, but obviously we can only do that during the lessons.
Comment: You have, in a way, answered your own question.  She slows down during the more complicated passages.  So its not the timekeeping, its the finding the notes that she is struggling with.  Tims answer hits the nail on the head
Comment: @JimM Perhaps I should clarify that she has no problem playing the more complicated passages (having practiced it for weeks), but still slow down and not aware she did that until I pointed it out to her.  I also tried playing together with her, which helps, but obviously only during the lesson.
Comment: That's some proof of what practice - repetition - can do. She's learned to slow down at that point through doing it lots of times! Now she needs to 'unlearn' the slowing down!
Answer: We've all slowed down during more complicated passages! Chances are she doesn't need a metronome to keep time. She just needs to play the better known parts  a little slower!
Metronomes have their place - although in 60+ yrs of playing and 50 odd yrs of teaching I've never been compelled to use or advocate use of one. With a fairly good sense of rhythm and tempo keeping, I'd say don't bother. Other teachers may well disagree - let's wait and see. Only when someone is wandering all over tempo-wise might it help, and I believe those players are rare. Playing to a metronome is, to an extent, soul destroying - literally to a degree, as unless it's dance music, it will fluctuate - and should fluctuate. Rant end!
Comment: I feel this isn't enough for a full answer, but I think having the metronome click on only every bar rather than every beat, is a good compromise between rhythmic accuracy and freedom/expression.
Comment: @awelotta - seems like an opportunity for another answer.
Comment: Thanks for your 2nd opinion.  I myself think the same way in my 40+ yrs of playing.  @awelotta's suggestion is excellent and I would prescribe it only to be used to verify once in a while (every 2-3 days).
Answer: We can argue about how to use the metronome while learning a piece.   But once you feel you HAVE learned it, I see no possible objection to checking that you can play it at a constant speed.
Answer: I would say that your student slows down because she has a problem with the theory, technique, coordination or fractions of that passage. A metronome could make things worse as it would force her to keep the tempo at the risk of sacrificing or glossing over something which could become hardwired improperly. A dozen lessons in theory, physics or math is order.
On the issue of metronomes . . . I advocate jogging. Our bodies have a wonderful sense of timing built into our two legs.  We only need to do our practicing while walking or jogging. Keeping time is more about feeling, going with and resisting gravity.  It is already within our bodies which we can transfer to our brains (heart and soul).  It doesn't need to be trained, only uncaged.  It is our birthright.  It is often muscular tension which gets in the way of timing and even playing.
Piano playing is not something separate from life or the laws of physics.  They are all one and we are merely the conduit unless . . . there is a break somewhere.
Comment: Wouldn't a desire to move our arms to facilitate walking or jogging get in the way of playing the string instrument better? I know I jog/run faster if my hands aren't in my pockets, regardless of how cold it is. If I were to play the violin or ukulele while jogging, I'd think the arm swinging or temptation to would get in the way.
Comment: You're probably correct, but the rhythm would at least be steady.
Answer: My thoughts on the matter are that metronomes are useful when practicing exercises and they probably help develop a feel for rhythm, but when performing a piece of music, we should be allowed to play what we feel and sometimes that may mean slowing down or speeding up. Don't get me wrong, I think uniformity can be an important skill, but I think flexibility and feeling certainly have a right to respect in our music also.
Answer: I have the same problem slowing down. I know what my problem is and more than likely hers. It is when I get to notes that I'm not as familiar with. I slow down to try to see what the notes are and their value. Although I practice the music for weeks. Maybe if you make sure she is very familiar with the notes she slows down on and have her study them more, she won't slow down and will be more confident.


How to use the Drupal flashcard module?

Question: I am not very familiar with Drupal modules.
I am using Drupal 7, and installed the Flashcard module. I enabled it, the flashcard-cycle module, and the libraries module; I also added the jquery plug-in as \sites\all\libraries\jquery.cycle\jquery.cycle.min.js.
I am getting the flashcard content type in structure/content types, but I don't know what else I have to do.
I need further instructions about what I need to do in order to use the flashcard module.  What is my next step?
Answer: Go to this content type's "Manage  fields" page and add new field with "Flashcard" type. Set number values you want (I think you will need Unlimited). Go to "Manage display" page and change your new field formatter from "Default" to "Cycle". Create a new node with a couple of Flashcard pairs.

How to use the Drupal flashcard module?


Is there a way to allow seasonality in regression coefficients?

Question: Say I have a time series, Gt, and a covariate Bt. I want to find the relationship between them by the ARMA model:
Gt = Zt + β0 + β1Bt
where the residual Zt follows some ARMA process.
The problem is: I know for sure that β0 and β1 varies with the time of the year. Yet I do not want to fit a separate model to each month because that introduces discontinuity into my time series, which means I cannot calculate the autocorrelation function of the final residuals.
So, is there a time series model (or family of models, I wonder) that allows the correlation coefficients of its covariates to change seasonally?
========================
Edit: Thank you for those who replied here. I decided to just use seasonal dummies, but got busy so failed to reply in time.
Comment: No this is not a dumb question, when you mean "change seasonality", do you mean seasonality changes over time and is not constant? if that is the case you need a model that handles stochastic seasonality, dummy coding will not work as it handles only deterministic seasonality. See my earlier [question](http://stats.stackexchange.com/questions/103193/stochastic-vs-deterministic-trend-seasonality-in-time-series-forecasting). Simply model $Z_t$ as ARIMA (p,d,q)(P,D,Q) this should do it.
Answer: Edit
(The same idea was proposed by Stephan Kolassa a few minutes before I posted my answer. The answer below can still give you some relevant details.)
You could use seasonal dummies. For simplicity I illustrate this for a quarterly time series. Seasonal dummies are indicator variables for each season. The $i$-th seasonal dummy takes on the value 1 for those observations related to season $i$ and 0 otherwise. For a quarterly series the seasonal dummies, $SD$, are defined as follows:
\begin{eqnarray}
SD = \left[
\begin{array}{cccc}
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1 \\
1 & 0 & 0 & 0 \\
\vdots & \vdots & \vdots & \vdots \\
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1 \\
\end{array}
\right] \quad
SDB = \left[
\begin{array}{cccc}
B_{1} & 0 & 0 & 0 \\
0 & B_{2} & 0 & 0 \\
0 & 0 & B_{3} & 0 \\
0 & 0 & 0 & B_{4} \\
B_{5} & 0 & 0 & 0 \\
\vdots & \vdots & \vdots & \vdots \\
B_{n-3} & 0 & 0 & 0 \\
0 & B_{n-2} & 0 & 0 \\
0 & 0 & B_{n-1} & 0 \\
0 & 0 & 0 & B_{n} \\
\end{array}
\right]
\end{eqnarray}
You can multiply each column in $SD$ by your explanatory variable $B_t$ and get the matrix $SDB$ defined above.
Then, you can specify your model as follows:
$$
G_t = Z_t + \beta_{0,s} SD_t + \beta_{1,s} SDB_t \,,
$$
where the index $s$ indicates the season. Observe that we now have four coefficients (12 in your monthly series) $\beta_{1,s}$, one for each column in $SDB$.
The same for the intercept $\beta_0$ except that we must remove one column in $SD$ in order to avoid perfect collinearity. In a monthly series you would include for example the first 11 seasonal intercepts in $SD$.
Fitting the model for example by maximum likelihood will give you one coefficient estimate for each season. You could also test whether $\beta_{0,s}$ are the same for all $s$ or similarly if $\beta_{1,s}$ are constant across seasons.
Comment: +1. Although you don't want to fit using Ordinary Least Squares if you have ARMA errors.
Comment: @javlacalle +1, Can we simply use $Z_t$ as ARIMA(p,d,q)(P,D,Q) instead of seasonal dummies to capture seasonality? That way you also account for stochastic seasonality in addition to deterministic seasonality. While this does not address the OP question on seasonality as regression coefficients it might be  worthwhile to highlight the difference.
Comment: @forecaster I think the pursue of the OP is to measure the influence of $B_t$ on $G_t$ at different seasons. This could be captured by allowing seasonally-varying coefficients, $\beta_{s,1}$. If $\beta_1$ is constant for all seasons then we are not able to measure the effect of $B_t$ at each season and test if differences are significant. Moreover, if $\beta_1$ is fixed, observing seasonality in the residuals could mean that there is a seasonal effect not captured by a single coefficient $\beta_1$, rather than the need to extend the model for $Z_t$ by means of a seasonal ARIMA model.
Comment: @Frank The intercept is set to zero for the season that is left out. The coefficients of the intercepts related to the remaining coefficients are interpreted as a change with respect to the average value of the deleted season (which is no necessarily zero, but the value determined by the coefficients and values of the remaining variables at that season).
Comment: @Frank If 11 colums are used in $SDB$, then you would in principle include a constant $\alpha$ (a column of ones); otherwise, the residuals may not be zero on average: $G_t = \alpha + Z_t + \beta_{0,s} SD_t + \beta_{1,s} SDB_t$. At the 12-th season (the one left out), the expected value of $G_t$ is $\alpha+\beta_{1,12}SDB_t$. The coefficients $\beta_{0,s}$, $s=1,\dots,11$ are interpreted as changes with respect to the estimate of $\alpha$.
Comment: @forecaster It depends on where the seasonal pattern comes from (from the influence of $B_t$, the structure of $Z_t$ or both). If $Z_t$ follows a seasonal pattern we should do what you mention. However, I left the term $Z_t$ aside because I think the OP is interested in the coefficients $\beta_0$ and $\beta_1$ and possible changes across seasons, probably due to a suspicion based on knowledge about the data.
Comment: You mention deleting a column of $SD$ for $\beta_0$. This is unclear to me. If you delete one column, then whenever that season comes up won't you have $G_t = \beta_1,s B_t + Z_t$? Doesn't this mean you set the intercept to zero for season $s$?
Comment: @javlacalle Thank you for the reply! If you leave out that season, you get $G_t = \beta_{1,s}B_t + Z_t$, and if $B_t = 0$ than $G_t = Z_t$ and $E(G_t) = 0$. So I think that you are saying $0$ is fine because the results are interpreted as a change in value from the left out season. In that case, how would you predict the true $G_t$? Would it be the expectation of the left out season * $(1+G_t)$?
Answer: Certainly there is. Simply include monthly dummies in an interaction with $B_t$. Let $M_{tm}$ denote a dummy that is 1 if time $t$ corresponds to month $m$ and 0 otherwise. Then fit the following regression with ARMA errors:
$$ G_t = \beta M_{t\cdot} + \gamma B_tM_{t\cdot} + Z_t $$
where $Z_t$ is ARMA(p,q) and $\beta$ and $\gamma$ are parameter vectors of length 12.
You can do the actual fitting using R with the nlme package, using the gls() function and specifying a corARMA() correlation structure.
Comment: @Frank: if we have too little data to support a complex model, then I personally would look to regularization, like the lasso, the elastic net or Bayesian approaches.
Comment: @Frank: yes, both should have 12 terms *since there is no intercept*. If you remove one term, say $\beta_1$, that means that the mean in month $1$ for $B_t=0$ should be zero, which will usually not make sense. Alternatively, you could include an intercept and a term for $B_t$ as a main effect (without an interaction with $M$), then leave one entry out of both $\beta$ and $\gamma$ - which would give you $1+1+11+11=24$ parameters, exactly as many as the model I propose. It's just a reparameterization. The model you propose in your comment works (assuming a deterministic $f$).
Comment: They should be, yes
Comment: What if you don't have many data points and want to preserve parameters? Is there a way to subtract a season while keeping parameters to a minimum?
Comment: Thank you for replying on such an old question. May I ask, should $\beta M_t$ and $\gamma B_t M_t$ each have 12 terms? Or should $\beta M_t$ have 11 terms? I learned about "the dummy variable trap," but I cant find a reference that clearly discusses this case.

For example, would this model work? Or do I need to decrease the length of $\beta$ vector by 1?

$Y_t = \beta M_t + \gamma B_t M_t + f(t) + Z_t$
Comment: Thanks!
In that case, are all of these models equivalent?
$\\
Y_t = \sum_{i=1}^{12} SD_i * [\beta_i +\gamma_i x_i] + \epsilon_t = \\
Y_t = \beta_0 + \sum_{i=1}^{11} SD_i * \beta_i + \sum_{i=1}^{12} SD_i \gamma_i x_i + \epsilon_t = \\
Y_t = \beta_0 + \gamma_0 x_i + \sum_{i=1}^{11} SD_i * [\beta_i + \gamma_i x_i] + \epsilon_t \\
\text{ Where} SD_i \text{ is a indicator variable which is $1$ at season $i$,}\\
\text{ $\epsilon$ is the error, $\beta$ and $\gamma$ are parameters, }\\
\text{$x_i$ is an external regressor, and $Y_t$ is the dependent variable}$
Answer: If you don't want to discretise the seasonal effect, you could assume that the regression coefficients vary in a cyclic manner as a function of the time of year, i.e. $\beta_0(t) = w_0 + w_1\sin nt + w_2\cos nt$ and $\beta_1(t) = w_3 + w_4\sin nt + w_5\cos nt$, then if you substitute these into your linear model, you should get something of the form
$G_t = Z_t + w_o +  w_1\sin nt + w_2\cos nt + w_3B_t + w_4B_t\sin nt + w_5B_t\cos nt$
You could fit this model by using OLS regression (or whatever method you are already using) with the additional covariates $\sin nt$, $\cos nt$, $B_t\sin nt$ and $B_t\cos nt$, where $n$ is whatever constant you need to represent a year ($2\pi/365$ for a daily time-series).
This wouldn't introduce any discontinuities in the model as the seasonality in the regression coefficients are smooth functions of time.  I suspect if you added sine and cosine components representing harmonics of the annual cycle you could model deviations from simple sinusoidal variation in the regression coefficients (Fourier series type approach).
Caveat: Been a long day, so I may have made a stupid error somewhere.
Comment: As far as I understood, the concern of the OP was with discontinuities in the residuals, fitting 12 regression models (one for each month) will lead to 12 series of residuals instead of one series of residuals where to carry out some diagnostics looking at their autocorrelations. Both the 0-1 dummies and the trigonometric dummies would be an appropriate way to deal with this issue. Which one is a more natural approach would depend, as you say, on the purpose of the analysis and the kind of information that is wanted.
Comment: (+1) A trigonometric approach is an interesting alternative. Another appeal of the trigonometric approach is that it may require fewer parameters. Your equation uses 6 parameters against 11+12=23 in the approach that I discussed in my answer. In practice we would probably need to include in addition to the fundamental seasonal frequency ($2\pi/12$ in a monthly series) some of its harmonics, which will require more parameters. But we may get a reasonable fit without including all the harmonics and hence the number of parameters to be estimated can be reduced.
Comment: A disadvantage that I see is that the interpretation is less straightforward in the context of a regression model. The interpretation of the 0-1 seasonal dummies
can be made in terms of months rather than cycles of seasonal periodicity. We may conclude for example that the effect of temperature on sales of a certain product is the highest in August and has no major effect in March. In the trigonometric approach we would conclude for example that the effect of temperature on sales follows a cycle that is repeated every 6 months. The former interpretation may be more informative.
Comment: You could still do that with this approach, you could plot the variation in each $\beta_0$ and $\beta_1$ by a weighted sum of the sine and cosine components, and you could discretise that to see how sales vary by month.  The original question suggested that discontinuities were not wanted, which implies a smooth variation.  At the end of the day, the right approach depends on what it is you are trying to find out.
Comment: Let's underline that the question is general and only the tag `econometrics` discloses the OP's interest in that side. For environmental time series data the trigonometric approach is often highly successful and natural, while conversely months have little or no meaning even if the data are reported in that way.
Answer: Fit the mean and the harmonics of the seasonal cycle to the time series of x and y. These provide the intercept terms. Then, subtract them from x and y to create anomalies. Use these anomalies x' and y' to compute seasonally varying regression slope coefficients: Fit the array product between the x' and y' with the mean and leading harmonics to the seasonal cycle. Do the same for the variance of the x'. Then divide the seasonal cycle fit to the covariance by the seasonal cycle fit to the variance to provide continuously evolving slope coefficients. For details, see
http://onlinelibrary.wiley.com/doi/10.1002/qj.3054/full