Refreshing AJAX Forms in Drupal 7

Question: My dynamic Competition form has two basic parts:
1- Basic Info

Name
Size Dropdown

2- Ajax Fields

Options dropdown based on Size

When I change the value in Size, a new combobox is loaded via AJAX with related values.
Here is part of my form:
if ($competition->size == 0)
    {
        $form['country'] = array(
            '#type' => 'select',
            '#title' => t('Country'),
            '#weight' =>3,
            '#required' => FALSE,
            '#zurb_class'=> 'cleared',
            '#default_value' => $competition->country_id,
            '#options' => _ajax_countries_dropdown_options(),
            '#prefix' => '
',
            '#suffix' => '
',
        );
    }
    else if ($competition->size == 1)
    {
        $form['continent'] = array(
            '#type' => 'select',
            '#title' => t('Continent'),
            '#default_value' => $competition->continent_id,
            '#weight' =>2,
            '#required' => FALSE,
            '#options' => _ajax_continents_dropdown_options(),
            '#prefix' => '
',
            '#suffix' => '
',

        );
    }
    else
    {
        $suffix = '

';
    }

    $form['size'] = array(
        '#type' => 'select',
        '#title' => t('Size'),
        '#default_value' => $competition->size,
        '#weight' =>1,
        '#options' => $options_size,
        '#required' => FALSE,
        '#ajax' => array(
            'callback' => 'ajax_competition_size_dropdown_callback',
            'wrapper' => 'dropdown-competition-size-replace',
            'method' => 'replace',
            'effect' => 'fade',
        ),
        '#suffix' => $suffix,
    );

Ajax Callback
function ajax_competition_size_dropdown_callback($form, $form_state) {
    $size = $form_state['values']['size'];

    if ($size == 0)
    {
        return array(
            '#name' => 'country',
            '#type' => 'select',
            '#title' => t('Country'),
            '#weight' =>3,
            '#required' => FALSE,
            '#zurb_class'=> 'cleared',
            '#default_value' => $competition->country_id,
            '#options' => _ajax_countries_dropdown_options(),
            '#prefix' => '
',
            '#suffix' => '
',
        );
    }
    else if ($size == 1)
    {
        return array(
            '#name' => 'continent',
            '#type' => 'select',
            '#title' => t('Continent'),
            '#default_value' => $competition->continent_id,
            '#weight' =>2,
            '#required' => FALSE,
            '#options' => _ajax_continents_dropdown_options(),
            '#prefix' => '
',
            '#suffix' => '
',
        );
    }
    else
    {
        return '

';
    }
}

Steps I made so far:

When I change the first select when editing, the values are not saved.
Debugging Backend
The form changes on the front (AJAX), but it does not change on the backend
Apparently the problem is I am creating the form fields based on size, Drupal will only recognize this fields. So when I change the select and load the second, it won't parsed by Drupal
I tried to use $form_state['rebuild'] = TRUE; but it didn't fix anything, actually avoided the redirection after submission.
Answer: The Ajax callback should only return the named FAPI element that is to be replaced. For example:
function questionaire_4D_selector_form_ajax_callback($form, $form_state) {
  // If the form has errors we show them in the Preview spot. There's not much
  // else we can do because jQuery is replacing out the specified
  // target element in the #ajax FAPI element.
  if (form_get_errors()) {
    return $form['preview-content-area']['esci-questionaire'];
  }
  return $form['preview-content-area']['esci-questionaire'];
}

Your form_builder function which build/rebuilds the form should examine the $form_state and alter the form as necessary based on the ajax submitted values:
/** Our form builder function */
function questionaire_4d_selector_form($form, &$form_state) {
 # for ajax inspection goto /tmp and view file output.
 #dd($form_state);
 #dpm($form_state);

  // This form responds to an AJAX submit button, and also a $_GET argument.
  $form_passed_get_parameter_questionaire_id = $form_state['submitted'] == FALSE &&
          count($form_state['build_info']['args']) &&
          ctype_digit($form_state['build_info']['args'][0]);
  #dpm($form_passed_get_parameter_questionaire_id ? 1 : 0);
  if ($form_passed_get_parameter_questionaire_id) {
    $id = $form_state['build_info']['args'][0];
  } else {
    $id = !empty($form_state['values']['questionaire_id']) ? $form_state['values']['questionaire_id'] : '';
  }
  #dpm($id);
  $form['questionaire_selector'] = array(
    '#type' => 'fieldset',
    '#title' => t('Select 4D Questionaire'),
  );
  $form['questionaire_selector']['access_id'] = array(
    '#type' => 'hidden',
    '#value' => drupal_get_token(QUESTIONAIRE_4D_PREVIEW_AJAX_SECRET_VALUE),
  );
  $form['questionaire_selector']['questionaire_id'] = array(
    '#type' => 'textfield',
    '#title' => t('Questionaire Number'),
    '#default_value' => $id,
    '#element_validate' => array('element_validate_integer_positive'),
    '#size' => 15,
  );
  $form['questionaire_selector']['submit'] = array(
    '#type' => 'submit',
    '#value' => t('Preview 4D Questionaire'),
    '#ajax' => array(
      'callback' => 'questionaire_4d_selector_form_ajax_callback',
      'wrapper' => 'questionaire-preview',
      'method' => 'replace',
    ),
  );
  $form['preview-content-area'] = array(
    '#type' => 'fieldset',
  );
  $form['preview-content-area']['esci-questionaire'] = array(
    '#prefix' => '
',
    '#suffix' => '
',
    '#type' => 'markup',
    '#markup' => '
Your Questionaire Preview will appear here.
',
  );

  // This says if we were given an remote 4D form id of say 23456, load it into the supplied FAPI
  // element.
  if ($form_passed_get_parameter_questionaire_id) {
    $remote_response = retrieve_questionaire_4d_response($id);
    if (strpos($remote_response->data, 'Error',0) === 0) {
      form_set_error('questionaire_selector', "Remote 4D Error
" . $remote_response->data);
    }
    $form['preview-content-area']['esci-questionaire']['#markup'] = questionaire_4d_selector_build_esci_questionaire_preview($id, $remote_response->data);
  }

  $form['#attached']['css'] = array(
    drupal_get_path('module', 'questionaire_4d_preview') . '/assets/questionaire_4d_preview.css',
  );
  return $form;
}

/* Implements hook_menu() */
function questionaire_4d_preview_menu() {
  $items['admin/config/esci-online/questionaire-4d-preview'] = array(
    'title' => '4D Questionaire Preview',
    'description' => 'View a preview of a remote 4D questionare.',
    'page callback' => 'drupal_get_form',
    // I'm not sure we have to, but explicitly pass in the 4th
    // argument on the URL as the questionairre ID. This is added
    // to build_info of form automatically.
    'page arguments' => array('questionaire_4d_selector_form', 4),
    'type' => MENU_CALLBACK,
    'access arguments' => array('administer esci'),
  );
  return $items;
}

Refreshing AJAX Forms in Drupal 7


Perform math equation inside of a loaded text file

Question: I'm currently working on a simple little application that keeps track of wins and losses for any sort of game. I'm storing the win/loss count in separate text files as single numbers. What I want the program to be able to do is look into the specified text file, and simply add 1 to the existing number. For example, if the entire text file is simply "0" and I input "win" into the application, it will perform 0+1 and change the text file permanently to the result.
Here's what I've got so far:
ld = open("data/lossdata.txt", "r+")
wd = open("data/windata.txt", "r+")
hlp = open("data/help.txt", "r+")
losread = ld.read()
winread = wd.read()
helpread = hlp.read()
to_write = []

print("Welcome to Track Lad. For help, input \"help\"\nCurrent Win Count: "+winread+"\nCurrent Loss Count: "+losread)
inp = input("Input: ")

if inp == "w" or inp == "win":
    for line in wd:
        num = int(line) + 1
        to_write.append(num)
        wd.reload()
    wd.seek(0)
    for num in to_write:
        wd.write(str(num)+'\n')
    wd.close()
    print("New Win Count: "+winread+"\nLoss Count: "+losread)
    input("")
elif inp == "l" or inp == "loss":
    ld.write("1")
    print("New Loss Count: "+losread+"\nWin Count: "+winread)
    input("")
elif inp == "help":
    print(helpread)
    input("")
else:
    print("Invalid input, try again.")
    input("")

Everything I've done so far is in the first if statement. I'm not getting any error when I run the code, even when I input "w", but the number in the text file doesn't change. Thanks in advance for any help, and I'll stay on the page to answer any questions that may help you figure out what's wrong.
Comment: Are you using pycharm by any chance?
Answer: I would recommend using a single file database like SQLIte instead of separated text files. You also can register all wins and losses (with timestamp, if you needed later).
import sqlite3

db = sqlite3.connect('winloss.db')
cursor = db.cursor()
cursor.execute('''
    CREATE TABLE IF NOT EXISTS winloss (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        t TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
        value TEXT
    );
''')
db.commit()

def add_value(db, value):
    cursor = db.cursor()
    cursor.execute("INSERT INTO winloss(value) VALUES(?)", (value, ))
    db.commit()

def add_win(db):
    add_value(db, "win")

def add_loss(db):
    add_value(db, "loss")

def count_value(db, value):
    cursor = db.cursor()
    cursor.execute("SELECT COUNT(*) FROM winloss where value=?", (value, ))
    return cursor.fetchone()[0]

def count_win(db):
    return count_value(db, "win")

def count_loss(db):
    return count_value(db, "loss")

if __name__ == '__main__':
    print "now"
    print "win:", count_win(db)
    print "loss:", count_loss(db)
    print "adding values"
    add_win(db)
    add_win(db)
    add_loss(db)
    print "win:", count_win(db)
    print "loss:", count_loss(db)

And it is easier to read and understand
Answer: As you seem to have very small data in your lossdata.txt and windata.txt, I would preferably do the following :

read the entire file content and store the data into variables (open the file in readmode, read data, then close the file),
overwrite the entire file content with the new value (open the file in writemode, write data, then close the file)

If you need to update a particular line of your file, know that it's better to make a copy of your input file and create the updated file as a new file. This is exactly what fileinput.input(inplace=1) does as noted in this SO answer.
So, try to do something like :
import fileinput

def process(line):
    return int(line) + 1

for line in fileinput.input(inplace=1):
    print process(line)


Upper triangular matrix and Lebesgue measure.

Question: Let $M$ be a $n\times n$ upper triangular matrix s.t.  diagonal components are $1$. $\Bigg($For example, $M=\begin{pmatrix} 1 & 3 & 2 \\ 0 & 1 & 5 \\ 0 & 0 & 1 \end{pmatrix}, \text{when } n=3.$$\Bigg)$.
And let $\mu$ be Lebesgue measure on $\mathbb{R^n}$ and $E \subset\mathbb{R^n}$ be a Lebesgue measurable set.
Then, prove that $\mu(ME)=\mu(E)$, where $ME=\left\{ Mx \in \mathbb{R^n} \mid x\in E \ \right\}$ by using Fubini-Tonelli Theorem.
I know that $\mu(ME)=\displaystyle\int_{\mathbb{R^n}} \chi_{ME}(x) dx$ and $\mu(E)=\displaystyle\int_{\mathbb{R^n}} \chi_{E}(x) dx$ but I don't come up with an idea how I can prove these are equal using Fubini-Tonelli.
This is a step in eventually proving that $\mu (ME) = |\det M| \mu(E)$ for any linear map $M$.
Comment: Or [this one](https://math.stackexchange.com/questions/52161/measure-of-image-of-linear-map)
Comment: Well, they don't use Fubini though.
Comment: @ArcticChar Actually, this is the step of proving the fact that $\mu(ME)=|\det E| m(E)$ for all regular matrixes $M$ so I mustn't use $\mu(ME)=|\det E| m(E)$.
Answer: Let $U$ be the set of such matrices. Geometry is the name of the game here, so consider what happens when you apply $M \in U $ to $[0,1]^n$.
This amounts to observing $M$'s effect on the $e_i$, which produces a vector of the form:
$$\begin{bmatrix} 1 + \sum_{j=2}^n a_{1,j} \\ 1 + \sum_{j=3}^n a_{2,j} \\ \vdots \\ 1 \end{bmatrix}$$
More generally:
$$Mx = \begin{bmatrix} x_1 + \sum_{j=2}^n a_{1,j}x_j \\ x_2 + \sum_{j=3}^n a_{2,j}x_j \\ \vdots \\ x_ n \end{bmatrix}$$
where $x = (x_1,\dots,x_n)$.
In other words, $M$ sends $x_n$ to $x_n$, $x_{n-1}$ to a linear combination of $x_n$ and $x_{n-1}$, and overall $x_i$ is sent to a linear combination of the $x_j$, $i \leqslant j \leqslant n$. One way to think of this is as a descending chain of subspaces.
In the $n=2$ case, this is just sending the square to a parallelogram.
As is the case with multivariable analysis, the $n=2$ case is notationally the easiest, but in this case is also conceptually the simplest.
Proposition: $\mu(MS) = \mu(S)$, when $M \in U \subset \mathbb{R}^{2 \times 2}$ and $S \subseteq \mathbb{R}^2$.
Proof: The proposition is found in the following series of equalities:
\begin{align*}
\mu(MS) &= \iint_{[a,b] \times [c,d]} \chi_{MS}(x,y) d(x,y)\\
&= \int_c^d \int_a^b \chi_{MS}(x,y)dxdy \tag{Fubini}\\
&= \int_a^b \left( \int_c^d \chi_{MS}(x,y)dy\right)dx \tag{Fubini again} \\
&= \int_a^b \left( \int_c^d \chi_{S}(x,y)dy \right) dx \tag{since $M$ is $\text{id}$ on $y$}\\
&= \iint_{[a,b]\times [c,d]} \chi_S(x,y)d(x,y) \tag{Fubini *again*}
\end{align*}
$\blacksquare$
The rest is below. Take a moment to think about how you could extend this technique before proceeding.

However, what is a $3D$ parallelopiped but uncountably many $2D$ sets "glued" together? This suggests induction, since we just proved that the $2D$ measure is invariant under $M \in U$.
Proposition: Let $M \in U \subset \mathbb{R}^{n \times n}$ and $S \subseteq \mathbb{R}^n$. Then $\mu(MS) = \mu(S)$.
Proof:
We prove by induction on $n$, the dimension of the space. The base case $n=1$ is just $\mu(S) = \mu(IS)$, which is clear.
Consider $n=k$. Then:
\begin{align*}
\mu(MS) &= \idotsint_{\prod_{i=1}^k[a_i,b_i]} \chi_{MS}(x_1,\dots,x_k) d(x_1,\dots,x_k)\\
&= \int_{a_1}^{b_1} \left(\idotsint_{\prod_{i=1}^{k-1}[a_i,b_i]} \chi_{MS}(x_1,x_2,\dots,x_k)dx_2\dots dx_k\right)dx_1\\
&= \int_{a_1}^{b_1} \left( \idotsint_{\prod_{i=1}^{k-1}[a_i,b_i]} \chi_{S}(x_1,x_2,\dots,x_k)dx_2\dots dx_k \right) dx_1 \\
&= \idotsint_{\prod_{i=1}^k[a_i,b_i]} \chi_{S}(x_1,\dots,x_k) d(x_1,\dots,x_k)\\
&= \mu(S)
\end{align*}
$\blacksquare$
The fact that this works can be seen as a consequence of the fact that the last $n-1$ rows and columns of an $n \times n$ upper triangular unital-diagonal matrix is in fact also such a matrix. That submatrix "doesn't touch" the first variable, so induction is nicely contained.


Editing / customising Cart blocks in Magento 1.8.1

Question: Hi I am trying to move some of the blocks around on the cart page, and have looted in loads of files and cant seem to find what I am looking to edit.
I want to reorganise the 5 blocks that display under the list of what is actually in the cart.
I have a Template Path Hints so I can see what blocks are there and where they are located, but I am beginning to wonder if this is more complicated than I first thought.
Does anyone have any suggestions on how I can tydy up this part of the page?
Answer: Please follow following steps-
1- find layout file in magento theme checkout.xml.
2- Find block name which you want to move.
3- Use insert after/before magento XML tag to move these blocks accordingly.
Note: kindly backup your files before editing.


Learning to create melodies

Question: I've researched my question on Google before and came across a few threads which I've read a couple of times. So far I've got into music theory and understand scales, how chords are formed, I know what intervals are in terms of half steps apart (e.g. maj2nd - whole step) but I can't seem to tie the information to create my own melodies.
I've seen the same terminology so am learning the relevant material but just can't seem to really know how to create a melody.
To me, a melody is a series of notes put together which sounds pleasing to the ear. The fact there are 12 notes in an octave and people make endless amazing melodies really inspires me, I've been reading a lot into music theory and would love to start creating music now. Any advice is always appreciated.
The one rule I'm going to stick to in terms of simplicity sake is to remain within a chosen scale so I know what notes to press but how do I learn what notes sound good together - is there any theory there which I can apply?
I've also read that making melodies is something that cannot be learnt - so how does music theory apply to it? I want to focus my efforts on creating melodies and building instrumentals from my knowledge so far and to build skills.
Comment: The creation of melodies is learned in the same way that any skill is learned, by practicing and experimenting. One does not learn an instrument simply by reading theories on how to play the instrument, you need to get the instrument under your hands and practice. Same thing applies for melodies, you can read about theory which I recommend and will greatly assist you, but you have to practice. Creating melodies can certainly be learned.
Comment: Creating melodies can be learned, so that bit of information is wrong. You do however, need to provide more information. Do you play any instruments or are you trying to create music using some sort of piano roll program?
Comment: Ok thanks, I've read on forums such as gearslutz attacking people asking this very question.

Sorry for my lack of background -- I've got a midi keyboard and want to create instrumentals and use Cubase as my DAW
Comment: Well we're not here to attack we are here to help, but that bit of information is wrong. I ask again though, do you play any instruments? I do need this bit of information before I can answer.
Comment: Keyboard :) Thanks, I've browsed through some relevant threads and there's good information on this forum.
Comment: I'm glad you both answer with stuff I've already covered and I had the clue that I do have the fundamentals down but need to put it into practice now. My plan was exactly to stay within a certain key and build an instrumental track around that. Really, it seems you can't learn to make melodies - just practice playing with notes within a specific key. When do intervals apply? Also, experimentation with playing the keyboard will teach me what notes sound good together, right? Thanks all for the responses :)
Comment: Can you provide some guidelines/ideas of how to get theory can help specifically? Other than that, I think you're right it is just with practice. Thanks
Comment: The theory can help you figure out what might sound good, but ultimately all you can do is try things and see. The composers I know sit down (or stand up) with their instruments and just play stuff. Sometimes they've had an idea in their head for a while and they try to put that onto the instrument. Other times they just mess around until they find something they like. Theory is more important for figuring out how to turn it into a full piece of music, especially if it will have multiple parts, chordal accompaniment etc.
Comment: There are, and have been, many, many  players who create great tunes. A lot of them have no theory knowledge to speak of. Some guys I have played with need to ask what the chord or note is that they're playing. It doesn't stop them producing some wonderful melody lines, chord patterns, etc. In fact, part of me, as someone who knows theory, says that this is precisely the reason they can do it so well - not bogged down with theory - which after all, doesn't tell anyone what to do, but tries to explain how it works after the event. Using theory is a little like painting by numbers!
Answer: Creating melodies or a melodic line is not much different than creating a figure in drawing.
I see two ends of a spectrum of choices with everything in between as possible.
1) derivative
2) original

I. Derivative melodies are a subjugation of environmental influences literally and figuratively. Examples include the following:
A melody based on songbirds.
A melody based on car horns in rush hour traffic.
A melody or texture based on the ocean surf.
A melody based on a woman talking (often used by Blues guitar leads).
A melody based on machinery, "Ballet Mécanique" by George Antheil.
A melody based on fire, wind, or rain, "The Four Seasons" (Italian: Le quattro stagioni)...a set of four violin concertos by Antonio Vivaldi, or "Sonic Seasonings" by Wendy Carlos formerly Walter Carlos.

II. Original melodic content is based purely on imagined sounds.
Examples include the following.
Improvised on the spot and in the moment such as melodies by Classic North Indian Raga Singers, and many Jazz and New Music composers/performers.
A melody where sound is totally aleatoric. (John Cage 4' 33")
A melody that accomplishes a specific compositional goal, e.g. 12 tone atonal music.
A melody where technical skills permit pitch organization in a super human way requiring mechanical and/or electronic and/or computer engineering such as the electronic music of Edgard Varèse, Karlheinz Stockhausen, Maggie Payne, The HUB, David Behrman and many others.

III. Somewhere between these two extremes:
A melody that furnishes a good thread to connect harmonic content or based on a chord progression, most of Jazz.
A melody that defines a musical form such as counterpoint.

While the above is brutally simplified, it allows a wide set of guidelines to permit almost anything to be constructed into melodic content.
Music theory from any culture, study of scales, study of musical literature all have a place in assisting the composer in sculpting a melody, but are as confining as using a ruler or French Curve for the artist. These things are best used as guidelines and measurements for proportion and compositional form but can not be greater than the emotional content endowed by the composer and brought to life by the performer.
Regarding "melodies is something that cannot be learnt".
If one can learn to walk then one can choose to journey.
If one can learn to talk then one can be heard.
If one can listen then one can learn from others.
If one can read and write then one can compose letters and understand those sent.
If one can make art (music, dance, figures) then one can connect from one heart to all.
Composing is simply making choices whether you are looking at melodic content or a symphony. The more you learn what options you have to choose from the better you will become with making your idea real.
Comment: @ filzilla - aleatoric is a dicey way of describing it. His follow-up, 9:06 , was twice as good...
Comment: @filzilla https://www.youtube.com/watch?v=HilGthRhwP8
Comment: @filzilla Why don't you consider it aleatoric? In a certain light, all performed music is aleatoric in that the performer is endowed with at least a modicum of choice in how to interpret the work. In Mozart this is manifest in the choice of tempo, vibrato, etc., and at the other extreme is 4'33". In my opinion there is a category of music nearer 4'33" than Mozart which is conventionally called "aleatoric" to which this piece by Penderecki belongs, especially because its "original melodic content" is a superposition of the semi-random choices of many performers arranged in a totally random way.
Comment: In what way is 4'33" aleatoric?
Comment: @ninemileskid 4'33" as you know is all 'rests', and there-fore the only sounds that occur are ambient, which is completely random.
Comment: @Tim 4'33" is better known so it serves as a more familiar example but thanks for chiming in.
Comment: This was, actually,a jest...  Although a silence for twice as long has to be twice as good ...
Comment: If by "totally aleatoric" you mean that the composer cannot specify *any* musical structure, then 4'33" is a good example, and perhaps the only one. (But how would this answer the OP's question?) Based on the other examples in your category II, I think you mean aleatory where the composer can still specify *some* degree of structure, in which case I wonder if a more deliberate example better serves your point, e.g. Threnody.
Comment: @Tim. There really is a 9:06 Tim (at least it last 9:06), but +1 for the jest of it. Here's a link, this one has sound unlike its half brother:http://www.allmusic.com/performance/first-construction-in-metal-for-6-percussionists-mq0000612101
Comment: @ninemileskid, Well actually Cage implemented rests which is giving instructions, and form. So like most of John Cage's work there is room for chance and structure. Please tell which Threnody you are referring to?
Comment: @ninemileskid +1 for a great find, I really love the animated score. Certainly one of the great pieces of the 20th Century by the expert of tone clusters, Krzysztof Penderecki. It certainly came from his imagination. While this is not aleatoric it absolutely belongs in the category of original, and should be included in that category. Please note my examples are not complete, these are simply some vague notions near the ball park. :>
Comment: @ninemileskid Thank you for bringing this to my attention. By your definition, all performances have some aleatoric process in the act of interpretation as does theater, dance, cinema, etc. I am drawing the line here that there are composers like John Cage that specifically intend some or all of the design to be decided by a chance process. I stand corrected on Penderecki's piece, yes I agree there are instructions that belong to chance. Thank you.
Comment: This is fascinating--I've always understood aleatoric to mean that choice is taken not only out of the hands of the composer but also the hands of the PERFORMER.  If "alea" comes from the Latin for "die/dice," then strictly speaking aleatoric music ought to be music in which choices (or choices in some specified subset) are determined not just by forces other than the performer (or, if my understanding is incorrect, by the composer) but by RANDOM forces, yes?
Comment: @Joel Derfner, Strictly speaking you are probably correct, but you will find many times in music that there can be a lot of 'grey' area. Remember music is an art, not a science although there are scientific principles at work. Consider too other terms that may or may not apply: stochastic processes, accidental, unplanned, fortuitous, desultory, chaotic, serendipitous... all of these may border to some degree on to the random landscape. To what end, may be just splitting hairs, but as humans tend to favor structure this may help define the boundaries more clearly.
Comment: This comes up all the time in music history class. Music in which aspects of performance are not controlled by the composer is "indeterminate." Music in which aspects of the compositional process itself are not controlled by the composer is "aleatoric." (Alea, as noted above, is the Latin word for dice, i.e., chance.) Music can be indeterminate but not aleatoric (Mozart concerto where the cadenza is left to the player); it can also be aleatoric, but not any more indeterminate than other music (imagine a piece whose notes and rhythms are chosen by rolling the dice, but is then written out).
Answer: As you listen to music analytically (especially music you like), there are various patterns and things you'll discover, which will help you learn how to write melodies.
For example, it's obvious that in many pieces, a melodic phrase spans a number of bars that's divisible by the number 4. If you can pin down the 4 parts of some such melodies (that you like, of course), you can figure their role in the melody. I noticed rather early, for example, that I really favored melodies where the first and the second part are the same, except for the last note. You may find some such patterns that work for you, as well.
Also, as you might have noticed already, movements in semitones are great for conveying sadness. Use that!
Or, if that's not your cup of tea, throw away those and stick with the pentatonic scale. It has a very unique and recognizable style and songs that use it are quite often described as "awesome".
Work out how the chord progression (and the expectations it creates to the listener) works together with your melody. For example, the same melody has an entirely different effect if it ends in the I chord (expected) than the same melody would if it ended in the VI chord ("deceptive"). And a melody can raise expectations if it rests in a V chord for a while - it often implicitly promises an equal amount of melody coming soon.
Perform mutations on shorter melodies. For instance, try repeating the chords of the melody, but in strategic places pick a different note (from the same chord) than you did last time. For example, in a V chord, where you played the II note in the first iteration, now play the V note instead. This trick is common to let the listener know that this particular section of the song is reaching or has reached its climax.
Alternate the mood as you go. If your melody has a minor feel, try following it up with something that has a major feel. Pieces that don't do that run the risk of sounding monotonous. In my experience, I've found that the human mind notices and appreciates transitions more than stationary situations. The happy part will sound happier if it follows a sad part and the sad part that comes after a happy one will sound even sadder and more dramatic.
Always keep in mind that, as it happens with all works of art, the point is not only the destination, but also the journey. A melody can be a journey to the other side of the scale and back. That doesn't mean that there aren't pieces where the melody stays around just a couple of notes. But usually, these kinds of pieces invest in something other than their property of being melodic - some kind of harmonic or even purely rhythmic effect, perhaps.
Of course, you can also just choose some chords (even at random, if you're willing to experiment), and write voices using the rules you know from studying music theory (if you have any background in that). There are some great melodies to be discovered there, while if you stick with more predictable chord progressions, your songs will tend to sound the same as other songs.
Try sticking to an unsual time signature (it may take a few minutes of practice beforehand) and explore what you can produce off that.
You can even just read a poem or some lyrics (which you've never heard music to) and try to notice how you instinctively read it in your head (or out loud), then write down whatever melody has popped up in your head.  It's fun and some recognizable melodies often come out of previously written verses.
Or you can literally watch birds sitting on power wires.
Comment: Thanks so much for this!! I'll re-read all the answers again, but this is the melodic style I'm after, can anyone bring up these notes so I can analyse the intervals etc. afterwards please:

This sounds like it is single notes being played out one after the other
https://soundcloud.com/mjohnson52/piano-1

This is a chord progression with a bit of passing tones I think in between - this is especially the one I'm after so I can learn to play that sort of style, sounds really good to me..

https://soundcloud.com/mjohnson52/chord-progression-piano
Answer: You say you know scales. Most melodies, especially the more simple kind, are made up from the notes of a particular scale, be it major or minor. To start with, try using the pentatonics. They are the major/minor scales with the awkward notes missing. As in C major, C,D,E,G,A. Not F or B. Using just those notes, many good melodies can be made, as leaving out the 4 and 7 means nothing will clash - ever !! And don't think the tunes will turn out 'Micky Mouse'. Listen to the solo in Sir Duke (Stevie Wonder) - apart from two little passing notes, it's all pentatonic.
Once you are happy with this idea, start putting the other two in. I think you're bogged down with theory. Theory for me comes way after where you are. If a set of notes sounds good in sequence, then it'll be justifiable in the theory. Using theory to create a tune won't do anything.
Above all, stay in one key throughout a song, and use only notes that come from that key. They will sound good together, and at least give you a basis on which to start.
Answer: First things first: I believe you are confusing a melody with a full blown music piece. In simple terms a melody, or melody line, is a sequence of monophonic notes (one note played at a time) and a music piece is a melody line played with an harmonization to support it.
For example, imagine you write the following melody line:

The first thing to do is to determine which tonality it is in and properly change the key signature. In this case we have a D minor denoted by the last long note being a D and after looking at all other notes used they matched to the D minor scale. Especially the natural F (which gives a minor third) and the Bb gives it away:

Here is how it sounds.
Then comes the part which for me is the most difficult part of building a music piece. I am still trying to learn how to do it properly: you add chords to harmonize the melody. This was the first piece I ever composed and harmonised on my own. I tried to use a II-V-I cadence very common in jazz. If there is one thing I have learned about harmony was that it is far from being an exact science (just like music or art in general) and it takes lots of time to learn:

Here is the sound of if.
Finally, by adding rhythm to the harmony and by adding a base line and drums I got this. (It is the full piece and not the first 16 measures as before) This last step can be easily done by using software like Band in a box or iReal Book on the iPhone.
I hope it helps.
Comment: In jazz, V-II-I is common? Maybe you meant II-V-I (or ii-V-I)?
Answer: Other posters have mentioned the key resources for you to check out. I won't rehash them. I will, however, give a more unconventional answer to balance them. By all means, soak up as much as you can re: theory, composition, etc. Let your curiosity really take you into the weeds. Examine the melodies of your favorite tunes with peculiar scrutiny, as if you were decoding them in search of some grand theory of melody.
Then sit down at your DAW and let all of that go. If you have zero inspiration in that moment, take the first hard step, and put down a chord progression- even one that doesn't particularly move you. Your mind will rebel because it sounds so plain, but as you said, everything comes from those 12 simple tones.
Then, let the magic of improvisation take over. How do you do this? Your mind will rebel against the very notion that you're capable of doing anything. "I'm not a musician. I don't know what I'm doing." Nonsense. Anyone and everyone can be musical. A baby can hum along to a tune, playing its first "instrument," if you will, before it knows anything of theory. Just listen to the progression playing in your DAW. The compulsion to immediately lay down an entire melody will make that difficult at first. Ignore that compulsion. Just listen to the progression. Then listen to yourself. You likely won't hear an entire melody. You will probably hear a one or two note embellishment to the chord progression. Fine. It's an idea. Loop it in your DAW. You now have the chord progression with a couple notes embellishing it. Listen to it. Listen to where your ear wants to take this little loop. Does it want you to extend this rudimentary idea you already have down? Does it want you to respond to it with another idea, another embellishment? You'd be surprised what you can build that way. Before you know it, you might have a more complex musical idea that calls you to forget that initial chord progression and give it another one altogether.
Think of yourself as a jazz musician soloing and let yourself go--whatever helps. The notion of jazz improvisation DEFINITELY helps. And if that means you can compose a great jazz solo a month from now, but want to create more focused, conventional melodies, then begin to limit yourself to the pentatonic scale and see how it goes.
If you have some time, check out Victor Wooten's book THE MUSIC LESSON. It's a little new-agey, sure, but it hammers home the notion that there is more to melody than theory, and more to music than melody (or simply "notes" as he might call them).
Don't get me wrong. I work in a quantitative, analytical field, and love getting really into the science of melody in its most nuanced form. But for every such foray into theory, I think you need another foray into dropping it all, letting it go, and letting yourself create.
Answer: If you've already worked out part of a piece of music- say the chord sequence, then a melody might "suggest itself" as you listen to the chords being played.
Also the set of notes in the scales used by the chords will of course narrow down the notes that are sensible. Sometimes this works in your favour in that making a melody is hindered by there being too many options. Using the chords as a limiting factor might help a bit.
Another way of going about this is to write a melody that you like then fill it out with accompanying chords.
Regarding "what notes sound good together" - well there's the alchemy.
It's subjective in that some tunes might sound great to some people, not to others. That doens't help you make melodies but it does hopefully help you understand that the "Universally lovely tune" is something that's very hard / impossible to make.
When it comes to writing music and melodies, you might do well to consider what mood you want to get across, and follow on from that.
For example major keys are generally considered a "happy" sound, minor keys "sad" but that that's just a guideline notion, there are plenty of examples where the rule is broken.
I'm not sure there are really any hard and fast rules to follow. In fact I'd argue the most interesting tunes (to my ear) are ones which seem to break percieved rules.
However I do think the skill of making a pleasant melody can be learnt, perhaps by learning how to build tension and release (cadence) and different moods (major/minor).
My experience is that experimentation while playing is probably the best way forward.
Answer: Almost any book on counterpoint offers advice on creating melodies. I would recommend taking that advice even if you're not interested in writing contrapuntal music (fugues, canons, etc.) If it's a good melody for singing it's probably also a good melody for most musical instruments.
Answer: There are a couple of outstanding answers here already, I'm just going to address the "cannot be learnt" fallacy, because it just might be well-intentioned.
I understand what is meant by "cannot be learnt", and disagree, I think it would be better to present the argument as "cannot be taught".  Like, nobody taught you to walk, or ride your bike, they just gave you endless encouragement until you got there yourself.  So here I go...
You wish to create something completely original, easy, anybody could do it, nobody needs to teach you to do that, so... there are probably a few additional requirements:

You probably would like to create something that you would like to hear again.
You need to be able to remember how to play it.

I would like to hear that again!
Why?  So, I assume it wasn't a collection of random notes.  Why wasn't it random to you?  Even if all the notes belong to a key, scale, mode, even if they harmonize with a chord progression, they can still be random, just random within those constraints.
When you heard those notes for the first time, something unusual happened, they made sense, even at the abstract level, you somehow understood that thing you just did, as if they told a story, and you liked it.  Which is fantastic!  That's the desired result.  I wish I could tell you how to do that, at will, and prolifically pen meaningful melody.
You can part with good money on books and courses that claim to teach just that, be my guest, but I think such material would be best-sellers if they really worked.

Now, how did I play it?
Enter music theory.  Especially scales, keys, modes, chords, harmony, rhythm... based on centuries of observation of musical phenomenon.
This will help you make sense of the mechanics of your melody more easily, it's not impossible to do this without any explicit musical knowledge, many people do, but it can reduce the time to find things you would like to hear again, and help you to mentally structure the information about your music so that you can remember how to play it.  Excellent.

However, did music theory teach you how to create a melody?  It certainly didn't hurt, but in itself, I'm not sure it provides the 'creative spark', the imagination, the ideas.  Of course, it could have, since anything can, but in my experience, creativity 'just happens', there is no formula, no secret to success, no hard and fast rules, you just do it.
If you cannot naturally mentally conceptualize an original melody (for example, if you are reading candidate lyrics and cannot 'hear' the melody you think would fit it well), it's nothing to worry about, that's only one technique, there are countless others (e.g. free play).
Music theory provides a framework, but don't expect it to be a portal into creating original melody.  Music theory can give you constraints for your music, and in certain situations, being constrained can be a creative boost (e.g. try making a melody from 4 random pitches - you'd be surprised how many people can pull that off).
Conversely, removing constraints is another method to boost creativity, studying a musical concept that is unfamiliar to you can broaden your horizon and draw in new inspiration.  Any shift to a new perspective can contribute to creativity, even non-musical changes, an emotional event in your life, for example.
Many artists have confessed at some time that "the melody just came", as if it was a divine experience. There is also the aspect of when trying to force creativity, it becomes blocked. Apparently, even boredom can help creativity!
Answer: It seems that you have at least a basic understanding of music theory. Understanding the concept of chords and scales is fundamental for music composition. I too find it so interesting and beautiful that with the 12 notes that are found in western music, there are so many possibilities.
There are a lot of complexities in music composition and the deeper that you get into music, the more that it will continue to surprise and delight you, as well as occasionally frustrate you.
What I would suggest to you is look up some common chord progressions, namely a I, IV, V, a ii, V, I, and a I, IV, vi, V. Stick to those chord progressions for your first few compositions and come up with melodies to them. A lot of melodies tend to follow a sort of question, and answer format. I think it's not only great practice, but also perfectly acceptable to keep your melodies simple and utilize this question and answer format.
Those chord progressions, are used frequently in all genres. The I, IV, V works so well because it contains all of the notes of a scale! Therefore if you stick to only one scale as you proposed above, it won't sound wrong. Now whether it sounds good, is up to how clever you can be with the notes of the scale. Ultimately you will have to be the judge of whether or not what you're writing sounds good.
Although theory exists, it is not a formula for successful music. Art is extremely subjective, if theory provided a formula for how to write good music, then technically every single song in the world that was written by someone knowledgeable in music would sound good, and as I'm sure you realize, this is simply not the case.
Answer: You might consider looking at Jack Perricone's Melody in Songwriting.  It's designed for, obviously, songwriters, but the basic principles of melody it sets forth are applicable to any kind of music.
Comment: Thanks I'll check it out, beside from that I've decided to really learn my scales, chords and get the most from music theory and apply it with practising other songs then making my own compositions
Answer: An old-fashioned but still relevant book on melodic creation is Percy Goetschius's "Exercises in Melody Writing" which is available as a free download here: https://ia802605.us.archive.org/16/items/exercisesinmelo00goetgoog/exercisesinmelo00goetgoog.pdf
It's a basic but very detailed description of parts of melodic structure. There are also lots of exercises.
Answer: Sing.
If you really want to create melodies, sing. And listen.  All the time.  Sing with bits of music you over hear or only half hear.  Sing with sound tracks. Sing with the radio. Sing your favorite tunes.  When you talk to yourself, sing the words. Make up songs like you did when you were a kid. When you hear yourself sing something cool, record it.
Sing riffs, short repeating melodic fragments 2-4 bars long.  Sing variations of each riff.
When you get back to your instrument of choice, play against the songs and riffs you've recorded.  Learn to accompany them. Sing with them. Sing contrapuntally.
Finally, start mashups of songs.  If you're unsure how to do this, listen to Umphrey McGee's "Zonkey" album.  Or if you're of more a classical bent, listen to Charles Ives.
Compelling melodies don't come from theory, they come from another place somewhere in our minds.  Theory helps support the tunes and sand off the rough bits

Learning to create melodies


Using tokens under exposed views filter

Question: I have created a view with an exposed filter. In my display I have an Using tokens section that I do not want to display:

Using the theme developer this is the closest I can get to it:

So it seems to part of views bulk operations. How can I get rid of this text?
Answer: I needed to overide a function in my template.php in my theme:
function blogbuzz_views_bulk_operations_fields_action_form($form) {
  $output = '';
  $access = user_access('Use PHP input for field settings (dangerous - grant with care)');
  if ($access && $form['#settings']['php_code']) {
    $output .= t('
Using the Code widget

Will override the value specified in the Field widget.

Should include <?php ?> delimiters.

If in doubt, refer to devel.module \'Dev load\' tab on a content page to figure out the expected format.

', array('@link_devel' => 'http://www.drupal.org/project/devel')
    );
  }
}

Using tokens under exposed views filter


How to delete existing NavHeader view while adding an updated one using NavigationView.addHeaderView()?

Question: I have a NavigationView in a Drawer Layout which contains a HeaderView followed by a Menu (as given in AndroidStudio's Navigation Drawer Activity's template). My Header contains an Image and couple of TextViews in a LinearLayout. I want to edit of one of the TextViews in the header from SharedPreferences. I use following function to do so
public void navUpdate()
{
        navigationView=(NavigationView)findViewById(R.id.nav_view);
        linearLayout=(LinearLayout) LayoutInflater.from(this).inflate(R.layout.nav_header_main, null);
        textView_nname=(TextView)linearLayout.findViewById(R.id.textView_nname);

        String name=sharedPreferences.getString("name","Your Name");
        textView_nname.setText(name);

        navigationView.removeHeaderView(linearLayout);
        navigationView.addHeaderView(linearLayout);
}

The problem is , though I've added the line
navigationView.removeHeaderView(linearLayout);

my previous headerview is still visible above the new one. And previous one is not updated. I read about the method navigationview.getHeaderAt() in many answers but believe me, that method is not available.
Here's how it looks
Is there any solution for this problem, if not is there any other way to do so?
Comment: thank u so much. Its done.
Comment: "navigationview.getHeaderAt()...that method is not available." - I believe you want the `getHeaderView(int index)` method. Also, if you just want to change the text in a header's TextView, you don't need to completely replace the header. Use the aforementioned method to get the header, and the `findViewById()` method to access the TextView you want to change.
Comment: @MikeM.-   I'm sorry , my bad, I meant navigationView.getHeaderView() only. But you might wanna take look at http://i66.tinypic.com/sm3j21.png
Comment: Make sure you're using the most recent library version. That method was just added about [three months ago](https://android.googlesource.com/platform/frameworks/support.git/+/cdccc4c6d5f1e40610ce4df4afeb1f99cbdb63e8%5E%21/#F2).
Answer: in order for you to get the Header from the NavigationView you'll have to call getHeaderView(int index) for instance:
View headerView = navigationView.getHeaderView(0);
textView_nname = (TextView)headerView.findViewById(R.id.textView_nname);
// set whatever you like on the textView.

Comment: This is the method I read about. but trust me, it is not available  http://i66.tinypic.com/sm3j21.png
Comment: Update your dependency to 23.1.1


Forecasting time series with several grouping attributes

Question: Let's imagine we have weekly sales data of a given set of products, and we're interested in forcasting the next week. Each product belongs to a product cluster (or "family"). I've tried both forecasting each product individually and forecasting an aggregate value for each family and then distributing this total using the typical "weights" of each product in the respective family, which work rather well depending on the family.
But what if there is another attribute that can be used to cluster past orders, such as the client? Is there a way this information can be introduced in the models?
Note: the same product can be sold to different clients.
Answer: There are two ways to do approach this:

You can aggregate the time series in a more specific way. When you aggregate you don't aggregate up to [product family], but to a combined [product family/client cluster] grouping.
You can keep your current hierarchical structure and pass the client cluster as an exogenous feature to your forecasting algorithm. Most forecasting algorithms accept exogenous features along with historical data.

In response to the comment:
The basic input to time series algorithms is the historical values of the time series. Many time series algorithms can also take in additional explanatory variables (called exogenous variables). You can add your client cluster variable as an input to your model.
See here for an example of how that works with ARIMA models.
Comment: Thank you very much Alex for your answer :) Would you mind elaborating on #2?
Comment: @PedroSchuller see edit


Como juntar 2 consultas de SQL em uma única (uma é um calculo medio de valor e outra criterio de busca com BETWEEN

Question: Estou montando um sistema que encontra os prestadores em uma região específica, calcula o valor médio cobrado por eles e traz esse valor.
Mas antes disso ele só vai pegar os prestadores que estão disponiveis na data requisitada. Tenho essas 2 consultas feitas e separadas, mas não estou conseguindo uma forma de juntar ambas em uma única.
Consulta da regra de disponibilidade:
SELECT *
FROM fcs_prestadores_pedidos
WHERE hora_entrada not BETWEEN '2017-08-10 16:00:00' AND '2017-08-10 18:00:00'
AND hora_saida not between '2017-08-10 16:00:00' AND '2017-08-10 18:00:00'

Consulta com calculo do valor médio dentro do raio de 90km
SELECT avg(valor)
FROM (SELECT valor,(6371 * acos( cos( radians(-23.5427450) ) * cos( radians( lat ) ) * cos( radians( lng ) - radians(-46.6513010) ) + sin( radians(-23.5427450) ) * sin( radians( lat ))))
AS distancia
FROM fcs_prestadores) virtual
WHERE distancia < 90

Como posso fazer ele primeiro fazer a consulta de disponibilidade, para em seguida calcular o valo médio?
Answer: Utilize a cláusula NOT EXISTS na segunda query utilizando os valores da primeira:
SELECT AVG(virtual.valor) AS valor,
       virtual.id
  FROM (SELECT fp.valor,
               (6371 * ACOS(COS(RADIANS(-23.5427450)) * COS(RADIANS(fp.lat)) * COS(RADIANS(fp.lng) - RADIANS(-46.6513010)) + SIN(RADIANS(-23.5427450)) * SIN(RADIANS(fp.lat)))) AS distancia,
               fp.id
          FROM fcs_prestadores fp) virtual
 WHERE virtual.distancia < 90
   AND NOT EXISTS(SELECT 1
                    FROM fcs_prestadores_pedidos fpp
                   WHERE fpp.prestador_id = virtual.id
                     AND (fpp.hora_entrada BETWEEN '2017-08-10 16:00:00' AND '2017-08-10 18:00:00'
                      OR fpp.hora_saida BETWEEN '2017-08-10 16:00:00' AND '2017-08-10 18:00:00'))
 GROUP BY virtual.id

As alterações realizadas na query foram:

Adicionado o id no retorno da tabela fcs_prestadores para que seja comparado com a fcs_prestadores_pedidos;
GROUP BY por id do prestador para garantir que será mostrada uma linha para cada um;
NOT EXISTS verificará se existe pedido para o prestador determinado no intervalo desejado. O vínculo é feito pela coluna id.

GROUP BY Modifiers
The GROUP BY clause permits a WITH ROLLUP modifier that causes summary output to include extra rows that represent higher-level (that is, super-aggregate) summary operations. ROLLUP thus enables you to answer questions at multiple levels of analysis with a single query. For example, ROLLUP can be used to provide support for OLAP (Online Analytical Processing) operations.
Suppose that a sales table has year, country, product, and profit columns for recording sales profitability:
CREATE TABLE sales(
  year    INT,
  country VARCHAR(20),
  product VARCHAR(32),
  profit  INT
);

To summarize table contents per year, use a simple GROUP BY like this:
SELECT year, SUM(profit) AS profit
  FROM sales
 GROUP BY year;

+------+--------+
| year | profit |
+------+--------+
| 2000 |   4525 |
| 2001 |   3010 |
+------+--------+

Em tradução livre:

A cláusula GROUP BY permite um modificadorWITH ROLLUP que faz com que a saída de resumo inclua linhas extras que representam operações de resumo de nível superior (isto é, super agregadas). O ROLLUP permite-lhe responder a questões em múltiplos níveis de análise com uma única consulta. Por exemplo, ROLLUP pode ser usado para fornecer suporte para operaçõesOLAP (Online Analytical Processing).
Suponha que uma tabela de vendas tenha colunas de ano, país, produto e lucro para registrar a lucratividade das vendas:
CREATE TABLE sales(
  year    INT,
  country VARCHAR(20),
  product VARCHAR(32),
  profit  INT
);

Para resumir o conteúdo da tabela por ano, use um GROUP BY simples assim:
SELECT year, SUM(profit) AS profit
  FROM sales
 GROUP BY year;

+------+--------+
| year | profit |
+------+--------+
| 2000 |   4525 |
| 2001 |   3010 |
+------+--------+

Comment: Funcionou certinho. Só mais uma coisa, no avg(valor) sabe como posso adicionar tbm quem está dentro do criterio do raio, trazendo a coluna id? ex: avg(valor)110 - id 1 | avg(valor)110 id 2
Comment: @LeandroMarzullo alterei a `query` para que retorne por prestador
Comment: eu inverti o between para AND ('2017-08-10 16:00:00' BETWEEN fpp.hora_entrada AND fpp.hora_saida

ele está trazendo o valor de cada e o id deles, queria que trouxesse o valor medio em algum lugar.


Eliminate lines based on values in multiple columns

Question: I'm trying to remove rows from a big table but conditioned that one column has one value and another column has other values.
So far I've been trying this but I guess I'm not combining the awk properly..
awk '$11 !="1"'| awk '$20==2 || $20==3' infile.txt >out.txt

The code is probably too simple but should work anyways..or not?
Thanks
edit:
This is what the table looks like
    5306083 TGATCAATCTCATAAC[A/C]AAAAAAAAA  consensus_24    211 1   species 0   0   0   T   0   7   T   recommended 0.708   F   0   -100    T   recommended
    5193751 AGTAGCTTGCGCGGA[C/T]GGGGGGGGG   consensus_32    227 1   species 0   0   0   T   1   1   T   not_recommended 0.75    F   0   -100    T   not_recommended
    5193254 TAAAAAAAAAAAAAA[G/T]ATTCATCC    consensus_26    192 1   species 0   0   0   T   1   0   T   not_recommended 0.726   F   0   -100    T   neutral

So if I filter based in that $11=1 and $20  needs to be "neutral" or "not_recommended" I would get this
       5306083  TGATCAATCTCATAAC[A/C]AAAAAAAAA  consensus_24    211 1   species 0   0   0   T   0   7   T   recommended 0.708   F   0   -100    T   recommended

Comment: You definitely won't remove lines from in file, I think here you will leave in file untouched and write some lines to out-file.
Comment: Your first awk is missing a file input,  also you don't need to invoke awk twice.  See the answer below.
Comment: Get the book Effective Awk Programming, 4th Edition, by Arnold Robbins ASAP!
Answer: awk '$11!=1 && ($20==2 || $20==3)' infile.txt > out.txt

should do.
UPDATE: based on the input given, you should get two lines in the output for this condition
$ awk '$11==1 && ($20=="not_recommended" || $20=="neutral")' file

5193751 AGTAGCTTGCGCGGA[C/T]GGGGGGGGG   consensus_32    227 1   species 0   0   0   T   1   1   T   not_recommended 0.75    F   0   -100    T   not_recommended
5193254 TAAAAAAAAAAAAAA[G/T]ATTCATCC    consensus_26    192 1   species 0   0   0   T   1   0   T   not_recommended 0.726   F   0   -100    T   neutral

But I guess, what you mean is you want the negation of this which is different from your original post
$ awk '$11!=1 || ($20!="not_recommended" && $20!="neutral")' file

5306083 TGATCAATCTCATAAC[A/C]AAAAAAAAA  consensus_24    211 1   species 0   0   0   T   0   7   T   recommended 0.708   F   0   -100    T   recommended

Comment: In this case. you could simplify slightly with a regex; `$20==2 || $20==3` could be written `$20 ~ /^[23]$/`.
Comment: Not working for me.. The outfile is empty. The idea is to remove those lines and get the table without them as output..
Comment: please post a sample input and expected output so that the scripts can be tested.
Comment: Mhmm I see what you mean.. I didnt know I had to add the ! expression to all of them. Thanks a lot. It seems to be working now


Translate node with node fields

Question: I'm using the Node Field module. When I translate a node its respective node fields are not copied to the translation, forcing me to recreate them by hand. Is there a workaround to this?
Comment: Is there a reason you are asking this here instead of in the module's issue queue? https://www.drupal.org/project/issues/node_field?categories=All  Questions like "Does this module support translations?" are probably best addressed in the drupal.org issue queues.
Comment: I've created an issue now: https://www.drupal.org/project/node_field/issues/2926048
Answer: I ended up solving the problem by using hook_node_insert:
<code>/**
 * Implements hook_node_insert().
 */
function mymodule_node_insert($node) {

  // Only act if we are dealing with a content type supporting translations.

  if (translation_supported_type($node->type)) {

    if ($node->tnid != $node->nid) {

      $sql = "INSERT INTO node_field( nid, type ,title ,machine_name ,show_title ,hidden ,value ,weight ,settings)
          SELECT :new_node, type, title, machine_name, show_title, hidden, value, weight, settings
          FROM node_field
          WHERE nid=:old_node";

      db_query($sql, array(':new_node' => $node->nid, ':old_node' => $node->tnid));
    }
  }
}
</code>

Translate node with node fields


Is this statement about credible intervals from Wikipedia correct?

Question: I read this statement on the Wikipedia article about Credible Intervals and it seems dubious to me. Although it has a reference associated with it, the book is not available to read online.

It is possible to frame the choice of a credible interval within decision theory and, in that context, an optimal interval will always be a highest probability density set.

Wikipedia: Credible Intervals

If it's correct, can someone explain how? Maybe I'm just misunderstanding one of the terms.
Comment: I think if the terms of the statement are not clear, it's not really appropriate to stand on its own in an encyclopedia.

That said, my interpretation (based on "within decision theory") was that a decision function based on the interval must minimize the expected loss with respect to an arbitrary loss function... which is why I found it dubious.
Comment: Optimal in what sense? If optimal is not defined, it is automatically true. The highest probability density set will always be of shortest length - so it is optimal in that sense.
Comment: I see what you mean. If you can find a loss function where this doesn't hold, perhaps you can bring this issue up on the talk page of the article.
Comment: My hope was that someone more knowledgeable would feel "bold" enough to edit it directly :) Does anyone even watch the talk page of obscure wikipedia articles?
Answer: The "will always be" part is a little confusing, because that depends on your loss function. For instance, if the action space is the set of Borel sets of $\mathbb{R}$, the posterior density is $\pi(\theta\mid x)$, and $\lambda$ denotes Lebesgue measure, then, for the loss function
$$
  L(\theta,B) = \lambda(B) + c\,(1 - I_B(\theta))
$$
the Bayes decision is an HPD set of the form $B=\{\theta:\pi(\theta\mid x)\geq 1/c\}$. But for different loss functions the Bayes decision may not be an HPD set.
Comment: That was more or less my reasoning (but being unexperienced with applying loss functions, I came up with a rather artificial loss function myself). So if the statement does not stand with any loss function, and in particular with some commonly used loss functions, would you agree the statement is not actually true?
Comment: The statement is not true for every loss function.
Comment: Also, ideally, your loss function is as personal and subjective as your prior is.


Ordinary Least Squared derivation using Index Notation

Question: The following function in $\mathbf{x}$ is given:
$$f(\mathbf{x}) = \frac{1}{2}\left\lVert \mathbf{Ax - b} \right\rVert_2^2$$
I want to calculate $\frac{\mathrm{d}f(\mathbf{x})}{\mathrm{d}\mathbf{x}}$ using index notation, where the conversion from linear algebra notation should happen immediately (i.e. no expansion of the square using $\left\lVert \mathbf{x} \right\rVert_2^2 = \mathbf{x}^\intercal\mathbf{x}$). Therefore the above is rewritten as $$ f(\mathbf{x}) = \frac{1}{2} (A_{ij}x_j - b_i)^2. $$ Using the chain rule, I immediately arrive at $$ \frac{\mathrm{d}f(\mathbf{x})}{\mathrm{d}\mathrm{x}_l} = 2\left(\frac{1}{2}(A_{ij}x_j-b_i)A_{ij}\delta_{jl}\right) = A_{il}^2x_l-A_{il}b_i, $$ or, converted to LA notation $$\frac{\mathrm{d}f(\mathbf{x})}{\mathrm{d}\mathbf{x}} = \mathbf{A}\mathbf{A}\mathbf{x}-\mathbf{A}\mathbf{b} $$ which is of course the wrong solution. I am fairly confident with matrix calculus using matrix notation, but I cant wrap my head around where the needed transpose would come from using the index notation (one can see that the result would be correct if the latter $A_{ij}$ in the above were $A_{ji}$, i.e. the transpose).
Answer: Since $f=\frac{1}{2}(A_{ij}x_j-b_i)(A_{ik}x_k-b_i)$, $\partial_l f=A_{il}(A_{ik}x_k-b_i)$. Contracting out the repeated index $i$ by the definition of matrix multiplication requires $A_{il}$ to be written as $A^T_{li}$, so $\partial_l f = A^T_{li}(Ax-b)_i=(A^TAx-A^T
b)_l$.
Comment: Thanks. I also tried expanding the squared in index notation, i just didnt realize i simply had to switch indices to introduce the transpose, what a bummer. Still, any idea how to come to this conclusion without ever exapnding the square, i.e. using the chain rule instead of the product rule?
Comment: @MartinZach It depends on whether you think computing $\partial_l x_j x_k$ uses the product rule. You can avoid that by considering separate cases where $0,\,1,\,2$ of $j,\,k$ equal $l$.


Giving a digital piano an acoustic sound

Question: When I was in my youth I played the violin for about a decade, and therefore, I suspect, I have become heavily attached to acoustic instruments. Now I'm in my mid thirties, bored of the violin, and have instead started to learn piano. All I got access to is a digital stage piano albeit a good one (Korg SP-250: weighted keys, decent banks). I also got high quality headphones, AKG K702.
But, despite the good equipment I find the sound utmost uninspiring. When I sit down at the acoustic piano at my parents' house it's as if I come alive with the complexity in the tonal picture that this instrument generates. The whole thing vibrates and what is playing is not just one sample per pressed key, it is a very complex picture involving this whole analog mass.
I now wonder, can I improve the electrical situation I am in? Sometimes I think the volume is low despite being at max. Is a headphone amplifier the way to go? I currently use the builtin headphone jack.
I also find that the sound is rather thin. Apart from the lack of complexity, it lacks "warmth" and "punch." I don't know if an amplifier fixes this, or how much the sound from the piano can be improved. Is
an equaliser(and/or amplifier) an alternative?
Or should I get a sound-bank and just use my piano as a MIDI-controller? (Costs is an issue because I'm a student.)
Comment: There are people who can't tell the difference between a real piano and a potato. You talk about the thing vibrating and all, so I guess you're one of the people who appreciate real instruments. The only object that can feel like a physical acoustic piano is an actual acoustic piano. Sorry. Good virtual pianos are better than bad ones, but there's no trick to make a stage piano vibrate under your fingers and come alive in the room.
Comment: What you could do is, forget about comparisons with an acoustic piano, and take the digital instrument for what it is. Connect it to loudspeakers or an instrument amp and use it like an electric piano. With a guitar or bass amp, an emulated Rhodes or clavinet sound can be very inspiring and credible, it's just a different instrument and it is used for different things. Choose a style of music that you _can_ do in an impressive way with the instrument you have. :)
Comment: Am I allowed to object to someone merely in their thirties referring to their 'youth' as though it was a long time ago? ;-)
Comment: Have you ever tried an electric violin? What would you tell someone who wanted theirs to sound and feel like an acoustic? The best digital piano is "dead" in a way the worst acoustic one isn't. It doesn't bother some people, but if it bothers you (and it sounds like it does) then there's nothing you can do about it.
Comment: First proposal to address the volume problem would be to find a different headset. If you can determine the electrical impedance of yours, be sure to get one with a lower impedance.
Comment: You need good sounds, if you're not satisfied with the keyboard's factory banks. That's it. There are two ways to do that: (1) Get yourself a Midi keyboard and a laptop with a decent processor and use VST instruments in a DAW. There are some awesome piano VSTs (paid) out there. (2) If your digital piano doesn't allow importing samples then, if you can, get a synth that does (my Korg KROSS 2 does). Using this feature, you can import WAV samples of any instrument you wish (yes, you can event create samples of your living room piano and import them) to create a patch on your synth.
Answer: I actually have the same keyboard, and it seems off that you don't think it's loud at max volume. I play with an 1/4" headphone jack plugged into the front port and I can't turn it up more than halfway or it starts to hurt... You're not trying to plug in an 1/8" jack into there without an adapter or something, are you? Can you try with another pair of headphones with a 1/4" jack to see if it makes a difference?
So that makes me think that you're just missing a lot of the sound complexity from whatever is stopping you from hearing it at normal volume.
That said, my opinion is that I love acoustic pianos but I think the Korg voices are plenty rich especially with reverb turned on. And be sure to leverage the main benefit of a digital instrument which is all the different voices, remember there are 3 banks for each voice. I usually play it set on the Fender Rhodes soundalike voice (Electric Piano 1) or the jazz organs, but the Piano 1/2 banks are great too.
Comment: I believe the perceived volume will depend on the impedance of your headphones. OP's AKG have 62 Ohm impedance, which is reasonably low. Maybe you are using some kind of studio headphones with higher impedance that usually require preamps?
Answer: Acoustic and electronic instruments are two different breeds. Yes, they both have black and white keys, and some even have similar actions, but that's where the similarities end. Just like you can't get a Rhodes or organ sound from an acoustic piano, you can't get a really authentic acoustic piano sound from an electronic piano - unless you pay thousands.
If it were mine, I'd be getting a nice little stereo amp and speakers - 50watts is plenty, and plug the keyboard into that. My Roland keyboards feed a 50w+50w amp into a couple of DAS 8" speakers, and that's sufficient for most halls that'll hold 200 audience - way too much for a small rehearsal studio, but the sound is as close as it can be, considering.
Good quality cans should be doing the job for you, but with an amp/speakers set up, you could plug the cans into that, to not upset the neighbours.
Answer:
What happens when you listen to a recording of an acoustic piano via your headphones? Part of what you're missing may be the acoustics of the room.
A headphone amp helps only if the problem is a matter of volume.

The amount of volume a set of headphones produces is determined by:

the sensitivity of the headphones (the amount of noise it produces when it's fed with 1W of power).
the impedance of the headphones, and the ability of the headphone amp to deliver power into that impedance.

A good headphone amp has a bigger power reserve than the standard circuit used in many instruments.
What a headphone amp can't provide is a sound (complexity) that's not there in the samples it's playing back.
So see if you can try a headphone amp, or a different set of headphones, or connect the piano to a set of loudspeakers.
Answer: The Korg SP-250 is a very competent instrument, but it's definately in the low-mid-range price bracket.  Closer imitations of an acoustic piano are available, but at a price!
Cheap steps first.  Yes, a headphone amp (or a more effecient pair of headphones) will make it louder.   Have you got a computer with a low-latency audio interface and a DAW program?  Try layering some computer sounds, adding some reverb...   (You'll need a mixer to hear everything at once, but a dirt-cheap little Behringer or similar will do.)
What sort of music do you play?  If you're aiming for Chopin and Rachmaninoff you're not going to be content until you can afford (and house) a Steinway.   If you get off on 'River Flows in You' you can spend rather less!


Safe and elegant way to access nested entities via fields

Question: I've seen a variety of approaches to access entity values in the theming layer. The code below is just a simple example using magic methods. It assumes all values are set - if any part of that returns a NULL, then in my experience errors ensue.
$value = $node->field_paragraph
    ->entity->field_media
    ->entity->field_file
    ->entity->field_text->value;

There are methods like ->hasField and ->count() to check things, and i have a variety in my themes. I can get the job done but sometimes it feels ugly. I'm interested the coding patterns that devs use to safely and elegantly drill through entity references.
Comment: There is a contrib module called reference_map. It's not very user friendly, but it was written expressly for this issue.
Answer: Suggestion, create a domain object that has the logic.
Normally these entities represent something that fits your business-domain.
E.g in your instance, the node might be an event.
So you might model a domain object called EventWrapper.
<?php

namespace Drupal\my_domain;
use Drupal\node\NodeInterface;
use Drupal\media\MediaInterface;
use Drupal\file\FileInterface;

class EventWrapper {
  protected $node;
  public static function fromNode(NodeInterface $node): EventWrapper {
    $instance = new static();
    $instance->node = $node;
    return $instance;
  }
  public function getMedia() : ?MediaInterface {
    if ($this->node->hasField('field_media') && !$this->node->get('field_media')->isEmpty()) {
      return $this->node->field_media->entity;
    }
    return NULL;
  }
  public function getMediaImage() : ?FileInterface {
    if (($media = this->getMedia()) && $media->hasField('field_file') && !$media->get('field_file')->isEmpty()) {
      return $media->field_file->entity;
    }
    return NULL;
  }
  public function getImageCaption(): ?string {
    if (($file = this->getMediaImage()) && $file->hasField('field_text') && !$file->get('field_text')->isEmpty()) {
      return $file->field_text->value;
    }
    return NULL;
  }
}

Then in your code:
<?php

$image_caption = EventWrapper::fromNode($node)->getImageCaption();

Answer: Although normally you render paragraphs recursively, you can retrieve a fixed structure non-recursively by recreating it with foreach loops:
foreach ($node->field_paragraph->referencedEntities() as $paragraph) {
  foreach ($paragraph->field_media->referencedEntities() as $media) {
    ...
  }
}

This avoids accessing empty fields and is able to process multi-value fields.
Answer: To expand on the great answer by 4k4 which shows a pattern we use widely but doesn't cover the existence of the field, you could create a simple class patterned after larowlan's great answer:
class EntityReferenceItemListHelper {
   protected $fieldItemList;
   static function create(?EntityReferenceFieldItemListInterface $fieldItemList) {
     $instance = new static;
     $instance->fieldItemList = $fieldItemList;
   }
   public function referencedEntities(): array {
     return $this->fieldItemList ? $this->fieldItemList->referencedEntities() : [];
   }
}

and then foreach (EntityReferenceItemListHelper::create($paragraph->field_media)->referencedEntities() as $media) {


Where does the importance of math come from?

Question: It is a somewhat philosophical question. I personally believe that the importance of math is due to its usefulness and lots of applications. Mathematics is used in everywhere nowadays; as Ian Stewart said in his book Letters to a young mathematicians, if we put 'MATH IN IT' stickers on everything that used math, then it will be hard to find a thing without the sticker. In this point of view, math is important because of its useful results.
However, some of the most sophisticated results of math, such as Fermat's last theorem, is not that useful or applicable in other fields. Despite their seemingly unusefulness, I think these results are also important because they have extended the boundaries of human knowledge. Knowing is important because it gives us some answers about who we are and how the universe is like. In the ancient times, people thought that the earth is flat and is the center of the whole universe. But now we know that the earth is round, and we are probably not the major characters in the history of universe. In this sense, mathematics is important because it gives us truths.
Now let me give you my personal anecdote. I was occasionally talking about the proof assistant Coq to my analysis professor. Of course Coq cannot prove everything, but she asked me whether I want all the problems in math to be solved by computers, supposing the proofs of computers are just like those of human mathematicians that no one can distinguish the proofs by computers from those of humans. I said yes, since I thought whether it is done by computers or humans, knowing the truth is important. But my professor said she don't like it. She thinks proofs made by computers are not as beautiful as those of humans. She thinks the importance of mathematics is in the joy of solving problems, and the beauty of mathematics comes from the efforts put in when solving problems. In this point of view, unknown mathematical truths are something like undisturbed nature that we have to prevent computers from pervading mathematical world.
Now I am confused. Where does the importance of math come from? Is using computers to prove theorems damage the very nature of mathematics?
Comment: Perhaps you should look into foundations and logic, especially the rigorous definition of "proof". I believe there are fairly old  theorems (~100 years) that show limitations on what computers can even theoretically do.
Comment: @Daniel, 100 years isn't "old"...
Answer: A huge proponent of using computers for math proofs is Doron Zielberger. You can read his opinions here.
Computer generated proofs have been in much controversy though- and it really depends on which person you are talking to; everyone have their own opinions.
For me- I do math, not just to find "truth" or beauty, but because I can't imagine my life without math. However, that's only a personal feeling.
Answer: All of that is secondary. The main reason is: because it's fun.
As for the computer question, I don't really like the idea, honestly. Math is a fun game, and if you start using cheat codes, you are depriving me of my fun game.
Answer: Math is the most contradiction-free, relatively true, succinct source of information in existence.  Also beautiful if you like weird symbols, commuting diagrams, and symmetries.   Also, we could be in a mathematically defined abstract universe right now.  As in we don't exist except in an abstractly defined structure.  Once our math is up to par we can recursively create sub-universes that seem to exist to us and unquestioningly exist to the beings inside.  Computers are a tool, and also can be tools at times.  :)
Answer: Personally I view much in the same way you mentioned. I view it as a tool for finding truths about the universe. Math and physics can be used to figure out where we came from. How we got here. I doesn't answer the question why we are here but math can point us in the right direction towards this meaning by showing us how we got here. Mathematics is truth. Provable, solid truth. I must also agree with your professor however. Computer generated proofs may not be as beautiful as proofs generated by humans but what should it matter? Beauty is subjective and is ultimately not the purpose of mathematics, simply a consequence. The importance of math is, in my opinion at least, to probe the unknown, figure why things the way they are, to find hidden truths in the universe. But ultimately math is what you yourself make of it. There is no one answer to this question. Everyone is going to have their own philosophy when it comes to mathematics.


How to update cell value to "" when other cell value changed?

Question: I need if I change the value of cell K2, the value of cell E2 should be equal to "".
I need to do it for all rows, K3 E3, K4 E4, Kn En .. .
Private Sub Worksheet_Change(ByVal Target As Range)

If Target = Range("K2") Then
    Range("E2").Value = ""
End If

End Sub

Answer: Let try:
   Private Sub Worksheet_Change(ByVal Target As Range)

   If Not(intersect(target,[K:k]) is nothing) Then Range("E" & target.row).Value = vbnullstring

   End Sub

Or better:
Private Sub Worksheet_Change(ByVal Target As Range)

If target.column=[k:k].column and target.cells.count=1 Then Range("E" & target.row).Value = vbnullstring

End Sub

Comment: You could change that to `IF Target.column=11 And Target.Cells.count=1 Then Target.offset(,-6)=""`. Think that correct syntax. Bit drunk :)


How to stop the book module defaulting to IE7 standards mode?

Question: I currently have a problem whereby when visiting a page created with the book module it forces IE into IE7 standards document mode.
I am running Drupal 7 with the Omega 3 base theme.  I have disabled the RDF module and added:
<code><meta http-equiv="X-UA-Compatible" content="IE=Edge">
</code>
Which forces all other areas of the site to render correctly, just not in the book module pages.
Has anyone else come across this problem?
Answer: I suspect you may have visited the book page in IE, and selected compatibility mode once, and it is now being remembered.  Another possibility is that you have an internet zone rule (probably an intranet rule) that is forcing compatibility mode on.  Your IT department may be forcing this via a policy, too.
I have had some problems with just using the meta in the header of a page, and have been using this:
<code>/**
 * Implements hook_html_head_alter().
 */
function MYTHEME_head_alter(&$head_elements) {
  // disable IE compatability mode
  // see http://msdn.microsoft.com/en-us/library/cc288325(v=vs.85).aspx

  $head_elements['x_ua_compatible'] = array(
    '#type' => 'html_tag',
    '#tag' => 'meta',
    '#attributes' => array(
      'name' => 'X-UA-Compatible',
      'content' => 'IE=Edge,chrome=1',
    ),
  );

  $head_elements['x_ua_compatible']['#attached']['drupal_add_http_header'][] =
    array('X-UA-Compatible', $head_elements['x_ua_compatible']['#attributes']['content']);
}
</code>
This adds the meta, so you can see it when you do a view source, and it also adds a proper HTTP header to disable it.
Just note that if this is being enforced via policy or IE settings, you may not be able to override it.
Comment: Hi, I tried this, however it doesn't seem to set the document mode on any part of the site, however if I manually put the HTML meta into the html.tpl.php template file it works - apart from on the book module.
Comment: @inertiahz Not sure why that doesn't work.  I have it live on a few sites, and see the HTTP header in the response.
Answer: I finally figured out that because I have access to the server the site runs on I could add the custom header X-UA-Compatible - IE=edge.  This has solved the issue server side.  I haven't found a solution if you don't have access to the server.
Comment: FWIW, you don't need server access to do this.  You can normally add this to the .htaccess for the site, too, unless your server admin has diabled Header in .htaccess.
Comment: Ah yes, good call - unfortunately I'm stuck using IIS6 so forgot about htaccess.


Java Applets and Proxies

Question: Assuming that I use the following code to connect to a SOCKS5 proxy, would connections or packets sent by an applet that I instantiate go through the same proxy?
System.getProperties().setProperty("socksProxySet", "true");
System.getProperties().setProperty("socksProxyHost", "*.*.*.*");
System.getProperties().setProperty("socksProxyPort", "*");

Applet is started using a classloader object from which a newInstance is created.
classLoader = new CustomClassLoader(/* Hashmap of byte arrays */); // Custom classloader that works using byte arrays
Applet applet = (Applet) classLoader.loadClass("class").newInstance();
applet.setStub(stub);
applet.init();
applet.start();
frame.add(applet);

Comment: There is [no such property as `socksProxySet`](https://docs.oracle.com/javase/7/docs/technotes/guides/net/proxies.html).
Comment: How do you intend to instantiate the Applet (what is it your class-loader hack does)? Where in the Applet(?) are you setting the above `socksProxy` settings? Finally, are the settings for a publicly routable IP address?
Comment: I set the settings in the main class, which then loads an applet. The hack simply allows me to use byte arrays instead of limiting me. It loads classes as normal.
Comment: Start over; a main class **or** an applet? They are not synonymous.
Comment: A main class **and** an applet. The applet is in the form of an object.
Comment: There isn't enough code here for me to test it; does it work?
Comment: That's my entire question.
Comment: That's odd, I thought there was one.
Answer: It appears that the answer to this question is yes.
The following code:
public class TestApplet extends Applet {

    private String ip;

    public void init() {
        try {
            URL ipCheck = new URL("http://checkip.amazonaws.com");
            BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(ipCheck.openStream()));
            ip = bufferedReader.readLine();
            bufferedReader.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public void stop() {
    }

    public void paint(Graphics g) {
        g.setColor(Color.BLACK);
        g.fillRect(0, 0, getWidth(), getHeight());
        g.setColor(Color.CYAN);
        g.drawString("Current IP: " + ip, 10, 20);
    }

}

and
public class Boot {

    public static void main(String[] args) {
        System.getProperties().setProperty("socksProxySet", "true");
        System.getProperties().setProperty("socksProxyHost", "220.127.116.11"); //Credits to HideMyAss.com
        System.getProperties().setProperty("socksProxyPort", "28045");//Credits to HideMyAss.com

        new Thread(new Runnable() {
            @Override
            public void run() {
                try {
                    Thread.sleep(5000);
                } catch (Exception e) {
                    e.printStackTrace();
                }
                TestApplet testApplet = new TestApplet();
                testApplet.init();
                JFrame jFrame = new JFrame();
                jFrame.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
                jFrame.setSize(500, 500);
                jFrame.setContentPane(testApplet);
                jFrame.setVisible(true);
            }
        }).start();
    }

}

Outputs:
Answer: Tyler, if I understand your question correctly - yes. But this comes with a caveat: the System properties that you set will persist only across the current JVM instantiation.
So, if like your example shows you set the properties and execute the Applet on the same JVM you should be fine.

Java Applets and Proxies


Send JSON data to sever - Javascript

Question: I have created a js file to get data from the form (text area) but I am not aware how to send this to server.
Could you please explain?
Here is my code
var myReview = element.myReviewTextarea.val();

      console.log("sendData()", "Sending data", { myReview: myReview });
      var promise = $.ajax({
        url: R.settings.endPoints.outgoing.sendData,
        type: "POST",
        data: {
          myReview: myReview
        }
      });

And I have some lines of code which receives data from json and I can see send data path is empty there. DO I need to create a json file and insert path there?
This is done by someone else and I am totally new to json.
endPoints: {
          incoming: {
            getData: "assets/js/tests/widgets/app-review-widget-test-data.json"
          },
          outgoing: {
            sendData: ""
          }
        }

Comment: you need to find out from the person thats giving you the json data. why is the outgoing url empty? obviously data will not send in that case
Comment: So it seems that you have an empty url? have you tried `sendData: "/some/url"`
Comment: empty URL would mean... the current directory?
Comment: That is my question, Do I need to create an json file for that if so what should be entered in that file to post the text area data to the server
Comment: json is just a data format. You are totally new to json when you've never seen it. Thus, you're not new to json ;)
Comment: You need to specify the url where you want to send you data. Try this `$.post('/path/to/file.php', {myReview: myReview}).done(function(){alert('...')}).fail(function(){alert('...')})`. Actually, here is the link: http://api.jquery.com/jQuery.post/
Answer: Use JSON.stringify() method to convert json object into string
var myReview = element.myReviewTextarea.val();

      var datatosend = { value1: "1",value2: "2" };
      var promise = $.ajax({
        url: R.settings.endPoints.outgoing.sendData,
        type: "POST",
        data: JSON.stringify(datatosend)
      });

Answer: JSON.stringify() will turn "datatosend" into a string so it's ready to be send to an external file
 var myReview = element.myReviewTextarea.val();
 console.log("sendData()", "Sending data", { myReview: myReview });

 datatosend = { endPoints: {
                  incoming: {
                     getData: "assets/js/tests/widgets/app-review-widget-test-data.json"
                  },
                  outgoing: {
                     sendData: ""
                 }
             }};

 $.ajax({
    url: R.settings.endPoints.outgoing.sendData,
    type: "POST",
    data: JSON.stringify(datatosend),
    success: alert('Done!')
  });

I'm not sure why you saved the AJAX call as a variable...
Comment: Anywhere within the ajax call. Check my updated answer in a sec

Send JSON data to sever - Javascript


Python script runs independently, but when called by flask app, throws permission error

Question: I have a Flask app running on Ubuntu that calls a separate python script. Within my Flask App, this is what calls the script:
args = ['python3', '/home/make_videos/code/master_script_slim.py', 'videos', '0,1', '01', '01']
output = subprocess.call(args)

When I run this, I get this error in the apache.log:
    Traceback (most recent call last):
  File "/home/make_videos/code/master_script_slim.py", line 90, in <module>
    make_dirs()
  File "/home/make_videos/code/master_script_slim.py", line 81, in make_dirs
    os.makedirs(sequence_creation_temp)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/make_videos/ASV_Tests/01'

But, when I run this script (master_script_slim.py) manually both in and out of my venv, I do not get this error.
Clearly it's a permissions issue while creating a series of directorys, but I can't exactly figure out how to fix it. I tried using sys.executable to work around the permissions, but the Flask app is python2, and I need to specifically call python3 for this script.
Here is the makedirs portion of the code that is bombing out:
sequence_creation_temp = working_root_path + '/' + artist_id + '/' + art_id + '/temp/'

thumbnails_final = working_root_path + '/' + artist_id + '/' + art_id + '/animated_thumbnails/'
thumbnails_image = working_root_path + '/' + artist_id + '/' + art_id + '/image_thumbnail/'
videos_final = working_root_path + '/' + artist_id + '/' + art_id + '/videos/'
scripts_final = working_root_path + '/' + artist_id + '/' + art_id + '/scripts/'

def make_dirs():
    if not os.path.exists(sequence_creation_temp):
        os.makedirs(sequence_creation_temp)
    if not os.path.exists(thumbnails_final):
        os.makedirs(thumbnails_final)
    if not os.path.exists(videos_final):
        os.makedirs(videos_final)
    if not os.path.exists(scripts_final):
        os.makedirs(scripts_final)
    if not os.path.exists(thumbnails_image):
        os.makedirs(thumbnails_image)
make_dirs()

Comment: apache mostly works as user `wwwdata` which has different  privilages and uses different environment - it can has different variables. it protects system against hackers.
Comment: that makes sense. so should maybe I should move these scripts/folders somewhere inside /var/www/FlaskApp?
Comment: probably it could be the best solution. Some people add `root` (or `sudo`) privilages to `wwwdata` but it can be the worst solution :)
Answer: furas provideded great info. I changed the script so python was creating the directories WITHIN the flask app, and everything went off without a hitch.


What's the first thing you check when an untouched unix server starts going berserk?

Question: So you have this neatly setup unix server and it's super fast and works swell and everything is great for months, and suddenly all kinds of weird errors start showing up for a variety of different services and none of them make a lot of sense on their own, much less together.
What are cheap things you should check as soon as you get your ssh session to the machine?
I'm specially interested in trauma stories that highlight non-obvious commands and rare situations, but I guess what's obvious varies from person to person, so we can just list them all freely.
Answer: First Order: Is it responsive?
If you can't log in, there's bigger problems afoot.  This generally comes in two flavors: hardware failure, and software failure.  Both are potentially catastrophic.  To prevent DFA errors, check the general hardware health first - a simple glance-over usually will suffice.
Second Order: Are the system's underlying structures in good health and order?
Check the "Golden Triad" of systems:

Enough CPU time is free for processing
Enough disk space is free for storage
Enough memory is free for workloads

In the last few decades, the triad has expanded into a "quad" which includes communications (networking):

Connectivity is functional, responsive, and has capacity

Third Order: What is the severity of the issue?
What programs or services are affected?  In decreasing order of severity, is it systemic (system-wide), clustered (a group of programs), or isolated (a specific program)?  Clusters of programs typically are tripping up because a specific underlying service has failed or gone unresponsive.  Systemic issues are sometimes related to this (think DNS or IP conflicts) but knowing where to look is usually the key.
Fourth Order: Are diagnostic tools providing useful data relevant to the issue?
Now that you have info about the health of the system (second order) and what parts of it are experiencing issues (third order) this should make it easy to narrow down where the problem is.
Error messages or log files should be a common waypoint on this journey.
CPU issues:

loadav
top
strace

Disk space / I-O issues:

df
du
lsof
iostat
vmstat

Memory issues:

free

Connectivity issues:

ping
route (and arp and rarp and friends)
iptables, ipchains, ipfw (for those BSD folks out there)
traceroute or mtr
hosts, nslookup, or dig
netstat

Most common complaint (that I hear):
Email is not delivering fast enough (more than a minute from send to receipt by recipient) or, email is rejecting my attempt to send.  This usually comes down to the rate limiter in Postfix kicking in during a spam-storm, which impacts the ability to accept internal delivery.
A real-life example:
However, this is not always the case.  One time, the issue persisted regardless of service restarts; so after 3 minutes it was time to start looking around.  CPU was busy but under 100%, yet the load had soared to 15 on a box of just 2 cores, and was threatening to go higher.  The top command revealed that the mail system was in overdrive, along with the mail scanner, but there were no amavis child processes to be seen.  That was the clue - the mail queue command (mailq) showed some 150+ undelivered messages, over 80% of which were spam, in the last 20 minutes.  A quick adjustment to lower the rate limiter (which reduced the intake rate of the spam storm) while increasing the number of child email scanner processes (to help process the backlog), followed by a service restart, resolved the issue and the system was able to complete deliveries in a short time.
The cause of the problem was that the amavis parent process had keeled over dead, and the child processes had eventually all run their course (they self-terminate after so many scans to prevent memory leaks).  So there were SMTP processes in postfix attempting to contact...thin air...to do the spam/virus scanning that was needed.  The distro I was using had out-of-date packages that would never be updated; as the installation was due to be replaced in a year or so, I manually "overrode" the install to the latest version, which included several bug fixes.  I haven't had the same problem since.
Answer: usually "who" followed by "last"
a heap of issues on machines I've managed over times have been because of a very loose definition of "untouched" - often someone has done something :)
Answer: Well, I'll start.
This one bit me once, I spent hours trying thousands of different things, disabling services here and there, rebooting, etc. What was the problem? Totally out of disk space.
So, here's the first thing I type when debugging a suddenly troubled server:
df -h

I never forget that now. It just saved me lots of wasted effort. Thought I'd share.
Answer: top (or htop)
Comment: or prstat on solaris.
Answer: If you can I would always try shutting down all NICs bar the management one.
Answer: First thing I check is 'top' (are there any strange processes; ones that hog memory or CPU time.)
If nothing turns up there, I'll check 'who' to see if anyone else is on my machine for some reason.
Maybe a filesystem got dismounted; check with a call to 'cat /etc/mtab' and then 'fstab' to make sure everything will come up right on boot.
Check uptime to make sure the # of users on the box is reasonable (should only be you) and then skim through var/log/auth.log to see if anything is awry there.
These are catch-alls. Depending on the errors your box is throwing, you may need to examine specific processes that are causing the trouble.
Answer: Running something like (at)sar on the host is almost mandatory.   The usefulness of being able to get historical snapshots of CPU, network, memory and disk I/O (amongst others) cannot be understated.
There have been many times that I have been able to diagnose a fault by examining what the host was doing in the past 24 hours, and seeing when things started going awry.
Answer: Checking dmesg for any errors - I usually start with a dmesg | tail, because chances are things are still going wrong and the server is still trying to do whatever is causing the error.
Answer: top
df -h
and ALWAYS check /var/log to make sure that partition hasn't filled up. That has caused total melt down on me a few times.
Answer: df -ha
to check if harddrives are full and someone hasn't received warnings
htop or top
to check memory and cpu usage isn't abnormally high.
Alternatively if the box isn't responding I go into the vm-ware client and check cpu/ram from there.
Answer: On linux, I usually check dmesg and /var/log/messages or /var/log/syslog. dmesg will indicate if it's a sudden hardware fault; quite a lot of other problems will show up in the system logs.
Answer: I suppose the first thing I do is a disk space check (as others have mentioned).  If the simple checks don't reveal a "common" problem then I'll investigate further.
One thing I like to do is capture a snapshot of the system.  I can grep these later to look for anything that has caught my eye.
lsof > /tmp/lsof.tmp &
ps auxfw > /tmp/ps.tmp &
netstat -anp > /tmp/netstat.tmp &

From there it's troubleshooting 101 but I find it a bit faster to grep the saved logs and if the condition clears while I'm logged in I have something to go on or look for changes.