How to predict the likelihood of multiple future occurrences based on probability

Question: I'm analyzing the game Yahtzee in which a player can roll five dice three times and at each interval can choose to roll/not roll certain dice or stop rolling altogether. The goal is to get different combinations like four-of-a-kind, a straight (1-2-3-4-5 or 2-3-4-5-6), etc.
I found that binomial probability was exactly what I needed. Using the formula P(X=r) = nCr * pʳ * (1-p)ⁿ⁻ʳ I could basically say "if I roll five dice, what are the chances 3 of them match" and that formula gives me a percentage chance. That works great!
However, referencing the research done here, your chances of getting a Yahtzee (five of a kind) is about 4% if you use all three rolls. Your chances of rolling a Yahtzee in your first roll is 0.08% but the probability of doing it over three rolls is much higher because you could set aside some matches and only roll the remaining dice you need to match.
But how can you calculate this at each interval?
Example: Say I want a four-of-a-kind and I roll a 4 5 3 4 2. A smart player would re-roll the 5 3 2 in hopes of getting more 4's. Binomial probability tells us you have a 6.94% chance of getting two 4's in the next roll. So how can I calculate the chance I get two 4's in either the next roll or the one after that?

Here's a representation of how you can accomplish this feat. If you need 2 4's, are rolling three dice, and can roll twice, you have the following win possibilities:

(1) 4 X X - (2) 4 X
(1) X X X - (2) 4 4 X
(1) 4 4 X

I can use binomial probability to calculate the chances of getting each of these. Should I average them to get what I need? I feel like I'm so close, hopefully one of you guys knows a formula or can help me out.
Answer: Some formality and structure should help you tackle the problem you mention in your example.
Take $E$ as the event of getting at least two more fours in the next roll or the one after that, $X$ as the number of fours in the first roll, and $Y$ as the number of fours in the second roll.
Using total law of probability, $$\begin{eqnarray*}\mathbb{P}(E)=\mathbb{P}(E|X=0)\mathbb{P}(X=0)+\mathbb{P}(E|X=1)\mathbb{P}(X=1)+\mathbb{P}(E|X\geq 2)\mathbb{P}(X\geq 2)\end{eqnarray*}$$ Since $\mathbb{P}(E|X\geq 2)=1$ and $X\sim \text{Binomial}(3,1/6)$ the aforementioned reduces to $$\mathbb{P}(E)=\frac{125}{216}\mathbb{P}(E|X=0)+\frac{25}{72}\mathbb{P}(E|X=1)+\frac{2}{27}$$ Now for $x=0,1$ $$\mathbb{P}(E|X=x)=\sum_{y=0}^{3-x}\mathbb{P}(E|X=x,Y=y)\mathbb{P}(Y=y|X=x)$$ Because $\mathbb{P}(E|X=x,Y=y)=1_{\{x+y\geq 2\}}$ and $Y|X\sim \text{Binomial}(3-X,1/6)$ we have $$\mathbb{P}(E|X=0)=\mathbb{P}(Y\geq 2|X=0)=\frac{2}{27}$$ Furthermore, $$\mathbb{P}(E|X=1)=\mathbb{P}(Y\geq 1|X=1)=\frac{11}{36}$$ Putting all this together, $$\mathbb{P}(E)=\frac{5203}{23328}\approx 0.22$$
Comment: I've figured it out, but how does the formula change if I was calculating the chances with three turns left? Wouldn't I have a `X`, `Y`, and `Z`, with `Z` being the number of fours in my third round? How can I implement that?
Comment: Are you familiar with Markov Chains?
Comment: A little above my math knowledge haha. They're referenced in the Yahtzee probability article (which achieves a similar goal of mine but doesn't show all the equations) so I'll do more research on that topic. Thanks!
Comment: Let $X_0=2$ and $X_j$ be the number of fours that appear after $j$ rolls. Then $\{X_j\}_{j\geq 0}$ is a Markov chain with states $\{2,3,4,5\}$ and has state transition matrix $$P=\begin{pmatrix}\frac{125}{216}&\frac{25}{72}&\frac{5}{72}&\frac{1}{216}\\ 0&\frac{25}{36}&\frac{5}{18}&\frac{1}{36}\\ 0&0&\frac{5}{6}&\frac{1}{6}\\ 0&0&0&1\end{pmatrix}$$ The sum of the third and fourth entries in the first row of $P^j$ indicate the probability of having at least four $4$'s after $j$ rolls.
Comment: So, since $$P^3=\begin{pmatrix}\frac{1953125}{10077696}&\frac{1421875}{3359232}&\frac{1035125}{3359232}&\frac{753571}{10077696}\\ 0&\frac{15625}{46656}&\frac{11375}{23328}&\frac{8281}{46656}\\ 0&0&\frac{125}{216}&\frac{91}{216}\\ 0&0&0&1\end{pmatrix}$$ The probability of getting at least four $4$'s after $3$ rolls is $$\frac{1035125}{3359232}+ \frac{753571}{10077696}\approx 0.38$$
Comment: FYI learning about Markov chain's is an exciting and surprisingly painless experience. I highly recommend it!
Comment: How do you know the probability of having at least four 4's is the sum of the third and fourth entries? The fourth entry assumes you get all four matching in one roll. If I understand it correctly, the third entry assumes you get three matching on your first roll. So why is it the sum?
Comment: The fourth entry in the first row of $P^j$ indicates the probability you observe five $4$'s after the $j^\text{th}$ roll. You don't have to obtain these five $4$'s in one shot for this to take place.
Comment: If you compute $P^2$ and sum the third and fourth entries in the first row you'll get the same exact value I obtain in my answer


How to handle private file uploads and downloads?

Question: For a project I'm working on, I need to be able to:

Upload a file as Admin, assigning it to a user (this is working using the "author" field at the moment), attaching it to a node (ie. using the "FileField" field type)
Store the file in a private directory (not accessible by public users)
Allow the Admin and "author" users to download the files

We've setup a test project using Drupal 7 with a FileField on a custom content type. The FileField is set to use "Private" uploads.
What seems to happen is that the files are not accessible by their path on the disk (e.g. http://sitename.com/sites/default/files/filename.pdf) but are accessible by the Drupal private file download path (e.g. http://sitename.com/system/files/filename.pdf) - even when users aren't logged in, the private file download path still allows the file to be downloaded.
What we need to do is:
Stop users who aren't logged in, and aren't the author or admin of the node, from downloading the file from the private path.
We've already got a hook_node_view() in a custom module which stops people looking at the node if they don't have permission (i.e. the $user->uid matches the $node->uid) - it's just the private file path we need to hook and validate (check the user is accurate).
Any ideas what we're doing wrong?
Comment: Are you using the field permission module?
Answer: If the whole node should only be visible to the author, then you should either implement hook_node_access() yourself and correctly deny view access or use a content access module (e.g. Content access) to do that for you. Note that hook_node_access() won't prevent that now to show up in listings, for that you need a content access module (or implement one on your own).
Alternatively, if you just want to hide the field, you could use the Field permission module and configure it accordingly.
Both ways should be able to to this because it is something that Drupal can check, it can't know that you prevent access in hook_node_view().
Comment: Thanks. I'll have a look at the hook_node_access function.
Answer: Either you are missing an .htaccess file in the system/files directory, or .htaccess has been disabled there. Without it, that directory is no different from any other publicly available part of the site.
If it is missing, try creating one containing just the line Deny from all.
If such a file does exist, your server configuration is ignoring it.
Personally, I never store private files below the public webroot. If my Drupal site is at /usr/local/www/mysite, I'll put the private files in /usr/local/www/mysite-files or some such.
See also http://drupal.org/documentation/modules/file .
Comment: I'd prefer to store the files outside the web root, but haven't been able to work out how to do that with Drupal. Can you share any insights into the upload and download process, and what modules you use?
Comment: Just re-read the Drupal site from your link, and think I'm a bit clearer. I had tried to set the path to ../private, but to no avail. Will try again in the morning.


Big O proof of Fourier Coefficient

Question: Let $f(x)$ be a $2\pi$ periodic function on R. Assume that Hölder continuous:
$$\sup_{x\neq y} \frac{|f(x)-f(y)|}{|x-y|^{-\alpha}} \leq C$$
for some constants $C$ and $\alpha \in \,]0,1]$.  Prove that
$$\hat{f}(n)= O(|n|^{-\alpha}).$$
I have been trying to work through using the corollary that states that if $f \in C^2$ on the circle then
$$\left|\hat{f} (n)\right| = O\left(\frac{1}{n^2}\right),$$
but I'm not reaching anywhere. Any help would be appreciated. Also if there is some general way of reaching the big-O status of Fourier coefficients please let me know. Would help me a great deal. I've searched through the postings but couldn't find a way to go about this problem. Thank you!
Ok so, the latest information I've been able to get is to reach a point in order to prove that $$\hat{f} (n) = -\frac{1}{2 \pi} \int\limits_{-\pi}^\pi f\left(x+\frac{\pi}{n}\right)\exp(-inx)dx ,$$
I am getting stuck after making the u-sub of $$\left(x+\frac{\pi}{n}\right) = u$$ and changing the limits of the integral for $\hat{f}(n)$to look like
$$\hat{f} (n) = \frac{1}{2 \pi} \int\limits_{-\pi+\frac{\pi}{n}}^{\pi+\frac{\pi}{n}} f(u)\exp(-inu) \exp(i \pi)du ,$$
Can someone at least help me with understanding how this integral would lead to the required form and I will be able to continue from there. I look forward to hearing something. Thanks!
Answer: By definition, we have
$$\hat{f}(n) = \frac{1}{2\pi}\int_{-\pi}^\pi f(x) e^{-inx}\,dx.\tag{1}$$
By periodicity,
$$\hat{f}(n) = \frac{1}{2\pi}\int_{-\pi}^\pi f\left(x+\frac{\pi}{n}\right) e^{-in(x+\pi/n)}\,dx = -\frac{1}{2\pi}\int_{-\pi}^\pi f\left(x+\frac{\pi}{n}\right)e^{-inx}\,dx.\tag{2}$$
Take the arithmetic mean of $(1)$ and $(2)$:
$$\hat{f}(n) = \frac{1}{2\pi}\int_{-\pi}^\pi \frac{f(x) - f\left(x+\frac{\pi}{n}\right)}{2} e^{-inx}\,dx.\tag{3}$$
Taking the absolute value in $(3)$, the Hölder continuity yields the result.
Comment: I didn't get to thank you earlier, but thank you so much :) Always great to have blind spots in my head cleared out

Big O proof of Fourier Coefficient


Using template variations from the modules folder

Question: I have the usual theming solution for a custom entity:
function module_theme($existing, $type, $theme, $path) {
  return [
    'customentity' => [
      'render element' => 'elements',
      'template' => 'customentity',
    ],
  ];
}

function template_preprocess_customentity(array &$variables) {
  $variables['content'] = [];
  ...
}

function module_theme_suggestions_customentity_alter(array &$suggestions, array $variables) {
  $sanitized_view_mode = strtr($variables['elements']['#view_mode'], '.', '_');
  $suggestions[] = 'customentity__' . $sanitized_view_mode;
}

The only problem is that when I want to use a twig specific to the view mode, customentity--viewmode.html.twig, it only works if the twig file is inside the theme. If it's in the module templates folder, it isn't picked up (carefully checked all cases with twig debugging).
So, in this regard, it's functioning. Still, I'd need the twig in the module itself because in this case, it's not a mere theming question, the view mode is both logically and programmatically a radically different way to present the contents. It's for internal use, not a contrib module, so normal user expectations don't apply.
Answer: You have to add the path to your theme hook like so:
'my_entity__mailteaser' => [
        'render element' => 'elements',
        'base hook' => 'my_entity',
        'template' => 'my-entity--mailteaser',
        'path' => drupal_get_path('module', 'asdentbase') . '/templates',
      ],

Comment: Unfortunately, if I do this, it will no longer find my `theme.inc` file because it will be expected in the same folder. There are no separate path settings for both...
Comment: Anyway, I tried with putting the `theme.inc` there, too. No, it doesn't work. `path` changes where the system tries to find the attached `file` but doesn't change where it looks for the templates.
Comment: `base hook` is the correct solution. Drupal is then able to locate the template in the module's templates folder, even without pointing at it via `template` or `path`. See https://drupal.stackexchange.com/questions/298844/use-custom-content-entity-theme-suggestions-in-own-module
Comment: Not by itself but only if I add an extra theme item with the view mode glued to it, yes. you're right. Would you care to put it into a short answer for me to accept? :-)
Comment: @Gábor, see the linked topic. The code in question+answer is the complete solution.
Comment: Yes, I know, thanks, I utilized it. I just wanted to be able to accept it here.


need a method to find a value in HashMap

Question:
Possible Duplicate:
How to sort a Map<Key, Value> on the values in Java?

I have a HashMap as following:
 Key Value
  A    5
  B    3
  C    10
  D    4
  E    1
  F    11

I need to find the one with highest value, what do you suggest me to do? should I sort them and get the first one or there is any other faster way?
Comment: Depending on what you want to do, you may also keep track of the largest value in the HashMap yourself as new key value pairs are inserted into the HashMap
Answer: I would not advice to do sorting for a search requirement. As adviced by @David Lam, you can perform a search(iteration) as below to find the highest value key.
    Set<String> keys = myMap.keySet();
    Iterator<String> keyIter = keys.iterator();
    String highestValueKey = null;
    while(keyIter.hasNext()){
         String key = keyIter.next();
         if(highestValueKey == null){
             highestValueKey = key;
         }else if(myMap.get(key).intValue() > myMap.get(highestValueKey).intValue()){
             highestValueKey = key;
         }
    }

In the end, highestValueKey will have reference to highest value element's key.
Answer: This is much more easily solved by using a SortedMap and passing in a Comparator for the values:
final Map<String, Integer> map = new HashMap<String, Integer>();
map.put("A", 5);
map.put("B", 3);
map.put("C", 10);
map.put("D", 4);
map.put("E", 1);
map.put("F", 11);
map.put("G", 11);
map.put("H", 10);

TreeMap<String, Integer> sorted = new TreeMap<String, Integer>(new Comparator<String>() {
  // Note: this comparator imposes orderings that are inconsistent with equals.
  @Override
  public int compare(String a, String b) {
    if (map.get(a) >= map.get(b)) {
      return -1;
    } else {
      return 1;
    } // returning 0 would merge keys
  }
});
sorted.putAll(map);

Entry<String, Integer> first = sorted.firstEntry();
System.out.println("Highest value: " + first.getValue() + is for key: " + first.getKey());

// If duplicate keys are never a concern, you can stop here.  Otherwise, one may
// continue below to find all keys that may be mapped to an equal highest value:
List<String> others = new LinkedList<String>();
for (Entry<String, Integer> entries : sorted.entrySet()) {
  if (entries.getValue().equals(first.getValue())) {
    others.add(entries.getKey());
  } else {
    break;
  }
}

System.out.println("All keys mapped to this highest value: " + others);

Prints out:
Highest value: 11 is for key: G
All keys mapped to this highest value: [G, F]

Comment: I have a bit of doubt whether the TreeMap can still be used as a Map after this.
Comment: I haven't tested, but I just have a bit of trouble imagining how get() and containsKey() will work.
Comment: @nhahtdh The logic is sound. Also see http://stackoverflow.com/a/1283722/24396
Comment: This solution breaks the contract written in the Map interface, and is fragile. Read the comments in the link for more information. -1
Comment: @nhahtdh I updated to prevent the overwriting of keys with the same values.  This example should resolve what the poster is looking for "the one with the highest value".  Agree that better solutions exist, but for this scenario, without adopting any third party libraries (Guava), this should suffice.
Comment: @Cuga: The code is at best patchy (and most of the functions in TreeMap is broken). If you have already coded this far, writing a reverse map from Integer -> List might be better.
Comment: It's not really patchy.  The `Comparator` is an anonymous class, used only in one location and for one specific purpose, so the `equals` contract with `compareTo` doesn't really come into play in this closed-example.  It completely solves the OP's quest which is just to find the one key.  After gotten, she can continue to use the original `HashMap` as she desires.  Only use the `TreeMap` for this one task-- finding the key with the max value in its own function, and then she'll always be fine.
Comment: @nhahtdh To be clear, I'm not saying he should ditch the `HashMap` and use a `TreeMap` from here on out-- merely to only use it for the purpose of finding the sought-answer, ideally in its own, focused function designed to do exactly this which will return the result.
Comment: @Cuga: Your method is no better than looping through all the values and get the maximum value. The TreeMap is totally unnecessary and has a bad example of implementing Comparator for TreeMap.
Comment: That's not at all true.  Simply looping thru and keeping a record of the max value will still need special care for recording duplicate keys.  In any event, it's an entirely acceptable solution to completely address the problem.
Answer: just iterate through the keys and keep track of the highest one O(n)
Answer: If you need to do the "find the largest value" operation frequently, I suggest you use TreeMap.
Else if you are inserting values into the HashMap and have complete control over it, then keep track of the largest value as you insert values into the HashMap.
EDIT:
You will need to use HashMap either way as shown here
Comment: TreeMap doesn't work here, since it order based on key, while OP wants to find the value.
Answer: Or you can use Bubble Sort to find the highest value


VMWare ESXi 5 - How to clone Windows 2008 SBS

Question: I am running two Windows 2008 Server on a VMware ESXi 5 Server (VMware vSphere 5 Essentials Kit). There are two different ESXi hosts within my VMware datacenter environment. Let us call them esxserver1 and esxserver2.
Both of my Windows 2008 Servers are currently running on esxserver1 and I would like to move both Windows Servers to esxserver2 due to performance reasons (esxserver2 is much more powerful).
I actually expected that I can just clone both Windows VMs using vCenter but unfortunately the VM clones did not work after they were cloned successfully from esxserver1 to esxserver2.
Both Windows 2008 Server were stuck with some strange krnl error messages like the following ones:
ntkrnlpa.exe missing or corrupt
After I ran a repair from Windows Installation DVD a both machines came up with a quite similar error:
ntoskrnl.exe missing or corrupt
I really wonder what I did wrong. Do I have to consider something else when cloning Windows 2008 servers from one esxserver to another?
Comment: If you want to move them then move them. There are a number of ways to do this: migrate, copy, etc. and a number of tools available to do this: WinSCP, Veeam FastSCP, Trilead VMExplorer, etc.
Comment: Please clarify what hardware esx1 and esx2 have.If they are using different CPU family this can cause the issue.
Answer: You shouldn't need to clone them. Just right-click in VMWare Manager and choose "Migrate". From there you can move them. Note: The machines will have to be powered off to do this.


Numpy: get the first different value along axis

Question: I have a 2D array, and I would like to know, for each "row", what is the index of the first value that differs from the first value of each row.
Here is an example:
arr = np.array([[0, 0, 1, 1, 1],
                [2, 2, 2, 3, 3],
                [9, nan, nan, 8, 8],
                [5, 5, 5, 5, 0]])

The solution I'm looking for would yield:
array([2, 3, 1, 4])

since in the first row, the first value different from 0 is the 3rd,
in the second row, the first value different from 2 is the 4th, and so on.
Thanks for your help !
Answer: Just compare all the values in the array to the first column and extract the index of the first true value.
(arr != arr[:,0][:, None]).argmax(1)

Out[]: array([2, 3, 1, 4], dtype=int64)


Geometric understanding of differential forms.

Question: I would like to understand differential forms more intuitively. I have yet to find a book which explains how the use of the exterior product in differential forms ties into the geometrical significance of it. Most books briefly introduce the exterior algebra and then go on to prove theorems about differential forms without ever geometrically motivating anything, and this does not help me to truly understand them. Could someone explain or give a reference to a text explaining differential forms more geometrically. I would very much like to have a deep understanding of them.
Thanks very much
Comment: Here is a list of some references: http://math.ucr.edu/~res/math246A-2012/diff-forms-refs.pdf
Answer: A differential k-form on an n-manifold can be visualised as a "density" of (n - k) submanifolds. In $\mathbb{R}^3$ a 3-form is a point density. The form $dx \wedge dy \wedge dz$ is a uniform point density such that there is 1 full point in a unit cube. The integral $\int_S dx \wedge dy \wedge dz$ measures the number of points inside $S$, which is equal to the volume of $S$. The form $f dx \wedge dy \wedge dz$ is a point density with density $f(x,y,z)$ at that point.
The form $dx$ is a density of planes of constant $x$ (i.e. $yz$-planes) such that there is 1 full plane in a unit of $x$. In other words, the line segment $(t,0,0)$ for $t=a$ to $t=b$ crosses $b - a$ planes (note that the orientation matters). As with a point density, it's not that we cross a plane discretely at $x=0, x=1, x=2$, rather, the $yz$-planes are distributed uniformly along the $x$-axis. For a curve $\gamma : [0,1] \rightarrow \mathbb{R^3}$ we can count the number of $dx$ planes it crosses. This is denoted $\int_\gamma dx = x(\gamma(1)) - x(\gamma(0))$. A general form $\alpha = A dx + B dy + C dz$ can be visualised as a density of surfaces with general orientation. The integral $\int_\gamma \alpha$ counts the number of times the curve $\gamma$ pierces a surface in the surface density. In other words, it counts the intersections of $\gamma$ with the surface density.
For a function $f : \mathbb{R}^3 \rightarrow \mathbb{R}$ the form $df$ represents surfaces of constant $f$. Viewing $x$ as a function that gives the $x$ coordinate for a point, you can see that $dx$ corresponds to planes of constant $x$. If $r = \sqrt{x^2 + y^2 + z^2}$ then $dr$ are the surfaces of constant $r$, i.e. spheres centered at the origin.
A 2-form $dx \wedge dy$ represents lines in the $z$-direction. Lines in the $z$-direction are formed by the intersection of a plane of constant $x$ and a plane of constant $y$, i.e. lines in the $z$-direction are lines of constant $x$ and $y$. For functions $f$ and $g$, the form $df \wedge dg$ represents curves of constant $f$ and $g$. For two general 1-forms $\alpha$ and $\beta$, which represent densities of surfaces, the form $\alpha \wedge \beta$ represents the density of curves formed by the intersection of those surfaces. The form $dx \wedge dy \wedge dz$ is the point density of intersecting the lines $dx \wedge dy$ in the $z$-direction with the $xy$-planes $dz$.
Given a parameterised surface $A : \mathbb{R}^2 \rightarrow \mathbb{R}^3$, the integral $\int_A \alpha$ of a 2-form $\alpha$ is the number of times the lines of $\alpha$ intersect the surface $A$.
The operation $d$ forms the boundary of the density of curves/surfaces/volumes. For example, of $\alpha$ is a 2-form representing a collection of curves, $d\alpha$ represents the collection of endpoints of those curves. We can understand the formula $d(df) = 0$; the curves $df$ of constant $f$ have no endpoints. The boundary of a density of volumes is a density of surfaces, the boundary of a density of surfaces is a density of curves, the boundary of a density of curves is a density of points, the boundary of a density of points is zero. Let's understand the form $x dy \wedge dz$. Visualize this as small lines of constant $y,z$ i.e. lines in the x direction. The lines have density $x$ near a point $x,y,z$. This can only be accomplished if the line density has a collection of boundary points. As we move further along $x$, the density of lines in the $x$ direction gets higher, and those lines have to start somewhere. Indeed, we see that $d(x dy \wedge dz) = dx \wedge dy \wedge dz$. The collection of lines $x dy \wedge dz$ has a uniform density of start points. On the other hand, $y dy \wedge dz$ has no net start points. This is just a collection of lines in the $x$ direction that gets denser as we move in the $y$ direction, but those lines still go on from $x = -\infty$ to $x = \infty$. Indeed, we see that $d(y dy \wedge dz) = 0$.
In summary, on an $n$-manifold

An $k$ form represents a density of $(n - k)$ submanifolds.
The wedge product represents intersection of densities.
The $d$ calculates the boundary.
The integral $\int_M \alpha$ of a $k$-form along a $k$-submanifold computes the number of intersection points of the $k$-submanifold with the density of $(n - k)$ submanifolds represented by the form.

We can also intuitively understand the general Stokes theorem $\int_M d\alpha = \int_{\partial M} \alpha$. Let's consider a function $f$ on $\mathbb{R}^2$. The form $df$ represents the countours of constant $f$ (think of a contour plot), with a density such that there are net $b - a$ contours between the countour $f(x,y) = a$ and $f(x,y) = b$. Now consider a curve $\gamma$ with endpoints. The integral $\int_\gamma df$ calculates the number of contours crossed by $f$. The integral $\int_{\partial \gamma} f$ is just $f(\gamma(1)) - f(\gamma(0))$, i.e. $f$ evaluated along the boundary of $\gamma$ (with the appropriate orientation). We can see that these two integrals are equal: the number of countours crossed by $\gamma$ is precisely the height difference between the start and endpoint.
Now let $V$ be a volume in $\mathbb{R}^3$ with boundary $\partial V$, and let $\alpha$ be a 2-form representing a density of curves. The integral $\int_V d\alpha$ counts the number of endpoints sitting inside $V$. The integral $\int_{\partial V} \alpha$ counts the number of curves piercing through the boundary $\partial V$. You can intuitively understand that these are equal: the curve emanating from an endpoint must either have its other endpoint inside $V$, in which case this part of the curve density does not contribute as one endpoint is positive and the other negative, or the curve has its other endpoint outside $V$, in which case it must pierce its boundary.
The Stokes theorem for a surface in $\mathbb{R}^3$ is a bit tricky to describe with words, but drawing a picture will convince you.
You can also use this picture to understand the pullback, Poincaré lemma, the formula for $d(\alpha \wedge \beta)$, the degree formula, Poincaré duality, and so on.
One last point: why is there this weird inversion of dimension that a $k$-form represents a density of $(n - k)$ submanifolds? Why not use $r$-vectors to represent densities of $r$ submanifolds rather than $n - r$ covectors (which is what differential forms are). In particular, why not use a normal vector field to represent a density of curves? The reason is that $r$ vectors do not have the best transformation properties. A point density on the plane $\mathbb{R}^2$ is something that assigns a real number to an area. The infinitessimal version of this is that for a 2-form $\alpha$ the quantity $\alpha(x,y)(u,v)$ counts the number of points in a small parallelogram spanned by the vectors $u,v$ at the point $x,y$. If we deform the plane then the parallelogram of the vectors will deform with it, and $\alpha(x,y)(u,v)$ stays constant. If we used $r$ vectors rather than $r$ covectors, we would only be invariant under isometries rather than general diffeomorphisms. Suppose we have a vector field on the plane and a curve $\gamma$ in the plane. There is no basis independent way to say how many times the curve intersects with the vector field. This question only makes sense up to some scale factor. Forms have a natural scale attached to them, because a form $df$ naturally eats a tangent vector $y'(t)$ as $df(\gamma(t))(\gamma'(t))$. To do this with the vector field you'd have to choose a basis/coordinate system.
I hope that helps.
Comment: This seems a lot like "On the Visualisation of Differential Forms": http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.126.1099&rep=rep1&type=pdf . I'd never heard of this picture before today (your answer and that link), except that I independently discovered part of it yesterday.
Comment: I'm sure it was well known at some point. This visual interpretation is common when teaching vector calculus / electromagnetism, and differential forms are just the generalisation of that to n dimensions. Strangely though, Cartan, the inventor of differential forms, never mentions any visual interpretation in his book. I've heard from several people who teach differential forms that they were unaware of this interpretation. It's a bit of a mystery where the visual intuition got lost. Perhaps Cartan was aware of it, but considered it too obvious to mention, or perhaps it got lost before Cartan.
Comment: It's on my todolist :D
Comment: This is a very nice answer.  But I'd like to gently push back on the implication that this visual, geometric interpretation has been "lost."  In fact, these perspectives are crucial to topology (via [Poincare duality](https://en.wikipedia.org/wiki/Poincar%C3%A9_duality)), to geometric measure theory (via the study of [currents](https://en.wikipedia.org/wiki/Current_(mathematics))), and to algebraic geometry (via [intersection theory](https://en.wikipedia.org/wiki/Intersection_theory)), all of which are active areas of research.
Comment: There surely is some kind of geometric intuition there, but I find the intuition quite vague. For example, it's clear that the wedge product is somehow related to intersection, but precisely how isn't described, and as far as I know, pictures are not drawn or described. Contrast this with physics books about electromagnetism: the pictures are there, and the intuition is clear and quantitative. Each electron has definite amount of field lines emanating from it, and that's why the flux through a surface is equal to the amount of electrons enclosed.
Comment: You're right that De Rham was very much aware of this intuition, as were Maxwell, Poincaré, and Gauss. However, if I remember correctly, de Rham's book about currents doesn't contain a single picture. So "got lost" is indeed the wrong phrase; but the intution is not conveyed to the student. In the class I took on manifolds and differential forms, neither the professor nor any of my fellow students were aware of it. Differential forms were abstract symbols with some very vague intuition. Maybe that's the exception, who knows..but if I look at the average differential forms book I doubt that.
Comment: @Jules, thank you for this nice answer. Do you have a nice book reference that treats the subject?
Comment: I don't, unfortunately. For vector calculus there are the electromagnetism books. You could mentally translate that to differential forms while reading.
Comment: @jules, thank you. It seems crazy that no mathematician wrote a book about this
Comment: @Jules, thank you so much for these beautiful set of intuitions! I have never understood differential forms. now at least i can claim i understand them to an intuitive degree. i do have a question though: you say: "For a curve :[0,1]→ℝ we can count the number of  planes it crosses. This is denoted ∫=((1))−((0))" what i don't get is that $\gamma$ could potentially pierce the planes from $x(\gamma(0))$ to $x(\gamma(1))$ so many times. yet the final formula seems to not to depend on that. how does that work?
Comment: unless you subtract whenever you cross forward and back, which in that case i guess that's the answer? :D
Comment: also could you please clarify what "general orientation" mean?
Comment: I mean that at every point $p=(x,y,z)$ we have a small piece of surface (say, a small rectangle) with arbitrary orientation, determined by the $A(p),B(p),C(p)$.
Comment: Trying to extract a more precise statement from this lovely answer - is it really true that given a k-form $\varphi$, we can construct some stochastic process that will produce an n-k random-submanifold $X$, such that the expectation values of intersection numbers of $X$ with any oriented k-submanifold $S$ will be proportional to $\int_S \varphi$? (let's assume the ambient space is compact and oriented)
Comment: In dimension $> 3$, we can consider a differential form $\varphi$ whose rank at a point $p$ is $\geq 2$, so we cannot express it as a single wedge product of 1-forms (in any coordinate system), only as a sum of such products, and the decomposition into a sum is not unique. What is $T_p X$ in this case? (more precisely, what is the distribution on the Grassmannian $Gr_{n-k}(T_p M)$ of possible tangent spaces at $p$, induced by the proposed submanifold density corresponding to $\varphi$?) if choices are involved, how can we make them consistently throughout the ambient space $M$?
Comment: @Jules, could you explain what "the integral $\int_A \alpha$ of a 2-form $\alpha$ is the number of times the lines of $\alpha$ intersect the surface $A$." means? How to quantify the number of times the lines intersect the surface. I guess there are an infinity of lines crossing the surface no?
Comment: One way to see that a little bit more formally is by considering a special case. Consider $\det(AB)$ where $A : U \to V$, $B : V \to U$ are rectangular matrices. Consider the preimage $A^{-1}[C]$ where $C$ is the unit cube in $V$, and the image $B[L]$ where $L$ is a grid of points in $V$. The quantity $\det(AB)$ is proportional to the cardinality of the intersection of these two sets as you make the grid finer.
Comment: This is related to the integral $\int_S \alpha$ in the following special case: $S$ is the preimage of a unit cube (note $\dim U \neq \dim V$ in general), and $\alpha$ is a constant differential form of rank 1. The general case of $\int_S \alpha$ where $S$ is a curved surface and $\alpha$ has arbitrary rank can be pieced together by splitting the surface into small pieces that look almost straight, and by splitting up $\alpha$ as a sum of rank 1 forms.
Comment: This isn't the kind of answer that Just Me is asking for, and it would be nice to have a more precise answer, but I don't know how. Maybe one could be partially satisfied with the preceding local picture when everything is smooth enough, and when thinking about the general situation only think about intersecting a little piece of the surface with a locally constant differential form.
Comment: @Jules - setting aside the "local vs. global" gap - I don't understand how to interpret a differential form of any rank > 1 (the rank being the minimal number of summands needed to express it as a sum of products of 1-forms). The picture you suggest works fine for each summand separately, but you need to somehow superimpose these pictures for different summands. Adding to the confusion is the fact the decomposition into summands is not unique, and the rank can "jump down" (is lower semicontinuous).
Comment: Indeed, I don't at present see a way around that. The superposition does make some sense: when measuring how many of lines of flux go through a window, we also have that you can superimpose two sets of lines and measure their sum flux. With superimposed lines, you can always find an equivalent set of non-superimposed lines, and you just cannot always do that with higher dimensions. The probabilistic interpretation where you split further into individual randomly sampled pieces does work. The way I would view it is that there are many different (but equivalent) ways to visualize a given form.
Answer: There are grave issues with the common thinking inherent to differential forms.  For instance, in vector calculus, you can consider a vector field
$$\mathbf G(\mathbf r) = \frac{\mathbf r}{4\pi r^3}$$
You can meaningfully integrate the divergence of this vector field,
$$\int_{M} \nabla \cdot \mathbf G \, dV$$
and if your region contains the origin, you get 1, else 0.
In differential forms language, you have as your only hammer the exterior derivative.  Instead of considering vector fields like you're used to from vector calculus, you have to convert $\mathbf G$ to a 2-form $\omega$.
$$\omega = \frac{z \, dx \wedge dy + y \, dz \wedge dx + x \, dy \wedge dz}{4\pi r^3}$$
And then you do an integral like
$$\int_{M} d\omega$$
as is usually done.  But this is the big problem with learning differential forms when coming from vector calculus.  You have to take all the stuff you're familiar with and turn it upside down.  You have to take vector fields and turn them into two-forms to take meaningful divergences; you have to convert them to one-forms to take meaningful curls (though this is much less peculiar).

What's actually going on here can be explained in a different formalism called geometric calculus.  It enforces that when we integrate over geometric regions (surfaces, volumes, and so on), the integration measure itself not only contains magnitude formation (the area of the surface, the size of the volume) but direction information, too.  So instead of $dV$, we integrate over $d\mathbf V$, a 3-vector measure.  Since all 3-vectors in 3d are scalar multiples of each other, we call this simply $\mathbf I \, dV$.
We can meaningfully integrate vector field divergences this way without arbitrarily converting them to two-forms. Rather, the "need" to do so follows from algebra.
$$\int_M (\nabla \cdot \mathbf G) (\mathbf I \, dV)$$
The 3-vector $\mathbf I$ turns directions to orthogonal planes and points to volumes.  I submit to you, for reasons too complex to get into, that $(\nabla \cdot \mathbf G) \mathbf I = \nabla \wedge [\mathbf G \mathbf I]$ using the underlying rules of Clifford algebra that geometric calculus is built on top of.  It is $\mathbf G \mathbf I$ that is the 2-form $\omega$, and it is $\nabla \wedge$ that truly represents, for all objects, the exterior derivative.
What this says geometrically is that there is an equivalence between the following: taking the divergence of a vector field (which yields a scalar) and converting that vector field to its orthogonal planes, finding the derivative orthogonal to all of those planes, and then converting the resulting volume back to a scalar.  That is the nature of the divergence as seen in differential forms, and it becomes very clear using geometric calculus.

Geometric calculus is a natural and powerful extension to the vector calculus you probably already know. The notation is a lot more similar, surely, while still being able to fully express everything you would need to do in higher dimensions.  It is called geometric calculus for a reason, as many of the main proponents try to emphasize, as much as possible, the geometric nature of the underlying algebra and the calculus.
If you become interested in this field, the canonical reference is Clifford Algebra to Geometric Calculus by Hestenes and Sobczyk.  It's a slightly older text, but it contains good material on the relationship between multivector functions and differential forms.  A more modern treatment is Vector and Geometric Calculus by Macdonald, which aims to be a more elementary text.  Geometric Algebra for Physicists by Doran and Lasenby contains many applications and a good chapter on geometric calculus that condenses Hestenes and Sobczyk into something a little more approachable.
Comment: For such an audience I would recommend Macdonald's book (well, both his books, the first being *Linear and Geometric Algebra*, which introduces more of the algebraic aspects). They are lean and efficient at what they do and in maintaining solid contact with traditional matrix algebra, vector algebra, vector calculus, and other more traditional approaches.
Comment: Clifford algebra is beautiful and powerful, but a lot to ask of someone at the freshman/sophomore level learning multivariable calculus. Of course, a good first course should emphasize all the correspondences between vector fields and differential forms in $\mathbb R^3$.
Answer: I'll also toot my own horn and add my own book Multivariable Mathematics: Linear Algebra, Multivariable Calculus, and Manifolds, which I wrote when my students couldn't penetrate Hubbard and Hubbard :) Differential forms come about by adding layers of calculus to the algebra of determinants, which compute oriented $k$-dimensional volumes.
I would also suggest Harley Flanders's Differential Forms with Applications to the Physical Sciences (now in Dover). At a higher level, especially if you're interested in differential geometry and physics, look at a surprisingly concrete exposition, R.W.R. Darling's Differential Forms and Connections.
Comment: Excellent reference on the use of Darling, this textbook got me through my first semester in Grad school and beyond, I adore it (notwithstanding the typos / errors!)
Answer: Here are a couple of references that you might find helpful.
Vector Calculus, Linear Algebra, and Differential Forms by Hubbard and Hubbard. This book is written for undergraduates and integrates (pardon the pun) the study of multivariable calculus, linear algebra, and finishes with differential forms. If you'd like intuition for the geometry of forms in $\mathbb{R}^3$, without too much high-level symbol-pushing, this is a good place to look.
The Geometry of Differential Forms, by Morita, is a monograph which starts with basic definitions and proceeds to describe the utility of differential forms in various contexts, including (if my memory serves) Hodge theory and bundle-valued forms.
Answer: One way (that I discovered accidentally reading a book on exterior algebras) to understand differential forms more intuitively is to see them as linear functionals instead of multilinear antisymmetric maps, what can be done if we see the tangent space of a manifold also as an exterior algebra. In short: I translate the abstraction of differential forms to an abstraction of multivectors in the tangential space, what is to me a very natural and intuitive way to think the notion of volume.
In the following if $V$ is a vector space of dimension $n$ then $\bigwedge V:=\bigwedge V^0\oplus \bigwedge V^1\oplus \ldots \oplus \bigwedge V^n$ is an exterior algebra on $V$.
Let $M$ an smooth manifold of dimension $n$, then every homogeneous differential form $\omega\in \bigwedge T^*M$ of $k$ degree induces a linear functional in $\bigwedge^k TM$ by $\omega(x_1\wedge x_2\wedge \ldots \wedge x_n):=\omega (x_1,x_2,\ldots ,x_n)$, by example any bilinear map $a\wedge b$, for $a,b\in \bigwedge^1 T^*M$, can be seen now as a linear functional on $\bigwedge^2 TM$ using the assignment $a\wedge b(c\wedge d):=a\wedge b(c,d)$.
Now: the meaning of an exterior product on a vector space can be understood as a way to define abstract volumes as if a volume is a kind of vector. Then from this point of view we can see a volume form in $\bigwedge T^*M$ as a linear functional on the volume vectors of $\bigwedge TM$, where by "volume vector" I mean an element of $\bigwedge TM$ of maximum degree.
Then each point of $M$ have attached a volume vector at each of it points, that is, if $\varphi =(\varphi^1, \varphi ^2,\ldots ,\varphi ^n)$ is a chart on $U\subset M$ then $\frac{\partial}{\partial \varphi^1}\wedge \frac{\partial}{\partial \varphi ^2}\wedge \ldots \wedge \frac{\partial}{\partial \varphi^n}$ defines a volume vector at each point of $U$, and following this point of view if $\omega\in \bigwedge^n T^*M$ then the integral $\int_U\omega$ can be understood as a "sum" of the scalars $\omega(\frac{\partial}{\partial \varphi^1}\wedge \frac{\partial}{\partial \varphi ^2}\wedge \ldots \wedge \frac{\partial}{\partial \varphi^n})$, what, at least to me, seems insanely intuitive :).
Comment: . . n i c e . .
Answer: I guess it's already too late though, for someone who landed here in the future, I found this lecture series very useful to grasp the intuition of the differential forms as well as a firm foundation on the theoretical aspects! He explains with very good practices so that we can review if we really get a point of each concept!!

https://www.youtube.com/playlist?list=PL22w63XsKjqzQZtDZO_9s2HEMRJnaOTX7
Answer: Steps which lead me to more of an understanding:
1.  Understand Rotation vs Reflection

The Orthogonal Group $O(n)$ contains all distance preserving transformations of euclidean space. (determinant equal $\pm 1$)
There are two connected components (sets are connected, if they can not be the union of two open sets), i.e. the two components are both connected sets but disjoints, so $O(n)$ itself is not connected - there is a discontinuity.
The special orthogonal group $SO(n)$ is the connected component containing the identity. It is a normal subgroup and we also refer to these operations as rotation. Its elements have determinant 1.
The other connected component are the reflections and have determinant $-1$. Since two reflections have determinant $1$, they do not form a group.

Since the reflections are not connected to the identity it is not possible to continuously transform an object into its reflection. Objects which are different from their reflections (even modulo rotation - continuous distance preserving operations) are called chiral.

image source
chirality allows us to understand why the multilinear maps we will consider later on should be alternating. I.e. swapping two coordinates flips us into the reflections. And swapping two coordinates will get us back to rotation. If one of the coordinates you swap is the same, then this operation only touches 3 dimension, so even in higher dimension it is possible to visualize how this is only a rotation since the other dimensions remain untouched.
2. Understand that determinants measure a (signed) change in volume.
Where the sign indicates whether a reflection took place. The sign is enough, because two reflections are simply a rotation no matter the dimension.
Proofsketch: Let $A$ be a linear map. $\lambda(A \cdot)$ is still a translation invariant measure, so by uniqueness of the Lebesgue measure (as the only translation invariant measure up to constants) it is just the Lebesgue measure up to a constant. To calculate the constant it is sufficient to consider $\lambda( A [0,1]^d)$. For diagonal $A$ it is immediately obvious that it should be the product of entries. But with the singular value decomposition we can decompose any general matrix $A$ into a positive definite diagonal matrix and distance preserving members of the orthogonal group.
3. Understand the set of alternating multilinear maps
Since the set of alternating $n$-multilinear maps in $n$ dimensions is one dimensional, the determinant is its basis with the property that the determinant of the identity is $1$.
Now we understand multilinear maps as a (weighted) measure of volume change. If we were to measure $k$-volume in $n$-dimensions, we have more possible directions in which we can measure volume, so we stop having only one basis element (the determinant) and instead get multiple. But the principle remains the same. :handwaving: (maybe someone can one up me here)


How to edit a ListPreference value from java code in Android

Question: I know how to change the value of a CheckBoxPreference, but I can't get it to work for a ListPreference.
Here is how my preference screen is built:

one CheckBox for the default
A ListPreference to select a color other than the default.

The ListPreference is defined with the key "titleColor", as follows :
CharSequence[] entries = { "Dark grey", "Light grey", "Light red", "Red" };
CharSequence[] entryValues = { "#4c4c4c", "#b5b5b5", "#ab6a68", "#962622" };
final ListPreference color = (ListPreference) findPreference("titleColor");
color.setEntries(entries);
color.setEntryValues(entryValues);

Now, when I select a color I do this:
color.setOnPreferenceChangeListener(new OnPreferenceChangeListener() {
        public boolean onPreferenceChange(Preference preference, Object newValue) {
            System.out.println("In the onClick method");
            System.out.println("change title color");
            Editor editor2 = defaultColor.getEditor();
            editor2.putBoolean("defaultColor", false);
            editor2.commit();
            return true;
        }

    });

And this seems to work, though I'm not sure to do this properly.
Now I have been trying to do something similar when I select the default color, but I can't get the list to either uncheck everything OR check a color that would be the default.
Any idea?
Comment: As far as I now the ListPreference works like a string value.
Answer: In the case of the CheckBox:
If you had a default color CheckBox it would be fixed using android:dependency, but since you don't have it, I guess that's the only way.
Related with the ListPreference, is quite similar:
ListPreference lp = (ListPreference)findPreference("listPreference_key");
lp.setValue("");

You can check the docs to know which methods are available.
Comment: @Sephy: Remember you must pass an entry not a entryValue.
Comment: I added the definition of my ListPreference, because I tried your code but although it works and I obtain no selection in the List, I tried to pass "Red" instead of "", and still got no selection... And I did read the docs before asking my question, that's because I didn't find that i asked it...
Comment: Ok my bad. I was doing it wrong. I was trying to get the color by its entry name ( which appeared to be the logic way for me, but apparently not.) I do have to give it the value and the List takes care of checking the right element for me. lp.setValue("#962622") to get the red. I'll let the empty string though.
Thanks for the help Macarse.


How to run on RedShift sql script file programmatically

Question: Is there any way to run SQL script file (located, for example, on S3) on Amazon Redshift using Java or C#? I can execute stand-alone commands using AWS SDK and Npgsql for .NET, but I can't find the method for running the stand-alone file.
Answer: No, it is not possible to tell RedShift to execute an SQL command stored elsewhere.
You would need to retrieve the script and then pass it to RedShift via JDBC/ODBC, in the same way that you are running stand-alone commands.


Limit Php.ini File Change Per User

Question: I'm using suPHP with Apache 2 and users is allowed to use their own php.ini
How can I limit changes of global php.ini values to only few modications?
Thanks
Comment: Do you mean you want to restrict users to only modifying certain options in php.ini or the number of modifications that can be made in a time period?
Comment: restrict users to only modifying certain options in php.ini. For example, one issue is that default php.ini memory value of 32MB is overwritten to 128MB
Answer: Without knowing about any other in built method, I think your best bet would be to write a page that allows users to modify only the values you want them to. This way, not only can you control which settings they can change, but also limit the values to your own preferences. E.g. only allow upload_max_filesize to be between 16M and 25M.
Comment: thanks for your suggestion, I find a way to allow users to override, there are few lines to comment in the suphp.conf. I will use suhosin to limit all allowed changes.

Limit Php.ini File Change Per User


Cannot bind port on haproxy

Question: I'm trying to bind multiple certs on different ports but I'm getting the following error:
[ALERT] 308/144418 (17671) : Starting frontend https_in: cannot bind socket [10.12.61.12:444]

This is the config
frontend https_in
        bind 10.12.61.11:80
        bind 10.12.61.11:443 ssl crt /etc/ssl/certs/domain1.com.pem
        bind 10.12.61.12:444 ssl crt /etc/ssl/certs/sub.domain1.com.pem

        mode tcp

        acl domain_site1 hdr(host) -i site.domain.com
        use_backend bk_domain1 if domain_site1

        acl domain_site2 hdr(host) -i site.sub.domain.com
        use_backend bk_domain1 if domain_site2

backend bk_domain1
        balance leastconn

        server node1 10.0.20.16:80 check
        server node2 10.0.20.18:80 check

Netstat doesn't show anything listening on port 444 so I'm not sure why this is failing.
This is output of ip addr
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
  link/ether 00:15:5d:14:91:07 brd ff:ff:ff:ff:ff:ff
  inet 10.12.61.55/24 brd 10.12.61.255 scope global dynamic eth0
    valid_lft 691041sec preferred_lft 691041sec
  inet 10.12.61.11/32 scope global eth0
    valid_lft forever preferred_lft forever
  inet 10.12.61.12/32 scope global eth0
    valid_lft forever preferred_lft forever
  inet6 fe80::215:5dff:fe14:9107/64 scope link
    valid_lft forever preferred_lft forever

Comment: In addition to @GregL: Is the ip `10.12.61.12` in the fourth line of your config correct?
Comment: could you specify your distribution and version
Comment: Can you post the output of `ip addr`?
Comment: I'm running Centos 7 with haproxy 1.5.4
Comment: @GregL
ip addr:

eth0:  mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:14:91:07 brd ff:ff:ff:ff:ff:ff
    inet 10.12.61.55/24 brd 10.12.61.255 scope global dynamic eth0
       valid_lft 691041sec preferred_lft 691041sec
    inet 10.12.61.11/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.12.61.12/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:fe14:9107/64 scope link
       valid_lft forever preferred_lft forever
Comment: @AM74: Did you found a solution for this?
Comment: yes, I changed distro to debian and version of haproxy to 1.6 and worked fine
Comment: @AM74: Alright, great...still I'm curious, what caused the error.
Comment: I'm running haproxy 1.6 on Debian now and it's working.
Answer: You might have to update the ports config on your server
/etc/services
snpp            444/tcp                         # Simple Network Paging Protocol
snpp            444/udp                         # Simple Network Paging Protocol

to
https           444/tcp                         # http protocol over TLS/SSL
https           444/udp                         # http protocol over TLS/SSL
https           444/sctp                        # http protocol over TLS/SSL

Comment: Tried that but still can't bind to port 444.

Cannot bind port on haproxy


Create folder automatically during file uploading in C# using Dropbox Core API

Question: Suppose I have a Dropbox account and I want to upload a file in the path "/Client1/2016/filename.txt". There are no folder in my dropbox account either "Client1" or "2016". My requirement is When the file will upload it will automatically create the navigation path folder "Client1/2016/" and upload the file.Is it possible?
I'm using the following code for file upload-
RequestResult strReq = OAuthUtility.Put(
                                    "https://api-content.dropbox.com/1/files_put/auto/",
                                    new HttpParameterCollection
                                        {
                                            {"access_token", "AccessToken"},
                                            {"path", "filepath"},
                                            {"overwrite", "false"},
                                            {"autorename","false"},
                                            {stream}
                                        }
                                    );

Comment: Cross-linking with https://www.dropboxforum.com/hc/en-us/community/posts/205869763-Create-folder-automatically-during-file-uploading-in-C-using-Dropbox-Core-API on the Dropbox API forum.
Answer: What you're asking for is the way that the API already works. If you upload a file to the path /foo/bar, the folder /foo will be created as necessary.


Best way to achieve the following layout

Question: I am pretty new to Drupal, I have a Content Type with a "Field Collection" say like below..

Title
Banner Image
Details - Field Collection Type (Short Title, Description, Links)

I would like to achieve the following style of output, basically upon "User Selection" in the LHS the RHS side should refresh and show the selected Field Collection item's Description. What is the best possible way to achieve this in Drupal 7 (Views with Contextual Filters?, Cycle? or Rotator components etc). Please advice.

Edit: It's only the Content area shown in the picture, there will be a Menu/navigation system wrapping this. Hence this actually becomes 2nd level navigation for that page/node.
thanks,
Vasanth.
Answer: Try Single Page Website module. Its a good option for this kind of website. Just create menu and give proper link.

This module provide the ability to display the content of all menu items of the menu that you indicate in settings on the Single Page Website module.
  On the page 'single_page' the links in the menu will be replaced to the links to the correspondent anchors.
  You can set URL 'single_page' as front page of your site, and your visitors will see the Single Page Website. But it will be the site managed by Drupal 7.
Comment: Hi, thank you for the suggestion - the output is exactly what I am looking for. It's my mistake that I missed to mention it's the 2nd level navigation which I'm looking for on my site. The screenshot is just the "Content Area" and I missed to include the Navigation/menu system around it. So what I am planning is to Theme My Custom Content Type and then include custom jQuery to dynamically show/hide respective regions on RHS. Thanks for your suggestion :)
Answer: Sounds like some form of a slideshow you're after. If you're on D7, try the Field Slideshow module. You may need to do some preprocessing to get the Titles linked as Pagers to the slideshow (currently no option to specify a field collection field as a pager). The module comes with 3 template files for next/prev and pager and main slideshow which you can copy to your theme folder and override.
After you install the module, go to the Manage Display of your content type and set the Format of the field collection field to Slideshow and then click the gear icon to configure.
To use titles as pagers, first go to Manage Display of your content type and set the Pager type to "Image". Ignore the Pager image style configuration.
Then paste the following code to your theme's template.php file and replace THEMENAME with the name of your theme, and replace FIELDNAME with the machine name of your title field inside the field collection:
/**
 * Override template_preprocess_field_slideshow_pager
 * @param array $vars
 */
function THEMENAME_preprocess_field_slideshow_pager(&$vars) {
    $pager = $vars['pager'];

    if($pager == 'image') {
        $items = $vars['items'];

        $thumbnails = array();

        foreach($items as $item) {
            $thumbnails[] = '' . $item['FIELDNAME']['und'][0]['safe_value'] . '';
        }

        $vars['thumbnails'] = theme('item_list', array(
          'items'       => $thumbnails,
              'attributes'  => array(
                'id'    => 'field-slideshow-' . $vars['slideshow_id'] . '-pager',
                'class' => 'field-slideshow-pager slides-' . count($vars['items']),
              ),
        ));
    }
}

Clear all caches.
To include any long text type fields in the slideshow that you couldn't select in the Manage Display gear icon configuration, simply print them inside the field_slideshow.tpl.php, if you know a bit of PHP it should be easy. All the fields should be available to you inside the tpl file in $items array. Look for the line that reads foreach($items as $num => $item)
Comment: Hi, this is exactly what I am looking for. Only part not clear is how I could customize the Field Slideshow's Pager part, any links for this would really helpful. Otherthan this it's exactly what I am looking for, Thanks!!
Comment: @Vasanth edited my answer to include some guide to make titles as pagers.
Comment: Hi Baysaa.. just tried as per your guidelines, and it worked like a charm :-)


Show that $\phi$ is a fibration.

Question: Show that if

is a pullback square and $p$ is a fibration, then $\phi$ is a fibration. I know the definition of a pullback square and the definition of a fibration, but still, I do not know how to proceed, could anyone help me, please?
Comment: You want to show that the map $\phi$ is a fibration. If you attach the diagram for $\phi$ to be a fibration to your pullback square, and you know that $p$ is a fibration, how can you show that $\phi$ satisfies the lifting property?
Answer: There are a number of ways to do this, but this is the most efficient way I'm aware of: Since your square is a pullback square, we can make it more concrete by writing it
$$\require{AMScd}\begin{CD}
E\times_BX @>{\pi}>>E \\
@V{\phi}VV @VVV{p} \\
X @>{f}>> B
\end{CD}
$$
with $E\times_B X = \{(e,x) \in E \times X : p(e)=f(x)\}, \phi(e,x) = x,$ and $\pi(e,x) = e$.
Now suppose we have a homotopy $h: Z \times I \to X$ and a lift $Z \times 0 \cong Z \xrightarrow{\tilde{h}_0} E \times_B X$, with $\phi \circ \tilde{h}_0 (z) = h(z,0)$. Then enlarge the square above to get
$$\require{AMScd}\begin{CD}
Z @>{\tilde{h}_0}>>E\times_BX @>{\pi}>> E \\
@VVV @V{\phi}VV @VV{p}V \\
Z\times I @>{h}>> X @>{f}>> B
\end{CD}
$$
If we define $g = f \circ h: Z \times I \to B$ and $\tilde{g}_0 = \pi \circ \tilde{h}_0: Z \to E$, then it's quick to check that $p \circ \tilde{g}_0 (z) = g(z,0)$ for all $z \in Z$. Then because $p$ is a fibration, there is a homotopy $\tilde{g}: Z\times I \to E$ lifting $g$ against $p$.
From $\tilde{g}$ we should be able to get a homotopy $\tilde{h}: Z \times I \to E\times_BX$ lifting $h$. To define $\tilde{h}(z,t)$ we need a pair $(e,x) \in E \times X$ such that $p(e) = f(x)$. The only course of action that should jump out to us is to get the coordinate in $E$ using $\tilde{g}$ and the coordinate in $X$ using $h$, so let's try $\tilde{h}(z,t) = (\tilde{g}(z,t), h(z,t))$. This is continuous because $\tilde{g}$ and $h$ are.
We need to verify three things: this actually takes values in $E \times_B X$, it lifts $h$ against $\phi$, and lastly that it agrees with $\tilde{h}_0$ on $Z \times 0$.
For the first point, we're supposed to check that $p \circ \tilde{g}(z,t) = f \circ h(z,t)$. But $g = f \circ h$ by definition, and $\tilde{g}$ is lift of $g$ against $p$, so that completes the first verification.
For the second point, we check that $\phi \circ \tilde{h}(z,t) = h(z,t)$. But $\phi$ is projection onto the second factor, so this is immediate.
Finally, for the third point, $\tilde{h}(z,0) = (\tilde{g}(z,0), h(z,0)) = (\pi \circ \tilde{h}_0(z), \phi \circ \tilde{h}_0(z))$ by the definition of $\tilde{g}_0$, the fact that $\tilde{g}$ is a homotopy lifting $g$ against $p$ and extending $\tilde{g}_0,$ and that $\tilde{h}_0$ is a lift of $h$ against $\phi$ on $Z \times 0$. But since $\tilde{h}_0(z) \in E \times_B X$, this equality tells us that $\tilde{h}(z,0)$ and $\tilde{h}_0(z)$ share the same coordinates in $E \times_B X$, so we are done.
Comment: I don't believe that will help. You're trying to show that the pullback of a fibration is a fibration, but the pullback lemma tells us when things can be pullbacks.
Comment: why did you define your $g$ like this?
Comment: can not we use the pullback lemma?
Comment: What are you referring to as the "pullback lemma"?
Comment: https://proofwiki.org/wiki/Pullback_Lemma
Comment: Why we need this pair to define $\tilde{h}$? in the following statement that you said  " To define $\tilde{h}(z,t)$ we need a pair $(e,x) \in E \times X$ such that $p(e) = f(x)$."
Comment: Why we need to verify those things specifically "We need to verify three things: this actually takes values in $E \times_B X$, it lifts $h$ against $\phi$, and lastly that it agrees with $\tilde{h}_0$ on $Z \times 0$."
Comment: I am speaking specially about the first and last conditions that should be satisfied.
Comment: I meant to ask also: how do you know that those are the conditions that we need to check?
Comment: Regarding your first question: that's because we need to make sure we are actually mapping into the pullback $E \times_B X$. Second question: (1) we need to make sure the function is well-defined, (2) and (3) are the conditions to satisfy the definition of fibration. Third question: whenever you define a function you want to make sure it does in fact have image in the codomain and is defined on all of the domain, so that's why I knew I had to check that. For the others, I noted that to show that $\phi$ is a fibration, I need do prove it satisfies the definition of fibration.
Comment: is the pushout of a fibration a cofibration? or the pushout of a cofibration is a cofibration? (I want to apply Eckmann Hilton duality to this problem this is why I am asking this question)

Show that $\phi$ is a fibration.


Can Windows nanoserver run inside Linux docker?

Question: https://msdn.microsoft.com/en-us/virtualization/windowscontainers/deployment/deployment_nano talks about running a nanoserver docker image under Windows:

docker pull microsoft/nanoserver

Does this mean that we should be able to run nanoserver anywhere that Docker runs (e.g. Linux)?
Comment: https://en.wikipedia.org/wiki/Operating-system-level_virtualization
Answer: Containers provide process isolation on a shared OS, rather than OS isolation on shared hardware that you get with VMs. Since the OS is shared, the host OS needs to be able to run the desired binaries. You will see this in the architecture of the docker engine and the architecture of the image you want to run, they must be compatible:
<code>$ docker system info --format '{{.OSType}} {{.Architecture}}'
linux x86_64

$ docker image inspect busybox --format '{{.Os}} {{.Architecture}}'
linux amd64
</code>
If you try to run an architecture that isn't compatible with your host, you will get an error because the binary format isn't recognized by the kernel:
<code>$ docker image pull --platform arm64 busybox:latest
latest: Pulling from library/busybox
acafde7ce2e7: Pull complete
Digest: sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812
Status: Downloaded newer image for busybox:latest

$ docker run -it --rm busybox:latest echo hello
standard_init_linux.go:190: exec user process caused "no such file or directory"
</code>
Desktop versions of Docker, and the Windows Server version of Docker, include a Linux VM under the covers to run Linux containers (Linux containers are the dominant container environment, so Docker implements this using Linuxkit, to simplify the developer workflow). For Windows, there is a switch in the engine to use the Linux VM or run native Windows containers.
However, Docker does not have an embedded VM for Windows to run their binaries on a Linux host (since Windows is not open source and would require licensing), so the only way to run Windows native containers is on a Windows host.
Answer: short and simply: no
Containers are about Kernel/OS virtualisation.
It's a group of processes you run distincly in a container hosts that shares the kernel. If that container host is a Linux deamon it cannot share the windows kernel, which the nanoserver image needs
Comment: So docker is more about virtualization of the local OS whereas full-fledged VMs (such as VMWare) are about virtualization of any OS. Is that correct?
Comment: No, classic VM hosts like vmware virtualize the hardware.
Answer: maybe - if you are running kubernetes: https://kubernetes.io/docs/getting-started-guides/windows/
Comment: Running windows native containers requires a windows host.


Is the ball live if a fielder drops a foul ball?

Question: I was watching a game and the bases were loaded. The batter hit pop up foul ball near first base in play. The first baseman dropped the ball and the game moved on. Isn't this a live  ball when the first baseman dropped it, meaning the runners could have advanced instead of the at bat continue
Comment: Follow-up question: is an error charged to the first baseman (assuming it was an easy catch)?  I believe the answer is `yes`.  I once saw Sam Horn drop two pop-ups in one inning (one was foul).
Comment: @jahroy please ask a separate question and link to this question for reference
Comment: @edmastermind29 - Um... nevermind.  It's a yes/no question that can easily be researched in about a second.  The answer is `yes`.  I couldn't remember the answer for a moment, so I figured I'd add it to the conversation...
Comment: @jahroy I wasn't aware. I thought it would be a judgment call. Thanks for adding.
Comment: @edmastermind29 - Well... all errors are judgement calls by the scorekeeper.  The question was "_can there be an error on a_ **foul** _ball" ([is an error charged to the first baseman (assuming it was an easy catch)?](http://sports.stackexchange.com/questions/2685/is-the-ball-live-if-a-fielder-drops-a-foul-ball#comment5474_2685)).  The answer is `yes`.
Answer:
Isn't this a live ball when the first baseman dropped it, meaning the runners could have advanced instead of the at bat continue?

This is a dead ball and runners return to their respective bases. See Rule 5.09(e) below.

Rule 2.00 in the 2013 MLB Official Baseball Rules state (emphasis mine):

A FOUL BALL is

a batted ball that settles on foul territory between home and first base, or
between home and third base, or
that bounds past first or third base on or over foul territory, or
that first falls on foul territory beyond first or third base, or
that, while on or over foul territory, touches the person of an umpire or player, or
any object foreign to the natural ground

A foul fly shall be judged according to the relative position of the ball and the foul line, including the foul pole, and not as to whether
  the infielder is on foul or fair territory at the time he touches the
  ball.

Rule 5.09 (emphasis mine):

The ball becomes dead and runners advance one base, or return to their
  bases, without liability to be put out, when—

e) A foul ball is not caught, in which case runners return to their bases. The umpire-in-chief shall not put the ball in play until all runners have retouched their bases
Comment: Additional note is that if the player catches the foul ball, runners are allowed to tag and attempt to advance.  This is why on a rare occasion a fielder will intentionally drop a foul ball.


How to reconstruct PCM samples from raw bytes?

Question: I am recording audio with the package react-native-audio-record. I want to use the temporary data to visualize stuff. As in the example (function AudioRecord.on) here:
const options = {
  sampleRate: 16000,
  channels: 1,
  bitsPerSample: 16,
  wavFile: 'test.wav'
};

AudioRecord.init(options);

AudioRecord.on('data', data => {
  const chunk = Buffer.from(data, 'base64');
  console.log('chunk size', chunk.byteLength);
  // do something with audio chunk
});

How can I decode the variable chunk to PCM? As far as stated in another forum, this chunk does not include the headerBytes of the .wav File.
Comment: This can be broken into multiple steps, you're asking how to convert base64 to a byte array https://stackoverflow.com/questions/21797299/convert-base64-string-to-arraybuffer?answertab=votes#tab-top and then how to convert an array buffer to an audio buffer https://stackoverflow.com/questions/50512436/how-to-convert-arraybuffer-to-audiobuffer#56747545
Comment: Your options then are wrap the native Audio APIs (tedious) or perform this in pure JavaScript  (also tedious). There are still multiple parts to those questions. The first is changing bytes to PCM, the second is [how to DFT / FFT in JavaScript](https://github.com/trekhleb/javascript-algorithms/tree/master/src/algorithms/math/fourier-transform)
Comment: Then there's your question: _How to reconstruct PCM samples from raw bytes?_
Comment: thanks for the reply: i just edited the question. I believe, I already have the array buffer via Buffer.from(data,"base64"). As I am working with react-native, i can not use BaseAudioContext. Is there a workaround?
Comment: The fft part should be quite clear - edited the question again. I'm stuck how to get to PCM.
Answer: As stated in the comments, reconstructing the PCM in pure JavaScript is tedious. I used therefore the node-package node-wav (only works if header is included) as follows:
Store the header of the specific wav options:
const header = Buffer.from([82, 73, 70, 70, 248, 167, ... ])

and concatenate the data as follows:
import wav from 'node-wav';

    AudioRecord.on('data', data => {
          const chunks = [header]
          const chunk = Buffer.from(data, 'base64');
          chunks.push(chunk)
          const pcm = wav.decode(Buffer.concat(chunks)).channelData
        });

This approach works only if you can reconstruct the header. The package node-wave on the other hand is stated as high performance WAV decoder and encoder .


Split multi-page tiff with python

Question: What's the best way to split a multi-page TIFF with python? PIL doesn't seem to have support for multi-page images, and I haven't found an exact port for libtiff for python.  Would PyLibTiff be the way to go?  Can somebody provide a simple example of how I could parse multiple pages within a TIFF?
Comment: PIL has limited support to "multi-image" documents - it cana t least laod, and leat you deal with, individual frames in animated GIFs - I don't know if the TIFF loading plug-in allows you to read each page, though. Can you post a link to one of your tiffs?
Comment: Unfortunately no... The material I'm dealing with right now is sensitive, and I have no clue as to how to generate a generic multi-page tiff from scratch.  I've tried exporting multiple layers in gimp to no avail.  Do you have any code examples?  I could try them out on my machine.
Comment: You might try Python Wand. It is built upon ImageMagick, which can do the separation.
Comment: Hello, I had the same problem and wrote this module to operate with multipage tiff
https://github.com/mpascucci/multipagetiff
You can open the multipage tiff and save pages one by one... if you really want to do it in python
Answer: A project (disclosure: which I am one of the main authors, this question was one of the things that prompted me to work on it) which makes this is easy is PIMS.  The core of PIMS is essentially a cleaned up and generalized version of the following class.
A class to do basic frame extraction + simple iteration.
import PIL.Image
class Stack_wrapper(object):
    def __init__(self,fname):
        '''fname is the full path '''
        self.im  = PIL.Image.open(fname)

        self.im.seek(0)
        # get image dimensions from the meta data the order is flipped
        # due to row major v col major ordering in tiffs and numpy
        self.im_sz = [self.im.tag[0x101][0],
                      self.im.tag[0x100][0]]
        self.cur = self.im.tell()

    def get_frame(self,j):
        '''Extracts the jth frame from the image sequence.
        if the frame does not exist return None'''
        try:
            self.im.seek(j)
        except EOFError:
            return None

        self.cur = self.im.tell()
        return np.reshape(self.im.getdata(),self.im_sz)
    def __iter__(self):
        self.im.seek(0)
        self.old = self.cur
        self.cur = self.im.tell()
        return self

    def next(self):
        try:
            self.im.seek(self.cur)
            self.cur = self.im.tell()+1
        except EOFError:
            self.im.seek(self.old)
            self.cur = self.im.tell()
            raise StopIteration
        return np.reshape(self.im.getdata(),self.im_sz)

Comment: What is the purpose of the numpy reshaping?
Comment: @speedplane Because `getdata` returns a flattened iterable of the image http://effbot.org/imagingbook/image.htm#tag-Image.Image.getdata
Comment: @tacaswell Is there a way to convert extracted frame to png or jpeg files after 'get_frame()' is done?
Comment: @comproch `PIL.Image.fromarray(stack_wrapper_instance.get_frame(i)).save('name.png')`
Answer: Imagemagick worked for me real good. Wnen splitting a tiff file, basically converting from tiff to tiff, one can use a flag to force saving output files to individual tiff files. To do that, try
convert input.tif output-%d.tif

The %d operator is a C-Printf style %d. So, if you need a 3 field running sequence, you can say
convert input.tif output-%3d.tif

and so on.. %d is replaced by "scene" number of the image. Now, scene numbers may or may not always start with 0 (or 1, if you want it that way). To setup a sequence the way you want, try
convert input.tif -scene 1 output-%3d.tif

This would start the sequence right from the count you provided.
convert -scene 1 input.TIF output-%d.TIF
output-1.TIF
output-2.TIF
output-3.TIF

Magick indeed!! :)
This link to documentation has more details. This works on my windows machine too.
Answer: I do use ImageMagick as external program to convert multi-page fax into viewable PNGs:
/usr/bin/convert /var/voip/fax/out/2012/04/fax_out_L1_17.tiff[0] -scale 50x100% -depth 16 /tmp/fax_images/fax_out_L1_17-0-m.png

does convert first page to PNG
aaa.tiff[1] would be second page, and so on.
Or to extract all images, do:
convert -verbose fax_in_L1-1333564876.469.tiff a.png
fax_in_L1-1333564876.469.tiff[0] TIFF 1728x1078 1728x1078+0+0 1-bit Bilevel DirectClass 109KiB 0.030u 0:00.030
fax_in_L1-1333564876.469.tiff[1] TIFF 1728x1078 1728x1078+0+0 1-bit Bilevel DirectClass 109KiB 0.020u 0:00.010
fax_in_L1-1333564876.469.tiff[2] TIFF 1728x1078 1728x1078+0+0 1-bit Bilevel DirectClass 109KiB 0.020u 0:00.010
fax_in_L1-1333564876.469.tiff=>a-0.png[0] TIFF 1728x1078 1728x1078+0+0 1-bit Bilevel DirectClass 12KiB 0.030u 0:00.019
fax_in_L1-1333564876.469.tiff=>a-1.png[1] TIFF 1728x1078 1728x1078+0+0 1-bit Bilevel DirectClass 8KiB 0.040u 0:00.039
fax_in_L1-1333564876.469.tiff=>a-2.png[2] TIFF 1728x1078 1728x1078+0+0 1-bit Bilevel DirectClass 32KiB 0.070u 0:00.070

So, to just split one multi-page TIFF into many-page TIFF you would have to execute:
convert in-12345.tiff /tmp/out-12345.tiff

and then work with temporary files: /tmp/out-12345-*.tiff
However ImageMagick can do a lot of processing, so you can probably achieve your desired result in one command.
Answer: The following splits a tif file with multiple frames into tif files where each file is one frame.
def parse_tif(filePath):
    img = Image.open(filePath)
    for i in range (numFramesPerTif):
        try:
            img.seek(i)
            img.save('Block_%s.tif'%(i,))
        except EOFError: #end of file error

Comment: numFramesPerTif - how do I calculate it? It's currently undefined.
Comment: numFramesPerTif = img.n_frames
Comment: It's the number of images inside the multi-image tiff file. I don't know how to determine that though
Answer: After installing ImageMagick in Windows. Try the following below in Command Prompt
1. Test ImageMagick tif file convertion
magick convert input.tif out.tif

2. Convert one multi page tif file to multiple files
magick convert -verbose input.tif otput.png

Answer: You could convert it to PDF and use pyPDF to split the pages
Comment: I rather not convert to PDF, split, and convert back to TIFF unless there is absolutely no other way to split multi-page TIFFs.  It seems like a backwards way of extracting individual pages, but feel free to correct me if I'm wrong...

Split multi-page tiff with python


Como suavizar uma curva e definir ylim para uma dada função usando ggplot2?

Question: Consegui o seguinte gráfico com a função curve:
f1 <- function(x){x^4 - 6*(x^2)}
curve(f1, xlim = c(-3, 3), ylim= c(-10, 5), ylab = expression(x^4 - 6*(x^2)))

No entanto, gostaria de usar o pacote ggplot2 para reproduzir este mesmo gráfico (com os mesmos limites de x e y).
Consegui desenvolver este simples código:
x <- c(-4:4)
y <- c(x^4 - 6*(x^2))
ggplot() +
   geom_line(aes(x=x, y=y))

Mas, precisaria de suavizar a curva e estabelecer os limites de x e y. Como posso obter um gráfico semelhante ao primeiro utilizando o ggplot?
Answer: Você vai usar o stat_function() para a função e ylim() e xlim() para definir os limites:
library(ggplot2)
x <- c(-4:4)
f1 <- function(x){x^4 - 6*(x^2)}
ggplot(data.frame(x), aes(x)) + stat_function(fun=f1) + ylim(-10, 5) + xlim(-3, 3)

Answer: Aqui está um código exemplo:
x = seq(from=-4, to=4, length.out=1000)
y = c(x^4 - 6*(x^2))
print(ggplot(data=data.frame(x=x, y=y), aes(x=x, y=y)) +
      geom_point(colour='red') +
      stat_smooth(se=F, colour='blue') +
      coord_cartesian(xlim=c(-3, 3), ylim=c(-10, 5)))

Ou com uma lenda:
print(ggplot(data=data.frame(x=x, y=y), aes(x=x, y=y)) +
      geom_point(aes(colour='source')) +
      stat_smooth(aes(colour='fit'), se=F) +
      coord_cartesian(xlim=c(-3, 3), ylim=c(-10, 5)) +
      scale_color_discrete(name='type'))


Using entity_lookup inside an entity_generate migration process plugins

Question: I'm trying to use an entity_lookup process plugin inside an entity_generate process plugin. The YML looks like this
field_validation:
    plugin: entity_generate
    access_check: false
    entity_type: node
    source: validation_name
    value_key: title
    bundle: validation
    ignore_case: true
    values:
      title: validation_name
      body: validation_description
      field_validation_type:
        plugin: entity_lookup
          access_check: false
          ignore_case: true
          source: validation_type

The error I'm getting is

Uncaught PHP Exception
  Drupal\Core\Config\UnsupportedDataTypeConfigException: "Invalid
  data type in config xxxxxx, found in xxxxxx : A colon cannot be used
  in an unquoted mapping value at line 282 (near "  source:
  validation_type")." at xxxxxx line 117, referer: xxxxxx

Because I'm relatively new in using the Migrate/Migrate Plus modules, I'm not sure if the error is triggered because I'm doing something wrong with the syntax or just because I can't use entity_lookup inside an entity_generate. So, the question is: can I do something like this, or I'm forced to use a separate migration instead of entity_generate and the call it with migration_lookup?
[later edit, just for context]
my field definition:
-
 name: validation_type
 label: 'Validation Type'
 selector: $.Description.Validation[*].Type #item_selector is set to $.Schema[*]

the relevant part of the source JSON:
{
  "Schema": [
    {
      "Description": {
        "Validation": [
          {
            "Type": "CUSTOM_TYPE",
            "Description": "lorem ipsum"
          }
        ]
      }
    }
  ]
}

Answer: It seems that you actually can use entity_lookup inside entity_generate. The problem here was with the wrong indentation bellow plugin: entity_lookup. Yup... it was that kind of "bug" (feeling embarrassed) :)


Gerenciar variáveis com sessão PHP

Question: Preciso pegar o valor de um input text armazená-lo e utilizá-lo em uma outra página que se abre automaticamente (popup) quando o botão Enviar do form é pressionado.
Ele consegue fazer isso, porém o que está acontecendo é que quando aperto o botão a primeira vez ele não pega, quando aperto para enviar o form uma segunda vez ele envia o valor que estava na primeira e assim por diante... ideias?
Comment: tente colocar `action="NomeDoPopUp"` no seu formulario.
Comment: continua dando o mesmo problema e além disso tem mais uma coisa, a página 1 é um form que faz uma entrada no banco de dados e só após essa entrada é criado um ID (autoincremental) que eu vou usar no popup... logo o action do primeiro form não pode ser o popup... =/
Comment: Vou tentar explicar o caso.
Página1: faz o cadastro principal gerando automáticamente um código na tabela do banco
Página2 (popup): traz uma lista de checkboxes com opções e que serão atreladas ao código que citei acima em uma outra tabela do banco
Por conta disso eu preciso que o primeiro form seja enviado primeiro e só então  uma outra página venha com os check.
O PopUp é somente questão estética, para não abrir uma página somente para isso... se tiver alguma outra solução estou ouvindo... (lendo haha)
Answer: Seu código está errado. Você está definindo o valor da $_SESSION['nome'] com um $_POST que ainda nem existe e a mesma situação se repete ao chamar o popup.
Faça assim
<?php

    session_start('professor');

    $chamarPopup = false;

    //Verifica se existe um $_POST chamado nomeP, que é o name do seu input.
    if (isset($_POST['nomeP']) && !empty($_POST['nomeP'])) {
        $_SESSION['nome'] = $_POST['nomeP'];
        $chamarPopup = true;
    }

?>

<form method="post" name="formTest">
    <input type="text" name="nomeP" value="Nome Professor">
    <input type="submit" value="PopUp">
</form>

No final da sua página, depois da tag  coloque isso
<?php

    if ($chamarPopup === true) {
        echo "";
    }

?>

No popup não precisa mudar nada.
Só pra constar: eu não recomendo usar popup, pois, a maioria dos navegadores modernos bloqueiam todos por padrão e, usuários menos experientes não sabem como fazer para que eles sejam exibidos.

Gerenciar variáveis com sessão PHP


How to attach profile type to specific user role?

Question: I have multiple profile types and multiple user roles.  I want to combine specific profile types with users of specific roles.  For example, customer roles have their own profiles, content authors have their own profiles, etc.
Is there a way to assign them together?
Answer: I was able to do this with permissions.


Como configuro o @media para celular?

Question: Tenho um celular Moto G e não consigo definir o @media para funcionar para ele.
Reduzindo a largura do navegador funciona perfeitamente, mas no dispositivo celular não se ajusta de acordo com o @media.
Abaixo coloquei o background do form para ficar cinza, e nada acontece. Veja o que fiz?
@media(max-width: 574px) {
.menutopo{
  display: none;
}
figure.logo{
  float: left;
  width: 100%;
  height: 80px;
  background-color: #FF8922;
  box-sizing: border-box;
}

section.frmbuscaguia{
  float:left;
  width: 100%;
  min-width: 200px;
  height: auto;
  background-color: #ccc;
  text-align: center;
  margin-bottom: 5px;
  box-sizing: border-box;
}
}

HTML

Buscar no Guia Comercial

            <form>
                <input type="text" name="buscar">
                <button type="submit">Pesquisar</button>
            </form>

Comment: já tentou `@media screen and ()...`?
Comment: sim, já tentei!
Comment: Se for o chrome, so vai funcionar se limpar o histórico do navegador.
Comment: Já tentou o `@media only screen and (max-width: 574px)...` ?
Comment: @Newtech mas nada de funcionar
Comment: Posta uma imagem de como está ficando a página na tela de celular. Verifica se a meta tag viewport está configurada
Comment: Só para constar, fiz uns teste aqui e é muito provável que o seu problema seja o viewport mesmo, Use a `` que a Bruna postou na resposta que vc vai resolver o problema. E para entender pq resolveu leia esse artigo https://developer.mozilla.org/pt-BR/docs/Mozilla/Mobile/Viewport_meta_tag
Comment: Configurou a tag `` no ``?
Answer: Eu testei seu código, sem nenhuma modificação e está funcionando corretamente.
Como você não enviou todo o código, a única coisa que eu fiz foi acrescentar um head e finalizar o código com o próprio padrão do bootstrap, vou mandar abaixo para você ver e conferir.
Você precisa verificar se está importando corretamente o css, ok?
Espero ter ajudado, lembrando que você precisa testar na resolução que está propondo no @media.

<!DOCTYPE html>

 <meta charset="utf-8" />
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <title>Page Title</title>
 <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
 <link rel="stylesheet" type="text/css" media="screen" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"/>
 <link rel="stylesheet" type="text/css" media="screen" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css"/>
 <link rel="stylesheet" type="text/css" media="screen" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css"/>
 <link rel="stylesheet" href="css/style.css">

Buscar no Guia Comercial

  <form>
   <input type="text" name="buscar">
   <button type="submit">Pesquisar</button>
  </form>

Comment: hehe é costume de ser cordial sempre, mas valeu pela dica!
Comment: Cumprimentar é bom e muito educado, mas no Stack Overflow costumamos ser o mais breve e direto possível, não precisa enviar abraços nem dizer olá. Mesmo assim, essa é uma boa resposta.

Como configuro o @media para celular?


How can I display my response messages that I send in node js on my frontend with Reactjs

Question: I am trying to build a basic login/signup app with the mern stack.I am new to it. While making the backend with node and express I have added lot of json messages in the code like-
return res.status(400).json({message:"Unable to find user by this id"})
return res.status(404).json({message:"User doesnt exist"});
How can I display this json data to my frontend with react?
This is the handleSubmit code-
const handleSubmit=(e)=>{
    e.preventDefault()
    console.log(inputs);
    if(isSignup){
      sendRequest("signup")
      .then((data)=>localStorage.setItem("userId",data.user._id))
      .then(()=>dispath(authActions.login()))
      .then(()=>navigate("/blogs"))
    }else{
      sendRequest()
      .then((data)=>localStorage.setItem("userId",data.user._id))
      .then(()=>dispath(authActions.login()))
      .then(()=>navigate("/blogs"))
      .then(data=>console.log(data))

    }
  }

This is the return div code-
      <form onSubmit={handleSubmit}>
      <Box display="flex" flexDirection={'column'}
      alignItems='center'
      maxWidth={400}
      justifyContent={'center'}
      boxShadow="10px 10px 20px #ccc"
      padding={3}
      margin="auto"
      marginTop={5}
      borderRaidus={5}>
        <Typography variant="h2" padding={3} textAlign="center">{isSignup?"Signup":"Login"}</Typography>
        {isSignup && <TextField name="name" onChange={handleChange} value={inputs.name} placeholder='name' margin="normal"/>}
        <TextField name="email" onChange={handleChange} value={inputs.email} type={'email'} placeholder='email' margin='normal'/>
        <TextField name="password" onChange={handleChange} value={inputs.password} type={'password'} placeholder='password' margin='normal'/>
        <Button type={"submit"} variant="contained" sx={{borderRadius:3,marginTop:3}} color="warning">Submit</Button>
        <Button onClick={()=>setIsSignup(!isSignup)} sx={{borderRadius:3,marginTop:3}}>
          Change to {isSignup ?"Login":"Signup"}</Button>
      </Box>
      </form>

  )
}

export default Auth

Comment: Can you show the `handleSubmit` code?
Comment: I have added the handelSubmit code
Answer: Just try this one.
const handleSubmit = async(...) => {
 ...
 try{
  const response = await axios.post(...);
  const data = await response.json();
  console.log(data);
 }catch(){
  ...
 }
}

Comment: I tried this but it did not work. I have added the handleSubmit code in the question could you check that?


What is meant by a "release"?

Question: In a Tom Cosm video tutorial on making beats in Ableton Live, he states (12:20):

See, we have that kind of suck now. That's another kind of tension
  thing. I find with these kind of tension things, you need a release
  after or just a few beats after to kind of get them some resolution.

What is he meaning by a "release", and where can I look to get more theory or understanding on that concept?
Answer: He's referring to tension and release, an element of music theory which states essentially that music uses tension to create an expectation, followed by a release that fulfills the expectation. Dissonance and consonance are a very common method for tension and release, but you can also use rhythm, dynamics, and other musical elements to create tension and then release it.
For example, when a song gets very quiet, we usually expect it to get loud. When somebody sings do-re-mi-fa-sol-la-ti, there's a strong expectation that they follow with do. When somebody knocks the rhythm "shave and a haircut," we expect "two bits" to follow. All of those are ways of creating tension, and the parts that satisfy it are called the release.
Answer: Tension and release is a very important concept in music structure. A lot of chord schemes are built so that a certain chord will cause a feeling of tension, which can then be released or prolonged as desired. This is something our ears pick up instinctively; anyone with or without musical knowledge will subconsciously notice this tension and release.
Take a look at this article if you want an in-depth explanation -- I think it is quite nicely written.
Comment: Ack, beat me by 25 seconds! Nice article on melodic tension and release. I suspect that the video might be using a different kind of tension though. Anyway, +1.

What is meant by a "release"?


Does the sitemap.xml file does from the xmlsitemap module actually get generated?

Question: I see that there is a 1.xml file generated in the sites/default/files/xmlsitemap folder, but I do not see a file in the Drupal root. However, when I call http://www.example.com/sitemap.xml I do get the info. Is this the normal behaviour?
Answer: Yes, that's normal behaviour.
xmlsitemap_menu() includes a route for /sitemap.xml specifically.
Answer: This is utterly normal behavior: the module caches the sitemap in the files directory, then delivers it via http://www.example.com/?q=sitemap.xml which gets mapped to http://www.example.com/sitemap.xml with clean URLs.


Statistical approaches to detect overfitting in simple models

Question: I read here that there are statistical approaches to assess whether a tractable machine learning model (e.g., a linear regression model) overfits a dataset:

Simpler models that have originated in statistics often don't need test datasets. Instead, what degree the model is overfit can be calculated directly as statistical significance: a 'p-value'.
These statistical methods are powerful, well established, and form the foundation of modern science. The advantage is that the training set doesn't ever need to be split and we get a much more precise understanding of how confident we can be about a model.

That is a huge advantage: we don't need to split the training data or perform cross-validation if the model is simple enough.
Unfortunately I haven't seen such a thing in practice. I know we get a p-value for each fitted coefficient when fitting regression models like here, but I don't believe these p-values determine overfitting.
Do you have a more elaborate explanation and preferably some examples of this use case?
Answer: The various information criteria (Bayesian, Aikake) and e.g. Cp try to estimate this. So for example, the Bayesian information criterion (https://en.m.wikipedia.org/wiki/Bayesian_information_criterion) measures fit by likelihood, except the final score is also penalised if you have more parameters because then you're more likely to be overfitting. Mallow's Cp and the Aikake Information Criterion works on similar principles but are tweaked slightly differently.
However, if you have access to a computer, cross validate. Cross validation is the solution to the problem.
Comment: Thank you. It'd be appreciated if you can expand your first sentence more.
Comment: Sorry I haven't seen that you added more clarification to your answer till now!

So with simple models, model selection can tell if a model is overfitting. However, it seems that this is applicable to complex models too; besides, I don't see how `p-values` are utilised here - reflecting on the source in my question..