XFS/EXT4: Why is actual file size on disk larger than apparent size? Clarification

Question: On a CentOS 7.1.1503 (3.10.0-229.el7.x86_64) system running KVM/qemu on an XFS filesystem, I'm attempting to work out what specifically is adding additional space to a VM diskimage file.
<code>$ ls -ls --block-size=1k
15728648 -rw-------. 1 qemu qemu 15728640 Aug  8 07:50 original.img
15728640 -rw-------. 1 qemu qemu 15728640 Aug  8 08:25 original.imgcopied
 2288960 -rw-------. 1 qemu qemu 15728640 Aug  8 09:36 original.imgsparsified
15728640 -rw-------. 1 qemu qemu 15728640 Aug  7 13:23 thickprov.img
       0 -rw-------. 1 qemu qemu 15728640 Aug  7 13:28 thinprov.img

$ filefrag *
original.img: 1 extent found
original.imgcopied: 1 extent found
original.imgsparsified: 713 extents found
thickprov.img: 1 extent found
thinprov.img: 0 extents found
</code>
This is the filesystem these files reside on:
<code># xfs_info /dev/mapper/centos_centsager-home
meta-data=/dev/mapper/centos_centsager-home isize=256    agcount=4, agsize=15968512 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=63874048, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=31188, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
</code>
From what I have been able to understand, the original.img file is showing the actual size on the filesystem is 8kb larger than the apparent size. I'd like to get a better understanding of this.  I also have a 50GB image file on an ext4 filesystem on another box that is showing its actual disk space use is 92k larger than its apparent size.  Both of these files were thick provisioned via KVM/qemu/libvirt and have been used in test VMs for various things, but their internal filesystems were never close to full.
The other files listed are just the result of some testing I was doing to get my head wrapped around what is happening. These reported sizes make sense to me.
<code>15728640  15728640  original.imgcopied     # 'cp' of original.img
 2288960  15728640  original.imgsparsified # 'virt-sparsify' of original.img
15728640  15728640  thickprov.img          # Fresh VM create with same parameters as original.img
       0  15728640  thinprov.img           # Same VM create but with a sparse allocation.
</code>
So, is the extra 8k in the original.img file due to extent information, file fragmentation, or something else entirely?
Thanks.
Comment: you need to look what is an sparse file.
Comment: Yep I understand how sparse files are allocated with empty blocks and that the 'cp' command without extra switches will fill in those null blocks and the actual file size will match the allocated size.  I'm not understanding the extra 8k in the actual file size of the original.img file though.  On another image file I have that is 50GB the actual file size is 92k larger. So I'm trying to understand where those extra blocks are coming from.
Comment: There may be a connection with [this question on unix.SE](http://unix.stackexchange.com/questions/330753/rsync-of-sparse-qemu-image-increases-disk-size). Maybe you can bring any insight there?
Answer: When a write triggers allocation of a new extent in a file, XFS will sometimes speculatively allocate extra extents beyond those required for the current write operation. The idea is that future writes may well cause the file to grow further, so by pre-emptively allocating extra extents extra performance can be gained.


Date formatting issue in python

Question: import time
import sys
import json as json
import spacy
from datetime import datetime
from dateutil.parser import parse
def format_source_date(date):
    if date != None:
        try:
            try:
                dt = parse(date)
                date_formatted=dt.strftime('%m/%d/%Y')
                print(date_formatted)
            except:
                print(date)
        except ValueError:
            print(date)
def has_seperator(text):
    if ',' in text or '/' in text or '-' in text:
        print(text)
        return True
    else:
        return False

date = 'Sep 25,2017'
has_seperator(date)
format_source_date(date)

The required answer is 09/25/2017. Instead taking current year 2021 . Is there any solution to solve this issue
Comment: It seems fairly clear to me. The output of `format_source_date` is `09/25/2021` when they expect it to be `09/25/2017`.
Comment: I'm afraid this question isn't very clear.  Perhaps boil it down to what you're trying to do and ask again.
Comment: I have give a date "Sep 25,2017". But if I run this program I am getting the output as 09/25/2021. I need to get output as 09/25/2017. This program is taking current year instead of 2017.
Answer: Seems your date string is not a valid format,
See the following
>>> parse('Sep 25,2017')
datetime.datetime(2021, 9, 25, 0, 0)
>>> parse('Sep 25 2017')
datetime.datetime(2017, 9, 25, 0, 0)

so already have a has_seperator function, use it remove the character or use a supported format
Comment: @Ajay You could do something like `parse(date.replace(",", " "))`.
Comment: A document is in date format like 'Sep 25,2017'.  How to change to Sep 25 2017 by removing comma
Comment: I'm surprised, I thought `dateutil` was smarter than that.

Date formatting issue in python


Splitting guitar signal and mic signal for recording in a live setting

Question: I've read already some things about splitting signals here but im still unsure about how to make mine.
So what i have is one acoustic guitar with an undersaddle pickup and a mic (K&K). Both signals go into a K&K preamp where I split the signals (undersaddle / mic). The undersaddle goes through a some pedals and later on in a mixer (or amp). The mic goes direct in a mixer (or amp). I'd like to record them separately to a zoom H6. There are no real good possibilities on the mixer to send the signals separately. So I was thinking using A) a simple cable splitter B) a splitter from radical engineering or similar.
As I would need 2 splitters (mic & undersaddle) this might become quite expansive for option B). So I'm really not sure (as i don't send the signal to 2 amps) how A) will affect the sound and all technical issues (grounding).
Similar problem with the mic (shure 58) from my singer....
Any clues?
Greetings
EDIT:
You right the original question i a bit misleading due to the equipment. We can simplify it. So lets imagine there are two acoustic guitars in a live setting. Sometimes they go into a PA , sometimes into an AMP directly. So as you have mentioned the option with the PA. Its very valid but sometimes the Mixer is that shitty or there is no mixer and I have to go directly to the AMP. In the amp there is an output which i could use, but for the mix. The second option with the DI is also known to me, but it might be quite expansive. So I was thinking to use just some Y cables. But here I heared some issues: the signal strength and grounding...
Comment: Thanks for clarifying. Though my answer is the same as I wrote below :) Using some Y-cables might work but is definitely in the "gerry-rig" category IMO. At a certain point you do need to spend more to do it right. If your area has a gear rental place, you could look into renting DI boxes and/or a mixer to save some dough.
Answer: I'm not sure I'm clear on your signal chain, but I'm inferring that your K&K preamp has 2 discrete inputs and 2 discrete outputs. And that the Zoom has 2 discrete inputs. If that's not correct, please provide the model of the preamp and Zoom so we can see what your routing might be.
If that is correct, one typical method of simultaneously recording a signal and sending it to a PA is to use a mixer with enough outputs -- in your case, a mono or stereo signal for the PA, one for recording the pickup, one for recording the mic. You might also want a DI box between the pickup signal and the mixer if the mixer doesn't specifically have an instrument-level input and/or the signal has to travel on a long cable before getting to the mixer.
For example, if you have a mixer feeding a power amp for a PA, you would use a send on each channel to route each input signal to your Zoom separately from the main output signal to the PA.
Going to an "amp" (assuming you mean a guitar amp) but not a PA, you would use a DI box on the pickup signal and route one output to the Zoom and another as a pass-through to the amp. Then duplicate that setup for the mic signal (assuming it's going to a different amp).
If you need an amp, PA and recording at the same time, a typical acoustic guitar signal chain is guitar -> DI, then DI output 1 -> mixer -> main out to PA and send out to recording; and DI output 2 -> guitar amp. Then a completely separate chain if the guitar is also mic'd: mic -> mixer -> main out to PA and send out to recording.
I've left out some details like if the mixer has its own mic pres, but it sounds like your mic is already going through a preamp.
There are probably more gerry-rigged ways to do this but IMO this is the tried-and-true way.
Good luck!
Comment: just edited my question. See my long respond there :) Thank you sofar


Testing for statistical differences of quantile regression line slopes

Question: If I were to compare the statistical similarity between the slopes of OLS regression lines from two independent samples, I would use a t-test to test if the slopes are equal or not. I'd like to compare the slopes of lines in a similar way obtained via quantile regression, however, I'm not sure if a t-test would be valid as the sample mean is used in the calculation.
Are there any specific methods used for this purpose? I've seen a lot of material on comparing OLS and quantile regression lines, or two OLS lines from independent samples but nothing on comparing two quantile regression lines in this way.
Comment: Have a look at the paper "Small sample performance of quantile regression confidence intervals", G. Tarr 2012, Journal of Statistical Computation and Simulation for an empirical comparison of the state of the art approaches and pointers to asymptotic comparisons.
Comment: @user603: do you want to flesh out your comment a little and turn it into an answer?
Comment: @user603: anything new on this? [It looks like we could really use some answers on testing quantile regression coefficients.](https://stats.stackexchange.com/questions/tagged/quantile-regression+statistical-significance)
Comment: @StephanKolassa: Thanks for the heads up. I will try to find time for it but it could be several weeks out.
Comment: @user603: ping?
Comment: A bit related: given a single sample, how to test a hypothesis that the slopes for several different quantiles are equal: ["Testing equality of quantile regression slopes at different quantiles"](https://stats.stackexchange.com/questions/502740).
Answer: I would approach this similar to how I would approach it in "regular" linear regression (OLS): model the two variables and their interaction, and do inference on the interaction term.
The interpretation of the interaction term is that it is the difference in slopes between the two levels of the group variable, and this is the same whether the approach is OLS or quantile regression. (The exact interpretation changes, since quantile regression models conditional quantiles instead of conditional means, but the idea of a line with a slope is the same.)
$$
\mathbb E[y\vert x_1, x_2] = \beta_0 + \beta_1x_1 + \beta_2x_2 + \beta_3x_1x_2
$$
Instead of fitting this model with OLS, fit it with quantile regression. Then you get a point estimate for $\beta_3$, the difference in slopes between the two groups in $x_1$.
Then you get into how to test for significance or create a confidence interval. I like the idea of testing via confidence interval and examining if $0$ is in the confidence interval. The quantreg package in R has a number of methods for calculating confidence intervals. I would do it with bootstrap. If you truly need a p-value, perhaps a permutation test of $H_0: \beta_3 = 0$ would work for you.
Answer: Couldn't you just run a pooled quantile regression y=betax + deltax*groupDummy? Seems easier to me, and presumably your favorite stats package will give you a confidence interval for delta. But I may be missing something.
Comment: Do you mean something like a hypothesis test of confidence interval on the interaction term?
Comment: Yes, or even simpler actually you look at whether 0 is in the CI for the interaction term.


Where should I start? (piano)

Question: I am a returning player. The most I've dove into piano was learning the Treble Clef, Bass Clef, the notes, keys, and different Key Signatures.
I was only able to play songs in the C & G Major Scales. Although I was able to play every major scale up to 2 octaves with both hands simultaneously.
I never owned a full size piano / keyboard, however a gracious friend was generous enough to gift me his.
-> I would kind of like to start from ground zero if possible.
Any music majors / professors have any advice on where to start? What resources could a minimum wage worker use to start learning the piano again.
Any advice is welcome~ Thank-you very much in advance.
Comment: I think this is essentially the same question as the linked duplicate.  If you have anything more specific to ask, please feel free to post another question!
Comment: The main advice you are likely to get is "find a good teacher".
Comment: @ToddWilcox - you beat me (and many others) to it!
Answer: A adult beginners book would be a good starting place to get going again.
If the cost of lessons is not too much than of course a teacher would help immensely.
Piano lessons online : https://www.hoffmanacademy.com/
coursera.org still offers some great resources for free or paid, probably better for music theory etc.
https://www.coursera.org/courses?languages=en&query=music
As a private instruction teacher I always tell my students not to let things hold them back if they want to learn, so find any of these resources and just get to work.
Comment: Regarding adult beginner books..  I'm using the Alfred Adult Piano Course http://a.co/ac51cm

Where should I start? (piano)


Return response and execute some method (Tomcat)

Question: Is there an option for tomcat to return some response to a request whilst at the same time executing a method in a backend service?
In my use case, I have a query which takes ~60s to execute. I'd like to respond a string to the request to let the user know that it will take some time whilst still executing the query itself in the backend service.
So instead of something like this:
<code>...
response = this.service.getSomeData(request);
return response;
</code>
I'd like to do this:
<code>...
return "Data not found"
this.service.getSomeData(request); // actually stores the result in a s3 bucket
</code>
Answer: You can execute it using another thread:
<code>class MyServiceExecution implements Runnable{
public void run(){
System.out.println("thread is running...");
}

public static void main(String args[]){
MyServiceExecution m1=new MyServiceExecution();
Thread t1 =new Thread(m1);
t1.start();
 }
}
</code>
Be careful when you implement threads... Read some about performance, garbage collection, etc.
Comment: This does not cover nearly enough detail to even begin to be helpful. Background processing of requests at the very least requires mention on Servlet 3.1 async and ideally requires a discussion of other options - that likely fit the OPs use case better. e.g starting the processing in the background and returning HTTP202 and allowing polling for a result.


Proper location for common models and utils, for multi-module maven project using spring-boot

Question: There is a multi-model maven project, with dir layout as following:

  xxx-parent/
      common/
      common-web/
      member-api/
      admin-api/
      market-service/
      ...

As you can see, there is a parent pom, and multiple sub modules, the common or common-xxx modules are the commons parts, which would be dependencies of other non-common modules, for code reuse.

Questions

A. Where would you put the model classes, which are usually POJO, with private fields, and getter/setters.
These models might be reused by multiple non-common sub modules, (e.g both market-service and admin-api).
B. For some api, those model could be reused in the returned json, but only need part of the fields, would you create a separate POJO (usually I call it entity) for that case? And, if so, you would probably put it in the specific sub module (e.g admin-api), not the common projects, right?
C. If this is a spring-boot based project, and it use Redis, and we wrote a redis util, which could be reused for most redis operations.
The problem is if we put the redis util and its maven dependencies into common, then it will connect to redis on application start automatically, but some of the sub modules don't need redis.
The solutions I am considering are:

Create a separate sub module common-redis for the util.
But there would be other utils too (e.g common-jdbc), and there seems to be too much of common modules.
Disable sprint-boot's auto config for a specific feature (e.g redis) via @EnableAutoConfiguration, so that, it won't try to connect to redis on start.
Answer: Your maven multi-module structure:
xxx-parent/
      common/
      common-web/
      member-api/
      admin-api/
      market-service/
      ...

Let me answer the questions individually.

A. Where would you put the model classes, which are usually POJO, with
  private fields, and getter/setters. These models might be reused by
  multiple non-common sub modules, (e.g both market-service and
  admin-api).

Typically, you'll have the model classes under a separate module called xxx-entities or xxx-defs (defs stand for definitions) under your parent project xxx-parent. I'm not sure what the common module contains, but some projects use the word xxx-commons to define the common models(/POJOs/Entities). This is the module that you'd give to other modules as dependencies. Example: If you have a separate persistence layer which would want to operate on the model classes and you'd want to separate that out from your REST client library that you've written for your service.

B. For some api, those model could be reused in the returned json, but
  only need part of the fields, would you create a separate POJO
  (usually I call it entity) for that case? And, if so, you would
  probably put it in the specific sub module (e.g admin-api), not the
  common projects, right?

You can reuse the same model library from my previous point. If you only need part of the fields, set only those fields (while other fields remain null) and return back. Most REST libraries have settings for serialisation which you can set to ignore null values while generating JSON from the object. Example: Ignore null from Json using Jackson.

C. If this is a spring-boot based project, and it use Redis, and we
  wrote a redis util, which could be reused for most redis operations.
  The problem is if we put the redis util and its maven dependencies
  into common, then it will connect to redis on application start
  automatically, but some of the sub modules don't need redis.

Your approach sounds right. You can have a single library something like say data-utils which can have wrappers to operations pertaining to databases (Redis, RDBMS, etc). And you can have init sort of methods which will ensure they don't load automatically unless explicitly called by their clients. Care should be taken to ensure that the configurations such as Redis hostname, port, etc should be loaded from the parent module which encompasses this data-utils client and that this library cannot have such configurations as part of it.
Comment: Good to hear how others would usually do it.
Comment: @EricWang, does that answer your question?
Comment: Honestly, I am very sure, part of the question is still blur, I have upvoted, I might accept it in future, if the blur parts are confirmed.
Comment: @EricWang, Sure.


Acquia Drupal 6 Multisite

Question: I am hosting an Acquia Drupal site on AWS:

I currently have 2 sites both located in the sites folder:

I am fairly new to working with Drupal Multi-sites and would like to know what is the best way to point to each folder's settings.php with this format and ignore the sites/default folder?
And if it would be simpler in another format what would you recommend?
Let me know if you have any questions.
Thanks!
Comment: In you title you have "acquia", is this on an acquia server?  Last time I worked on their server was with a multi-site docroot for Drupal 6, and it was difficult.  Their configuration was not conducive but it eventually worked.
Comment: I'm also wondering why the AWS tag? Is that a by-proxy type thing (because Acquia use AWS) or is your own site hosted on AWS outside Acquia?
Comment: @DeveloperWeeks I added the "Acquia" work because that is what I see when I get the site running.
Comment: @Clive I am currently hosting my site on AWS.
Comment: This is a server configuration thing.  I am not familiar with the AWS specifics, but you need the apache (or whatever engine) configuration to know it is listening for two urls.  The [multisite configuration](https://www.drupal.org/documentation/install/multi-site) guide seems to imply that the Drupal part will just know if the site addresses correctly match the folder names.
Comment: @DeveloperWeeks Can this be done in which ever folder the site is located?
Comment: schnippy has the right answer.  If your apache (or nginx or whatever) is pointing both sites at that html folder then they should find drupal's index.php which will load the correct sites folder to match the name of the site requested.
Answer: Assuming you have your web server (Apache, etc) pointing both sites correctly to your docroot, then Drupal will automatically route requests to the correct folder (and settings file) for each domain.
If you want to have it ignore the default folder and its settings.php file, you can delete the existing "default/" folder and create a symbolic link to one of your other two sites. For example:
<code>ln -s alumni.something.edu default
</code>
and all requests to your docroot that don't match a domain in your multisites folder will be sent to alumni.something.edu (or whatever you prefer).
Comment: I ran this command: ln -s /var/www/html/sites/magazine.something.edu /var/www/html and now I a symbolic magazine.something.edu link in /var/www/html. When I go to EC2IP/magazine.comething.edu I get a: 403 Forbidden You don't have permission to access /magazine.something.edu/ on this server. Apache/2.2.29 (Amazon) Server at EC.2.IP Port 80 My permissions are set at 777 and I changed the following on my httpd.conf file: DirectoryIndex index.html index.html.var index.php index.htm index.sh Also, if I delete the .htaccess files I can see my files in the browser. Any ideas?

Acquia Drupal 6 Multisite


What makes a composer decide between time signatures 3/4, 3/8, or 3/2?

Question: Is there a logic to why, for example, a composer would choose 3/8 or 3/2 over 3/4?
It seems like the n/8 would consume more ink than n/4, or make me look at more stuff at a time (cognitive overload for me). The larger the denominator, the more cramped the score looks.
I see a lot of adagio music with 32nd and even 64th notes, and they are actually slow. Why not write them in n/4. Yes, it'll result in more pages, but I think it will be easier on my eyes?
I also see hollow notes that are supposed to be relatively fast, pieces in n/2 or even n/1. That's tough to read because you'll end up with fast quarter notes that are not beamed in groups of 3, 4, or 6. Why would a composer or engraver want to do this?
If I were to hear something for the first time, I would be shocked to find out that the time signature is something like 12/2 or 4/16.
So tell me, what circumstances would call for 3/8 and not 3/4? 4/2 and not 4/4? Tell me if it's just perspective and that the performer just has to suck it up.
Comment: Each of these time signatures has a different feel
Comment: What exactly does the word "feel" mean in this context? What do each of 3/8, 3/2, and 3/4 convey differently?
Comment: I closed this question as a duplicate of a nearly identical question that was already asked. The examples and details in the answers though weren't on a very good level though so I put a bounty on it. Hopefully though that question's answers and newly placed bounty you'll get an excellent answer.
Answer: Consider you're walking up the stairs.  You could have a lot of small steps or just a handful of huge steps.  As you walk up the steps, somebody is trying to match your movements with the drum in front of them.
So maybe it only takes two huge steps to get to the top of your staircase, or to the end of the phrase or the song.  What if you have four steps, or maybe 16 steps?  It's easier for the drummer to track when to strike a note/beat.
Ideally one is conforming their time signature to the convenience of the performer, and thus there are more natural and less natural fits.
Comment: I have absolutely no idea what this answer is supposed to mean.
Comment: It's a visual for breaking down timing resolutions.  Perhaps it was lost in translation.
Comment: @sova most composers don't view time signatures with the same number of beats this way. I can have each half note, quater note, or eighth note to be any tempo I'd like espically if the time signature remains constant.
Comment: @Dom well, okay.  as a modern-day person who is using DAWs to work on music, i abide by BPM or beats per minute.  This is different from time signatures but amounts to the same thing: the heart beat of the music


What fixed effects should be controlled for in a DDD specification?

Question: I am following work by Borusyak, 2021 which uses a triple-difference design:

In triple-difference designs, the data have three dimensions, such as
regions $i$, demographic groups $g$, and periods $t$. Conventional
static OLS estimation is based on the regression
$Yigt = αig + αit + αgt + τDigt + εigt$

Therefore, what he did is: when observations are defined by $i,g,t$ when, say, $i$ are counties and $g$ are age groups, specify a variable $ig$ identifying the $(i,g)$ pairs as the unit identifier, add appropriate FEs.
did_imputation Y ig t Eig, fe(ig i#t g#t) cluster(i)

the event time Eig term should be specific to the $i,g$ pairs, not to the $i$. For instance, Eig is missing for a never-treated age group in a county where other groups are treated at some point.
I am wondering why this setting is quite different from what Kota Mori explained to me here:
$$
Y_{i,t} = \alpha_i + \beta_t + \gamma D_{i,t} + \delta g_i D_{i,t}
$$
where person and time dummies for DiD purposes are $\alpha_i$ and $\beta_t$.
I am wondering why there are differences between these two models and why Borusyak controlled for up to three types of fixed effects.
Answer: The difference-in-difference-in-differences (DDD) estimator you're referencing is typically used in settings where a law (i.e., treatment) is enacted in some non-uniform (i.e., staggered) manner across jurisdictions (e.g., states). In practice, you may have a mixture of early- and late-adopter states, and another subset of states that never adopt the new law in all time periods under observation. States rarely impose new legislation at the same time, and evaluators typically wish to exploit this variation in treatment timing. We may also suspect this new legislation affects sub-groups differently within those treated states. This is yet another layer of variation we can exploit, assuming we actually observe individuals over time within each state.
Suppose during your evaluation of the effect of anti-corruption laws on wages in the United States you acquire detailed employment records for individuals nested within states. Now suppose you suspect the law has a differential impact by age group or gender. Say, for example, you age stratify workers within each state. For simplicity, a theory suggests the law affects the earnings of younger employees differently than older employees. You dichotomize workers accordingly, with all employees under 35 years of age falling into the younger age category. Well, the new law now varies over three dimensions: age group $a$, state $s$, and year $t$. It's important to be aware that because the laws are introduced at different times, a standard three-way interaction term isn't going to work. We must define the law dummy to account for the staggered adoption periods.
The more general representation of the DDD equation is as follows:
$$
Y_{iast} = \gamma_{st} + \lambda_{at} + \eta_{as} + \delta L_{ast} + u_{iast},
$$
where $Y_{iast}$, which denotes the earnings of individuals in age group $a$ in state $s$ and year $t$, is regressed on a full set of state $\times$ year effects (i.e., $\gamma_{st}$), age $\times$ year effects (i.e., $\lambda_{at}$), age $\times$ state effects (i.e., $\eta_{as}$), and a law dummy (i.e., $L_{ast}$). It's also permissible to include a concatenated version of, say, state-year and then letting software 'dummy out' all the relevant state-by-year effects for you. Software will invariably drop some of the second level terms to break the collinearity, but it shouldn't affect your estimate of $\delta$. I should stress that you must include all of the second level interaction terms or your model may be misspecified.
Note, the principal variable of interest, $L_{ast}$, 'turns on' (i.e., switches from 0 to 1) in those $a$-$s$-$t$ combinations where the law is in place. It is your triple interaction term just defined in a different way. According to your other post, it appears there is no well-defined period delineating pre-/post-treatment, so we must define the law indicator in a way that it captures the staggered onset of treatment across states, which invariably includes the subset of individuals more sensitive to the anti-collusion legislation (i.e., younger employees). Put differently, imagine the law dummy is a column of $0$'s. As you work your way down the rows, assign the observation a value of $1$ if the individual in your sample is under 35 years of age and nested within a treated state and is in a year after the anti-corruption law went into effect.
In the paper you referenced, it appears the author is clustering by region and estimating all second-order interaction terms. You could call these terms "fixed effects" but it's a bit misleading, because it assumes you only need a fixed effect at the individual, unit, and/or time level. But this is not the case. You should also attempt estimation of all state-year, age-year, and age-state effects to preserve the hierarchy.
Comment: You may have two separate columns, say state *and* year, in which case it's permissible to simply multiply the two variables together (e.g., `state#year`). In other circumstances, you may have one variable denoting say, state-year (e.g., "California-2020"), in which case you could let software 'dummy out' the respecting state-year pairs for you (e.g., `i.state_year`). Does that make sense?
Comment: A very great answer, @Thomas, can I ask what does "It's also permissible to include a **concatenated version of, say, state-year** and then **letting software 'dummy out' all the relevant state-by-year effects** for you." mean, I did not get it fully
Comment: it makes sense, thanks :D


How to find the invalid controls in Angular(v2 onwards) reactive form

Question: I have a reactive form in Angular like below:
this.AddCustomerForm = this.formBuilder.group({
    Firstname: ['', Validators.required],
    Lastname: ['', Validators.required],
    Email: ['', Validators.required, Validators.pattern(this.EMAIL_REGEX)],
    Picture: [''],
    Username: ['', Validators.required],
    Password: ['', Validators.required],
    Address: ['', Validators.required],
    Postcode: ['', Validators.required],
    City: ['', Validators.required],
    Country: ['', Validators.required]
});

createCustomer(currentCustomer: Customer)
{
    if (!this.AddCustomerForm.valid)
    {
        //some app logic
    }
}

this.AddCustomerForm.valid returns false, but everything looks good.
I have tried to find with checking the status property in the controls collection. But I wonder if there is a way to find the invalid ones and display to the user?
Comment: Of you just want to display the fields with an error, you can use css to highlight or color the invalid fields. Each invalid field has an "ng-invalid" class appended in its class list
Comment: It's an open issue in Angular. You can give it a thumbs up to potentially raise the priority here:
https://github.com/angular/angular/issues/10530
Comment: When formGroup.errors gives `null` and formGroup.valid gives false, it might be because of html tags like `maxlength` etc. They should not be mixed
Comment: Regarding the open issue 10530 the TLDR; is that they aren't adding such functionality because it would increase the bundle size of all apps. Which I'm afraid is a pretty poor excuse given possible alternatives.
Answer: You can simply iterate over every control and check the status:
    public findInvalidControls() {
        const invalid = [];
        const controls = this.AddCustomerForm.controls;
        for (const name in controls) {
            if (controls[name].invalid) {
                invalid.push(name);
            }
        }
        return invalid;
    }

Comment: @AngularInDepth.com   -  if the one of the controls is a form group, your function will return the invalid form group and not the specific form control which is invalid
Comment: thanks for this but I tried this and even this returns nothing my form is still invalid, this is weird. I mean this code looks good but it doesn't make any sense why form.valid returns false
Comment: it returns nothing, invalid is empty. I checked one by one in debug watch screen, all the controls are valid but this.AddCustomerForm.valid still returns false.
Comment: i had the same issue, that my form was invalid but all fields where valid. I updated the field via `field.setValue('1234', {onlySelf: true})`. As soon as i remove onlySelf:true it was working as expected!
Comment: what does `findInvalidControls()` return you?
Comment: I think I found out. there is an email field and regular expression but somehow the control's status is PENDING and it might be the cause
Comment: @junky, yeah, it can be it :). Pending means that async validation is in progress
Comment: I'm not sure it's a good idea but I changed the if line like this,  if (controls[name].status != "VALID" {
Comment: that's the [internal implementation](https://github.com/angular/angular/blob/master/packages/forms/src/model.ts#L126), no need to do that :)
Comment: In my case, none of the controls were invalid, but I realized that I disabled the form before checking if the form was invalid, so that produced that the form was invalid
Comment: could write this in shorter using ES6 features. `return controls.filter((control) => control.invalid).map((control) => control.name)`
Comment: works perfect. Mine was a bit more advaned as I have formgroups in formgroups
`let invalid = [];
    const controls = this.entryForm.get('entry') as FormGroup;
    for (const name in controls.controls) {
      if (controls.controls[name].invalid) {
        invalid.push(' ' +(+name + 1));
      }
    }`
Comment: In the case that a field does have value that passes all validations but is still logged as invalid, be sure that you're not manually assigning value to formControl.value like so `someControl.value = value` or `someControl.value.push(value)` (common anti-pattern for multi-select fields). Instead, use `someControl.setValue(value)`.
Answer: An invalid Angular control has the CSS class named 'ng-invalid'.
Under DevTools in Chrome, select Console tab.
In console prompt run the following command in order to get the invalid Angular controls that bear the CSS class 'ng-invalid'
document.getElementsByClassName('ng-invalid')

The output should be similar to this:

In this case, the underlined text is for the form control listen-address and the encircled text: .ng-invalid indicates that the control is invalid.

Note: Tested in chrome
Comment: This works really well thank you
Comment: What if only form itself is returned with ng-invalid?
Answer: I just battled this issue: Every form field is valid, but still the form itself is invalid.
Turns out that I had set 'Validator.required' on a FormArray where controls are added/removed dynamically. So even if the FormArray was empty, it was still required and therefore the form was always invalid, even if every visible control was correctly filled.
I didn't find the invalid part of the form, because my 'findInvalidControls' function only checked FormControl's and not FormGroup/FormArray. So I updated it a bit:
/*
   Returns an array of invalid control/group names, or a zero-length array if
   no invalid controls/groups where found
*/
public findInvalidControlsRecursive(formToInvestigate:FormGroup|FormArray):string[] {
    var invalidControls:string[] = [];
    let recursiveFunc = (form:FormGroup|FormArray) => {
      Object.keys(form.controls).forEach(field => {
        const control = form.get(field);
        if (control.invalid) invalidControls.push(field);
        if (control instanceof FormGroup) {
          recursiveFunc(control);
        } else if (control instanceof FormArray) {
          recursiveFunc(control);
        }
      });
    }
    recursiveFunc(formToInvestigate);
    return invalidControls;
  }

Comment: Oh wow. Thanks a lot that saved me a lot of investigation-time!
Comment: I was battling the same issue, but even `findInvalidControlsRecursive` didn't find any invalid control. The problem was that I was replacing a form group like `form.controls.someFormGroup = updatedFormGroup;`. This mostly worked, but introduced suble issues with validation, etc. since the `parent` does not seem to get associated to the `FormGroup`. Using `removeControl`+`addControl` fixed it.
Answer: Now, in angular 9, you can use the markAllAsTouched() method to show the invalid controls validators:
this.AddCustomerForm.markAllAsTouched();

Comment: Going to give this a +1, as it helped me find out what I needed to know --- which is to show validation messages when the user hasn't necessarily touched the inputs.
Answer: There exists an .error property on each the control of the reactive form. If this .error is set to true it indicates that a control is invalid. Thus, looping through the controls and checking this .error field will let us know which fields/ controls are invalid.
The below code will log all the invalid the controls
for (let el in this.ReactiveForm.controls) {
      if (this.ReactiveForm.controls[el].errors) {
        console.log(el)
      }
 }

One can alternatively append the field name to an array or a string and indicate the user which fields are invalid
Answer: Both the forms and all your controls extend the angular class AbstractControl. Each implementation has an accessor to the validation errors.
let errors = this.AddCustomerForm.errors
// errors is an instance of ValidatorErrors

The api docs contains all the references
https://angular.io/api/forms/AbstractControl
Edit
I thought the error accessor worked this way however this link to github shows that there are some other people who thought same as i did
https://github.com/angular/angular/issues/11530
In any case, by using the controls accessor you can iterate over all formControls in your form.
Object.keys(this.AddCustomerForm.controls)
    .forEach( control => {
        //check each control here
        // if the child is a formGroup or a formArray
        // you may cast it and check it's subcontrols too
     })

Comment: this returns null even there are empty controls
Comment: It should return null when there are no errors. Can you post your template?
Comment: Yeah, this won't work, the different validations set on each form controls, those each form controls contain their errors, the form doesn't. You need to iterate the controls like Maximus have given answer.
Comment: I can access errors for each individual contorls like this.form.controls['Email'].errors
Comment: @AJT_82 indeed the Form itself can show errors if a validator has been set for the formGroup ( check the docs about cross field validation, which makes sense to validate on the group and not in the control )
Answer: try this
 findInvalidControls(f: FormGroup) {
    const invalid = [];
    const controls = f.controls;
    for (const name in controls) {
      if (controls[name].invalid) {
        invalid.push(name);
      }
    }
    return invalid;
  }

Answer: So I too have countered this dragon.
And like the brave knight I am, I first gathered my weapons, read the maps and then fought this awful beast.
Note
This is not an acceptable answer for complex forms or structures, but I found it working for the easy ones without to many complexity
The code does the following:

Get the form controls as array
loop over and check if form control is invalid
if invalid the filter will INCLUDE it
if any was invalid, result array will be filled with that controls
if the length of this result is equal then 0, we can state that no controls were invalid and by that the whole form is valid

isFormValid = () :boolean =>
    Object.values(this.form.controls)
        .filter(c => c.invalid).length === 0

// or single lined
isFormValid = () :boolean => Object.values(this.form.controls).filter(c => c.invalid).length === 0

You can use it in the place you want, on submit buttons, onSubmit or on your own sweet spot.
Comment: You have fought well, gallant knight
Answer: I took the liberty to improve AngularInDepth.com-s code, so that it recursively searches for invalid inputs in nested forms also. Wether it be nested by FormArray-s or FormGroup-s. Just input the top level formGroup and it will return all the FormControls that are invalid.
You can possibly skim some of the "instanceof" type checks away, if you would separate the FormControl check and addition to invalid array functionality into a separate function. This would make the function look a lot cleaner, but I needed a global, single function, option to get a flat array of all the invalid formControls and this is the solution!
findInvalidControls( _input: AbstractControl, _invalidControls: AbstractControl[] ): AbstractControl[] {
    if ( ! _invalidControls ) _invalidControls = [];
    if ( _input instanceof FormControl  ) {
        if ( _input.invalid ) _invalidControls.push( _input );
        return _invalidControls;
    }

    if ( ! (_input instanceof FormArray) && ! (_input instanceof FormGroup) ) return _invalidControls;

    const controls = _input.controls;
    for (const name in controls) {
        let control = controls[name];
        switch( control.constructor.name )
        {
            case 'AbstractControl':
            case 'FormControl':
                if (control.invalid) _invalidControls.push( control );
                break;

            case 'FormArray':
                (<FormArray> control ).controls.forEach( _control => _invalidControls = findInvalidControls( _control, _invalidControls ) );
                break;

            case 'FormGroup':
                _invalidControls = findInvalidControls( control, _invalidControls );
                break;
        }
    }

    return _invalidControls;
}

Just for those that need it, so they don't have to code it themselves..
Edit #1
It was requested that it also returns invalid FormArray-s and FormGroups, so if you need that also, use this code
findInvalidControls( _input: AbstractControl, _invalidControls: AbstractControl[] ): AbstractControl[] {
    if ( ! _invalidControls ) _invalidControls = [];
    if ( _input instanceof FormControl  ) {
        if ( _input.invalid ) _invalidControls.push( _input );
        return _invalidControls;
    }

    if ( ! (_input instanceof FormArray) && ! (_input instanceof FormGroup) ) return _invalidControls;

    const controls = _input.controls;
    for (const name in controls) {
        let control = controls[name];
        if (control.invalid) _invalidControls.push( control );
        switch( control.constructor.name )
        {
            case 'FormArray':
                (<FormArray> control ).controls.forEach( _control => _invalidControls = findInvalidControls( _control, _invalidControls ) );
                break;

            case 'FormGroup':
                _invalidControls = findInvalidControls( control, _invalidControls );
                break;
        }
    }

    return _invalidControls;
}

Comment: I tried it, but it doesn't find any invalid FormGroup or FormArray... only invalid FormControl's. I made the same mistake... see my answer.
Comment: I improved my answer, to fit your use case.
Comment: you can use this to literate over the controls:
Object.entries(input.controls).forEach(([, control]) => {
  switch (control.constructor.name) {
  ...
  }
})
Answer: If you are not having much fields in the form, you can simply F12 and hover over the control, you will be able to see the pop-up with field's pristine/touched/valid values-
"#fieldname.form-control.ng-untouched.ng-invalid".
Answer: you can log value of form console.log(this.addCustomerForm.value), it will console all control's value then null or ""(empty) fields indicate invalid controls
Answer: I think you should try using this.form.updateValueAndValidity() or try executing that same method in each of the controls.
Answer: In my case, I had all form controls disabled.
It appears to be an open bug in Angular: https://github.com/angular/angular/issues/39287
Comment: Your case, can you explain if that fixed the problem or that it was the problem itself that you had the form controls disabled. Blame English for being interpretive.. :)
Comment: My issue was that the form appear invalid, but when i tried to find an invalid form control, none of them was. The "fix" was to set some field as not disabled, but I don't know if it is appliable to every use case.
Answer: A cleaner and immutable recursive version of solution to above problem:
P.S: you will need both methods.

Working tested uptill Angular 11

In case compiler complains about flatMap, refer to this(Typescript flatMap, flat, flatten doesn't exist on type any[]), and don't forger to restart ng serve

findInvalidControls(controls = this.defaultFormGroup.controls) {
    const ctrls = Object.values(controls);
    const names = Object.keys(controls);
    return ctrls.map((a,i) => [a, i])
      .filter(a => (a[0] as FormControl).invalid)
      .flatMap(a => {
        if (a[0] instanceof FormArray) {
          return this.findInvalidArrayControls(a[0].controls);
        } else if (a[0] instanceof FormGroup) {
          return this.findInvalidControls(a[0].controls);
        } else {
          return names[a[1] as number];
        }
      });
  }

  findInvalidArrayControls(controls: AbstractControl[]) {
    const ctrls = Object.values(controls);
    const names = Object.keys(controls);
    return ctrls.map((a,i) => [a, i])
      .filter(a => (a[0] as FormControl).invalid)
      .flatMap(a => {
        if (a[0] instanceof FormArray) {
          return this.findInvalidArrayControls(a[0].controls);
        } else if (a[0] instanceof FormGroup) {
          return this.findInvalidControls(a[0].controls);
        }
         else {
          return names[a[1] as number];
        }
     });
  }

Answer: Create the flag
inProcess: boolean= false

   this.AddCustomerForm = this.formBuilder.group({
       Firstname: ['', Validators.required],
       Lastname: ['', Validators.required],
       Username: ['', Validators.required],
       Password: ['', Validators.required],
       Address: ['', Validators.required],
       Postcode: ['', Validators.required],
       City: ['', Validators.required],
       Country: ['', Validators.required]
   });

onSubmit()
{
   if(this.AddCustomerForm.invalid)
   {
     return
   }

  this.inProcess = true

// pass form value to restapi

}

and this inProcess flag you used in HTML form disable button
<button mat-button [disable]="inProcess"> ADD </button>

Once all form values are correct then only ADD button is visible
hope it will help u guys!!!
Answer: For anybody who needs to filter formArray, to only have valid controls here's the solution;
const validControls = this.participantsDetails.controls.filter(control => control.valid);

and for invalid ones of course;
const validControls = this.participantsDetails.controls.filter(control => control.invalid);

Answer: Worked for FormGroup and FormArray. This method returns invalid control names.
private getInvalidControlNames(input: FormGroup | FormArray): string[] {
  let invalidControlNames: string[] = [];
  Object.keys(input.controls).forEach((controlName) => {
  const control = input.get(controlName)!;
  if (control.invalid && control instanceof FormControl) {
    invalidControlNames.push(controlName);
  } else if (
    control.invalid &&
    (control instanceof FormGroup || control instanceof FormArray)
  ) {
    invalidControlNames.push(...this.getInvalidControlNames(control));
  }
  });
  return [...new Set(invalidControlNames)];
}

Comment: I tried this one and it does seem to work. but the form.valid still returns false. might be from some external component embedded into the form
Answer: In my case, I found out that it was because I called markAsPristine() without manually refreshing the form validity... So the form was in an invalid state.
Calling formGroup.updateValueAndValidity() fixed the issue for me.
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answer: If you have an invalid form group, but all controls inside are valid, the controls might have been added to more than one form group.
Answer: Check empty or null form control value in html page
Form controls value: {{formname.value | json}}


É possível criar aplicativos para Android usando HTML5?

Question: Queria muito começar a desenvolver aplicativos para celulares Android, porém tenho um conhecimento quase nulo sobre o SDK do sistema operacional, tenho mais conhecimento sobre HTML, Javascript, CSS.
Eu tenho em mente que criar um app Android deva ser bem diferente da constituição de um web site em html, mas talvez deva existir algum jeito emular um arquivo HTML como se fosse um app ou algum outra forma.
Queria saber se existe algum plugin para isso ao algo que deixe pelo menos mais 'fácil' e amigável escrever os códigos para aplicação.
Comment: React Native é uma boa opção.
Comment: Aconselho fortemente você investir um tempo em conhecer o SDK Android. Excelente plataforma de desenvolvimento. Se desejar mais detalhes pode me contatar: email@example.com
Comment: Cara, se você quer criar apps estáveis e robustos, trabalha com o SDK.
O nativo é **sempre** melhor. http://www.businessinsider.com/the-mobile-web-is-dead-its-all-about-apps-2014-4?fb_action_ids=10152705418059746&fb_action_types=og.recommends&utm_content=bufferee5dc&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer
Comment: Dê uma olhada no http://www.appgyver.com/ - uma variante híbrida do PhoneGap.
Comment: Eu já tentei várias vezes, mas não consigo de jeito nenhum entender a lógica do java e dos códigos do Android. Também tenho dificuldade na hora de assinar os apks =(
Answer: Existem alguns Frameworks que usam exatamente o que você procura: HTML5, CSS3 e JavaScript.

Phonegap
Titanium
Sencha Touch
JQuery Mobile

O banco de dados é o SQLite, usado pelo local storage do HTML5. Com esses Frameworks você consegue gerar Apps para Android, IPhone, Windows Phone e Blackberry, tudo com o mesmo código.
Comment: Na verdade ele usa um WebView como "gatilho" para um arquivo .html por exemplo. Dai a navegação é igual a de um site. Você faz a programação usando HTML5, CSS3 e JavaScript igual a um site notmal.
Comment: Depende do que você esta querendo implementar. Logicamente o app nativo em alguns casos tem uma performance melhor do que o app gerado com algum framework. Mas, em muitos casos, eles atendem bem as necessidades mais comuns. Tais frameworks são utilizados por diversos aplicativos conhecidos e por grandes empresas. Pode usar sem medo.
Comment: Iago, não tem problema em fazer isso. Normalmente nas empresas o desenvolvimento é colaborativo. Eu por exemplo uso o Github para isso. Mas de qualquer forma é interessante que você aprenda a levantar o ambiente em sua máquina para evitar de ficar de um lado para o outro com os arquivos que você mesmo poderia estar alterando e testando. Isso economiza bastante tempo. Segue uns tutoriais que vão te ajudar nesse ponto:   http://www.loiane.com/2012/06/comecando-com-phone-gap-cordova-hello-world-android  e   http://www.mobiltec.com.br/blog/index.php/tutorial-de-phonegap-primeiros-passos
Comment: Eu já tinha ouvido falar sobre o phonegap, não achava que era para isso '-'. Como ele funciona? Seria criar um arquivo html normal, ou ele só oferece uma api que é parecida com o html?
Comment: Mas por acaso você sabe se trava, da algum problema que uma aplicação Android não daria...?
Comment: Eu ia começar a usar o Parse (https://parse.com/) como banco de dados. Vocês acham ele bom?
Comment: ok kkk eu nem sei instalar os programas para começar '-'
Comment: Hugo, por exemplo: se alguém cria-se só o começo do app, tipo instalar o Phonegap e tudo mais, e essa pessoa me passa-se esses arquivos para eu continuar o desenvolvimento do html, javascript, teria algum problema ou é melhor eu começar sozinho?
Pq eu acho que no eclipse já sai o arquivo no formato apk que é semelhante ao zip ae daria para eu ir editando. Ou não é bem assim?
Comment: ok, achei que teria algum problema e tals. Mas da pra criar o arquivo apk dentro do eclipse ou só no Google Play?
Comment: **Obrigado n__n =D**
Comment: Desisto! Isso não é pra mim :p _é muito complicado_


An example of a a convolution of singular distribution and a Gaussian distribution that has a 'simple' pdf

Question: I am looking for a nontrivial example of a singular distribution that when convolved with a Gaussian distribution has a pdf of a 'simple'  form.
I let 'simple' be something that you interpret yourself.
Singular distributions are an import class of distributions that are often 'swept under the carpet.'   I would like to see a nice illustrative example of how to work with such distributions.
One way to do this is to give a characteristic function $\phi(t)$ that when multiplied by $e^{-t^2/2}$ has a simple Fourier inverse.
However, I  don't have a good choice  of the characteristic function $\phi(t)$ that would lead to  a meaningful result.
For example, for the Cantor distribution, the characteristic function is given by
\begin{align}
\phi(t)=e^\frac{it}{2} \prod_{i=1}^\infty \cos \left( \frac{t}{3^k} \right).
\end{align}
However, it and is not easy to work with this characteristic function
\begin{align}
\phi(t) e^{-\frac{t^2}{2}}.
\end{align}
In particular, it is difficult to fuind its Fourier inverse.
Edit:  By singular distributions I mean:  A singular distribution is a probability distribution concentrated on a set of Lebesgue measure zero, where the probability of each point in that set is zero.
Edit 2: Another approach we can take is to look at the convolution directly. That is look at the  $U=X+V$ where $V$ is has a singular distribution and $X$ is Gaussian, in this case, the pdf of $U$ is given by
\begin{align}
f_U(u)=E\left[ \frac{1}{\sqrt{2 \pi}}  e^{-\frac{(u-V)^2}{2}} \right].
\end{align}
I was wondering if we can come up with a sequence of random variables $V_n$ that converges in distribution to some $V$ with a singular distribution, for which we can compute the limit
\begin{align}
\lim_{ n \to \infty}E\left[ \frac{1}{\sqrt{2 \pi}}  e^{-\frac{(u-V_n)^2}{2}} \right].
\end{align}
Comment: @Boby As I understand it, the distribution $\ \mu\ $ you're looking for should have the property that $\ \mu(A)=1\ $ for some set $\ A\ $ of Lebesgue measure zero, and $\ \mu(\{a\})=0\ $ for every $\ a\in A\ $. Is that correct?
Comment: Am I right in saying that the Dirac delta distribution is a singular distribution? If so, it's convolution with a Gaussian, when normalised, would be a Dirac delta distribution.
Comment: Hi, @DanielBeale   I added a definition of what I mean by singular distribution.
Comment: Sorry I misunderstood the "where the probability of each point in that set is zero" part of your definition.  I've only seen this referred to as a "singularly continuous measure" in the past, but I see Wikipedia uses that same definition...
Comment: @fourierwho  No problem. Most likely, it is difficult to find a good example. Here is an interesting paper that I found this topic it might be of interest to you http://www.jstor.org/stable/pdf/2316383.pdf?refreqid=excelsior%3A3c5396efe9671cf6bec744dc82514ff3
Comment: @lonzaleggiera  Yes, that is correct.
Comment: I wrote about something related a while back. Not sure if this is of any use here, but you can check the section "Exact Distribution of Z" in my article "New Perspective on the Central Limit Theorem and Statistical Testing", at https://www.datasciencecentral.com/profiles/blogs/new-perspective-on-central-limit-theorem-and-related-stats-topics
Answer: I may be taking a very liberal interpretation of "simple", but hear me out. Define
\begin{align*}
Y_n = 2\sum_{k=1}^{n}\frac{X_k}{3^k}
\end{align*}
where $X_1, X_2, \cdots \overset{\text{iid}}{\sim} \text{Bernoulli}(\frac{1}{2})$. It's a well-known result that $Y_n$ converges in distribution to a Cantor random variable. Trivially,
\begin{align*}
\mathbb{P}(Y_n = c) = \begin{cases}
\frac{1}{2^n} & c \in S_n \overset{\text{def}}{=}\{2\sum_{k=1}^{n}\frac{x_k}{3^k}: x_1, \cdots, x_n \in \{0, 1\}\} \\
0 & \text{otherwise}
\end{cases}
\end{align*}
For a discrete random variable $D$ taking values $(d_1, \cdots, d_n)$ with probabilities $(p_1, \cdots, p_n)$, the convolution of $D+Z$ for a standard Gaussian $Z$ is a mixture of Gaussians with density
\begin{align*}
f_{D+Z}(t) = \sum_{k=1}^{n}p_k\phi(t - d_k)
\end{align*}
where $\phi(t) = (2\pi)^{-1/2}e^{-t^2/2}$ is the density of a standard Gaussian. So
\begin{align*}
f_{Y_n + Z}(t) = \frac{1}{2^n}\sum_{c \in S_n}\phi(t-c)
\end{align*}
Checking all necessary conditions and taking the limit $n \rightarrow \infty$, we have the convolution of a Cantor and a standard Gaussian. So, this convolution can be interpreted as an (uncountably) infinite mixture of Gaussians. This isn't quite satisfactory because an infinite mixture of Gaussians can pretty much approximate any continuous distribution, and therefore the answer is still "something", but at least this way we can have a constructive approximation.


Rules: after saving new content and after updating existing content both triggered on node create

Question: I define two rules that should send an email once new node has been created. And second rule once user updated his content.
The problem I am having here is that both rules trigger in one time when your create a new node.
I found the same issue has been asked in anther question but not very clear and i am not able to understand enter link description here
But I am not able to understand.
Thanks for the help.
Answer: This is very easy to accomplish, you have a few attributes of the node to compare and check in order to decide which trigger to be true/false.
Method 1:
Put the following PHP code into your both conditions (create & update) by selecting execute PHP code
CREATE
if($node->created == $node->changed) {
   return true;
} else {
   return false;
}

UPDATE
if($node->created != $node->changed) {
    return true;
} else {
   return false;
}

Method 2:
UPDATE:
if($node->is_new) {
     return false;
} else {
     return true;
}

OR
if(!empty($node->nid)) {
    return true;
} else {
    return false;
}

I have not checked the Method 2 but, method 1 is easier and should definitely work out for you.
Don't forget to Accept this answer solve your problem.
Cheers.
Raf.


Clone Colorbox Image Link with JQuery

Question: I need to clone a colorbox image link (the first of all colorbox links on the loaded page) and add it onto a div-tag. how could I do that with javascript/jquery? thnx!
Answer: I found this solution:
        $('#main-stuff').append('

').css("cursor","pointer");
        $('#bgImgOverlay').click(function() {
            $("a[rel='gallery-all']").colorbox({open:true});
         });

Comment: This does not clone a link, so either the question is not clear on what you want or this answer is not correct.

Clone Colorbox Image Link with JQuery


Who can I create a Share Android App to send to my clients or colleagues? Like iOS AdHoc

Question: I'm over an Android App for a client, eventually I want to show him some demo, but he doesn't have the IDE, so I'm thinking to send him an APK ... In iOS it's pretty easy through AdHoc and the Mobile ID...
I just want to send him an APK for him to install in his phone... (Not Android Market YET)
How can I do this on Android?
Thanks!
Comment: send to gmail and click attachment to install it on the android phone
Answer: You could host the apk file on a web server and then share the url with your friends.
When they open the url on there android phones they will be prompted by the browser about whether they want to install the apk.
Answer: Just send him the APK from your bin folder :) He can easily install it if sent to his gMail account, he only has to click on the atachment in his gmail client. Othervise, he will need a file manager.
In both cases he will need to set Authorize unsigned apps on his phone.


What is Connectionist Temporal Classification (CTC)?

Question: I'm looking to do an optical character recognition (OCR) project. After doing some research I've come across an architecture that seems interesting: CNN+RNN+CTC. I'm familiar with convoluted neural networks (CNN), and recurrent neural networks (RNN), but what is Connectionist Temporal Classification (CTC)? I'd like an explanation in layman's terms.
Comment: I recommend you to read the origin paper: Graves, et al. (2006) Connectionist Temporal Classification: Labelling Unsegmented
Sequence Data with Recurrent Neural Networks, *Proceedings of the 23 rd International Conference
on Machine Learning* ([pdf](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75.6306&rep=rep1&type=pdf)).
Answer: You have a dataset containing:

images I1, I2, ...
ground truth texts T1, T2, ... for the images I1, I2, ...

So your dataset could look something like that:

A Neural Network (NN) outputs a score for each possible horizontal position (often called time-step t in the literature) of the image.
This looks something like this for a image with width 2 (t0, t1) and 2 possible characters ("a", "b"):
<code>
  | t0  | t1
--+-----+----
a | 0.1 | 0.6
b | 0.9 | 0.4
</code>
To train such a NN, you must specify for each image where a character of the ground truth text is positioned in the image.
As an example, think of an image containing the text "Hello".
You must now specify where the "H" starts and ends (e.g. "H" starts at the 10th pixel and goes until the 25th pixel).
The same for "e", "l, ...
That sounds boring and is a hard work for large datasets.
Even if you managed to annotate a complete dataset in this way, there is another problem.
The NN outputs the scores for each character at each time-step, see the table I've shown above for a toy example.
We could now take the most likely character per time-step, this is "b" and "a" in the toy example.
Now think of a larger text, e.g. "Hello".
If the writer has a writing style which uses much space in horizontal position, each character would occupy multiple time-steps.
Taking the most probable character per time-step, this could give us a text like "HHHHHHHHeeeellllllllloooo".
How should we transform this text into the correct output? Remove each duplicate character? This yields "Helo", which is not correct.
So, we would need some clever postprocessing.
CTC solves both problems:

you can train the network from pairs (I, T) without having to specify at which position a character occurs using the CTC loss
you don't have to postprocess the output, as a CTC decoder transforms the NN output into the final text

How is this achieved?

introduce a special character (CTC-blank, denoted as "-" in this text) to indicate that no character is seen at a given time-step
modify the ground truth text T to T' by inserting CTC-blanks and by repeating characters in in all possible ways
we know the image, we know the text, but we don't know where the text is positioned. So, let's just try all possible positions of the text "Hi----", "-Hi---", "--Hi--", ...
we also don't know how much space each character occupies in the image. So let's also try all possible alignments by allowing characters to repeat like "HHi----", "HHHi---", "HHHHi--", ...
do you see a problem here? Of course, if we allow a character to repeat multiple times, how do we handle real duplicate characters like the "l" in "Hello"? Well, just always insert a blank in between in these situations, that is e.g. "Hel-lo" or "Heeellll-------llo"
calculate score for each possible T' (that is for each transformation and each combination of these), sum over all scores which yields the loss for the pair (I, T)
decoding is easy: pick character with highest score for each time step, e.g. "HHHHHH-eeeellll-lll--oo---", throw away duplicate characters "H-el-l-o", throw away blanks "Hello", and we are done.

To illustrate this, have a look at the following image. It is in the context of speech recognition, however, text recognition is just the same. Decoding yields the same text for both speakers, even though alignment and position of the character differs.

Further reading:

an intuitive introduction: https://medium.com/@harald_scheidl/intuitively-understanding-connectionist-temporal-classification-3797e43a86c (mirror)
a more in-depth introduction: https://distill.pub/2017/ctc (mirror)
Python implementation which you can use to "play" around with CTC decoders to get a better understanding of how it works: https://github.com/githubharald/CTCDecoder
and, of course, the paper Graves, Alex, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. "Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks." In Proceedings of the 23rd international conference on Machine learning, pp. 369-376. ACM, 2006.
Comment: thanks for the feedback. I've added an illustration about the dataset and improved the description. Further, the term time-step simply means "horizontal position" or x-coordinate in the context of text recognition. The term most likely originates from speech recognition (a similar task), where time-step refers to the position in time in the audio signal.


Problema ao deixar link ativo - Rotas Angular 2

Question: Estou com problemas ao adicionar a classe no meu link da página ativa.
Funciona em todas as páginas, porém o da index fica sempre ativo. Por exemplo: se eu estiver na página de aplicação fica ativo os dois links.
Menu:

Página inicial

Aplicações

Route:
{ path: '', component: HomeComponent },
{ path: 'aplicacao', component: AplicacaoComponent },

Alguém já teve problema parecido?
Comment: Tente aplicaro seu routerLink com os colchetes de atributo: Bob. Todos os exemplos na documentação estão usando: https://angular.io/docs/ts/latest/api/router/index/RouterLinkActive-directive.html
Comment: Não funciona. Já havia testado. :/
Comment: Você definiu uma  no seu head?  Olhando no Tour of Heroes percebi que eles mudaram algumas coisas do RC5 para o lançamento, tente ajustar seu router de acordo com o exemplo deles: https://angular.io/docs/ts/latest/guide/router.html#!#base-href
Comment: lfarroco, defini sim. Estou fuçando aqui enquanto isso.
Answer: Por padrão, o Angular 2 coloca como active, qualquer url que ao menos tenha a url configurada no routerLinkActive. Ou seja, a URL "/" existe no "/aplicacao". Assim sendo, o Angular 2 coloca active tanto no primeiro elemento, quanto no segundo.
Para resolver este problema:
Nos docs da diretiva routerLinkActive existe uma outra diretiva onde você passa as opções do mesmo, essa diretiva se chama [routerLinkActiveOptions], na qual você pode passar como valor, um objeto com o parâmetro 'exact' (booleano). Configurando este parâmetro como true, fará com que o routerLinkActive só adicione a classe que você passou no "routerLinkActive" nos links exatos. Assim sendo, caso sua URL seja exatamente "/", ou exatamente "/aplicacao", será adicionada o active respectivamente.
Exemplo na prática

Página inicial

Aplicações

É da sua escolha usar a diretiva "[routerLinkActiveOptions]" no "/aplicacao". Você faria isso se você deseja criar outro sublink, por exemplo: Com a URL "/aplicacao/app1".
Porquê o Angular 2 faz isso como padrão?
Por exemplo, essa funcionalidade é muito útil quando você deseja criar um "Nested menu" (Menu dropdown com múltiplos níveis), com esse recurso você consegue aplicar o CSS em todos os actives de forma prática, em todos os níveis de menus ativados.
Docs:
https://angular.io/docs/ts/latest/api/router/index/RouterLinkActive-directive.html
Answer: Fala Brother!
cara, passei exatamente por este problema quando comecei a programar com Angular.
Pode se dizer que quando você coloca o Path "/", em qualquer rota que seja adicionada ele ainda vai reconhecer a "/", por isso ele deixa ela Ativa.
Cheguei na solução da seguinte forma:
Vamos supor que sua página inicial seja a HOME. No seu App-routing-module.ts faça desta forma:
{ path: 'home', component: HomeComponent},
{ path: '', redirectTo: 'home', pathMatch: 'full'}

e no seu Menu, direcione para "/home" ao invés de "/":

Desta forma, qualquer outro trecho de seu código você pode definir o routerLink para "/" que automaticamente vai redirecionar para "/home", assim não afeta os demais links de seu Menu. O unico detalhe é que sempre vai estar com o /home, "www.site.com/home" e não "www.site.com", tem gente que tem toc com isso, mas é uma solução bem efetiva para o problema do Menu.


How many ways can N elements be partitioned into subsets of size K?

Question: Let's have $2$ numbers, $N$ and $K$, where $K$ divides $N$. The number of $K$-combinations from a given set $S$ of $N$ elements is a well-known formula. Let's concatenate $N/K$ groups (resulting in $N$ elements) such that the resulting set is $N$. How many possibilities are there, i.e. what's the formula?
For instance:
$N=4$, and $K=2$, gives $\binom{4}{2} = 6$ $\{1,2\}$, $\{1,3\}$, $\{1,4\}$, $\{2,3\}$, $\{2,4\}$, $\{3,4\}$
The $3$ possibilities are:
$\{1,2\}$, $\{3,4\}$
$\{1,3\}$, $\{2,4\}$
$\{1,4\}$, $\{2,3\}$
I generated these combinations and I think the number goes like this:
$$
\begin{align}
(4, 2)&: 3 \\
(6, 3)&: 10 \\
(6, 2)&: 15 \\
(10, 5)&: 126 \\
(9, 3)&: 280 \\
(10, 2)&: 945 \\
(14, 7)&: 1716 \\
(12, 4)&: 5775 \\
(15, 5)&: 126126 \\
(15, 3)&: 1401400
\end{align}
$$
Apparently, the result always divides with $(N-1)$.
Comment: The order of the groups doesn't matter, i.e. {3,4},{1,2} = {1,2},{3,4}, {2,4},{1,3} = {1,3},{2,4}, etc.
Comment: In what context did you encounter this problem? Did you try anything / do you have any thoughts on a solution?
Comment: The context for those interested: [Stack Overflow](http://stackoverflow.com/questions/21105154/n-choose-k-k-n-k-2n-etc-recursion-within-recursion/21148261#21148261)
Comment: Cool, I was just wondering myself about the problem of generating these partitions; have posted an answer there on Stack Overflow.
Answer: You want to count the number of set partitions of a set of $n$ elments, into $n/k$ parts each of size $k$.
(It is assumed that $k$ divides $n$.)
Method 1.
We can generate such a partition by writing down the $n$ elements in a sequence, and then declaring that the first $k$ elements are the first part, the next $k$ elements are the second part, and so on. There are $n!$ ways of writing $n$ elements in a sequence, but each partition is generated multiple times: for each of the $n/k$ parts, there are $k!$ orderings of the $k$ elements in that part that would lead to the same partition, as you don't care about the order within each part. Further, there are $(n/k)!$ orderings of the parts themselves, for the same partition.
The number of partitions is therefore:
$$\frac{n!}{(k!)^{n/k} (n/k)!}$$
Method 2.
You can choose the elements of the first part in $\binom{n}{k}$ ways, then choose the elements of the second part as $k$ out of the remaining $n-k$ in $\binom{n-k}{k}$ ways, and so on. But as different orderings of the $(n/k)$ parts don't change the partition, the number of partitions is
$$\frac{\binom{n\vphantom{k}}{k}\binom{n-k}{k}\cdots\binom{k}{k}}{(n/k)!} = \frac{n!}{(k!)^{n/k}(n/k)!}$$
as before.
You can verify that this accords with all your cases. For instance, for $n=15$ and $k=5$, you get $\frac{15!}{5!^3 3!} = 126126$. These numbers are tabulated in OEIS A060540, and no simpler formula is listed.
Comment: It also validates for `(15, 3) = 1401400`, just added. I'll have more time the next days.
Comment: The prove seems correct and the formula is valid for the given numbers, but I still wait for a few days before accepting the answer.
Answer: I might be wrongly interpreting your question (forgive me if this is the case). But based on certain phrases (i.e., "number of $k$-combinations of a given set" and "The order of the groups doesn't matter [from comments]) in addition to the numbers you put forth in your question, it seems to me that you are looking for the formula for:
The number of ways to choose $k$ items at a time from a set of $n$ items.
It turns out that this idea of combinatorics has very deep and interesting applications to nearly every branch of mathematics. The terminology for the number you seek is called a Binomial Coefficient and the notation for the statement above is $$ \binom{n}{k} = \text{number of ways to choose $k$ items from a set of $n$ items}$$ That is why the notation $\binom{n}{k}$ is also pronounced "$n$-choose-$k$". There are other notations including $C(n,k$), $C_k^n$, $_nC_k$, $C^k_n$, but I will continue to use the notation $\binom{n}{k}$ in this answer. For those of you wondering how to type it in a math-environment like TeX,LaTeX,etc. (because eventually I'm sure you will encounter the occasion to use the notation in an answer - that's how prolific binomial coefficients are), you want to use $\$\text{\binom{n}{k}}$\$.
There are many different formulas for binomial coefficients and it is a shame that I can't list them all. However, before we jump into some formulas, it is worth our time to introduce some useful notation is that presented by the Factorial. Incidentally, the number of ways to arrange (order matters) $k$ objects from a set of $n$ objects is related to Permutations which uses the factorial extensively.
Define $n! = \prod\limits_{k=1}^n k = 1\cdot2\cdot3\cdots(n-1)\cdot n$. The convention for "vacuous products" lets us define $0!=1$ but this can also be verified using the definition of the Gamma Function where the notion of a factorial can be extended to all real numbers (except negative integers). Since $n! = n\cdot(n-1)\cdots2\cdot1$, we can condense this into $n!=n(n-1)!$ and immediately see a recursive relation defined by the factorial. Some values that are worth memorizing include: $$0!=1 (\text{by defintion})\\ 1!=1 \\ 2!=2=2*1 \\ 3!=6=3*2*1 \\ 4!=24=4*3*2*1 \\ 5!=120=5*4*3*2*1$$.
Back to combinations, one of the most fundamental formulas for a binomial coefficient is:
$$ \binom{n}{k} = \frac{n!}{k!(n-k)!} = \frac{n(n-1)\cdots(n-k+1)(n-k)!}{k!(n-k)!} = \frac{n(n-1)\cdots(n-k+1)}{k(k-1)\cdots(1)} \\ \binom{n}{k} = \frac{n^{\underline{k}}}{k!} = \prod\limits_{j=1}^{k} \frac{n+1-j}{j} \quad \text{integer $k \geq 0$}$$
In particular, it is straightforward to verify that
$$\binom{n}{0} = \binom{n}{n} = 1 \qquad \binom{n}{1} = n \qquad \binom{n}{2} = \frac{n(n-1)}{2}$$
Also, the following identity is one reason why high symmetry exists in Pascal's Triangle:
$$ \binom{n}{k} = \binom{n}{n-k}$$

So now you have your formula(s) for $\binom{n}{k}$ and a little crash course on factorials. I have computed the same numbers below that you input originally and computed there corresponding binomial coffecients.
$$\binom{4}{2} = 6, \quad \binom{6}{3} = 20, \quad \binom{6}{2} = 15, \\
\binom{10}{5} = 252, \quad \binom{9}{3} = 84, \quad \binom{10}{2} = 45, \\
\binom{14}{7} = 3432, \quad \binom{12}{4} = 495, \quad \binom{15}{5} = 3003
$$
Some of your numbers are either correct or halved (if combinations are what you're after). Given the definitions above, it wouldn't be hard to write a new program with inputs $n$ and $k$ to output $\binom{n}{k}$.
Here's an answer for your bounty message:
"...I'm not a mathematician. Are they on this site?"
Yes. Yes we are.
Comment: I don't have the time right now, but the formula from the answer of ShreevatsaR works for the next 3 tests: (15,5), (15,3), (10,2).
Comment: Thank you for the generous answer, but I'm not looking for the k-combinations in n, but for the  sequences of such combinations that cover the entire set (given that k divides n).
For exemple, for (6,3):
{1 2 3} {4 5 6},
{1 2 4} {3 5 6},
{1 2 5} {3 4 6},
{1 2 6} {3 4 5},
{1 3 4} {2 5 6},
{1 3 5} {2 4 6},
{1 3 6} {2 4 5},
{1 4 5} {2 3 6},
{1 4 6} {2 3 5},
{1 5 6} {2 3 4}

(10 solutions)


Table v.s., node

Question: I am migrating from Drupal 6 to Drupal 7 and have the opportunity to restructure data.
Assume storage of contacts: A node was created of type contact, but information was stored, such as birthday, in other fields.  I didn't design it and don't know why another field wasn't added to the node via CCK.
Nonetheless, is there any reason from a performance standpoint to create a table called "contact" and store all relent information there (e.g., is there performance overhead in using the "node" structure v.s., a simple table - and yes, I know a node is a table).
The database engine in use is MySQL.
Answer: Yes - there is definitely a performance overhead to dealing with your content in Drupal nodes / cck vs using a simple table, assuming you are keeping all of your contacts data in that one table. The performance overhead comes from the abstraction built into the CCK system that requires complex SQL queries to construct your data record from multiple tables (ex. fields, input formats, other modules).
For the most part, you probably want to use CCK because it makes it easier to manipulate this data and add new fields and it exposes your data to dozens of other modules. If this performance hit is an issue you can counteract it by using caching at either the mysql, php, apache, or drupal layer. Otherwise, you can keep your data in a simple (non-drupal) table and expose it to views and other modules by using the Data module.
And Last but Not least, you can run your own storage strategy for a node it the standard fields approach is not enough.
Comment: Well, CCK isn't an option in Drupal 7 because it is, for the most part, built into core (although, there is dev version available).
There must be some benefit that outweighs the multiple fetch operations required to get the contents of a node. Almost every scrap of useful data from a node is stored in a separate table.
For one, it is fairly obvious that there are "build in" structures that support node management that otherwise would have to be re-invented.

Table v.s., node


How do I make a mailto: form in my HTML file?

Question: I am making a website and am using Notepad++. How do I make a mailto form in my HTML document?
Thanks, Andrew
Comment: Stack Overflow is a very poor replacement for Google.
Answer: You use the mailto function thats built in. To use it in a form you do;
  <form name="your name" class="your class" action="mailto:email@example.com">
    </form>

you then need to add the address you want the form to be emailed to.
Answer: <!DOCTYPE html>

Send e-mail to email@example.com:

<form action="mailto:firstname.lastname@example.com" method="post" enctype="text/plain">
Name:

<input type="text" name="name">

E-mail:

<input type="text" name="mail">

Comment:

<input type="text" name="comment" size="50">

<input type="submit" value="Send">
<input type="reset" value="Reset">
</form>


Refatoração de uma classe java

Question: Tenho uma classe Java que contem 1756 linhas(obviamente não é tudo código, tem linhas em branco, muitos comentários e algum código comentado para o caso de ser necessário no futuro)
Estou a implementar a estrutura MVC e esta classe pertence ao controlo.
Esta classe é responsável por:

gerir clientes( comunicação com os cliente, guardar quem está conectado, enviar informação para eles, saber quem saiu etc)
buscar dados a um webservice e a uma base de dados(sincronizadamente)
Processar esses dados e enviar essa informação para os clientes
Guardar informação dos dados importantes na base de dados do sistema

Estrutura da classe ServidorService:
//variaveis
 + 8 variaveis
 + de 12 HashMap //quase todos os dados estão relacionados key->value ex: clientex->outputx

//classes

Problema
A classe está a ficar muito grande mesmo para mim, mas pensando no futuro de alguém precisar de alterar alguma função acho que vai ser difícil para ela dar a manutenção no código.
Qual a melhor maneira de a refatorar?
Principais duvidas:

Crio outra classe tipo ServidorService_2 e tento a dividir?
Já tentei criar outra classes para apoiar a principal(ServidorService) mas estava com problemas no acesso as variáveis, qual a melhor maneira de fazer para a outra classe ter acesso as variáveis?  através de gets?
Comment: No momento que você listou 4 responsabilidades desta classe, você mesmo identificou no mínimo 4 novas classes. Para você gostar do seu código, cada classe deve ter uma única responsabilidade. Não faça `ServidorService_2` e  `ServidorService_3` pois isso não vai melhorar em nada o seu código (talvez piore). Para ter acesso às variáveis , sim, é através de `get`s. Mas você não deve publicar "variáveis" e sim "atributos" da sua classe (a técnica é a mesma mas o conceito é diferente e ajuda a perceber se o código está ok). Não vou colocar isso como resposta porque acho que é off-topic para o SO.
Comment: `(...)e algum código comentado para o caso de ser necessário no futuro(...)` Isso é mais tóxico do que urânio. Livre-se desse código. Se algum dia esse código se tornar necessário, vocês podem recuperá-lo a partir do repositório. Se isso acontecer, jogue na loteria também, porque uma vida de experiência com isso diz que código comentado voltar a ser útil é algo que ocorre uma vez a cada lua azul. E se não tiver controle de versão, parem o projeto até começarem a usar um.
Comment: Independente do que fizer, **não crie ServidorService_2**!! Acredito que boa parte desse código deveria estar no Controller, outra parte do DAO com o BD local, outra no DAO para webservice (independente do nome que preferir dar para essa camada)
Comment: obrigado @Math quanto ao ServidorService_2 foi só um exemplo estúpido mas não sabia mesmo que outro nome poderia dar agora com a ajuda do @caffé e com a resposta do @Cold já tenho uma ideia de como fazer... Para mim o "Servidor_Service" é o meu Controller e nele sempre que preciso de algum dado, seja ele do webservice ou BD só tenho de chamar o método que está em outras classes: Ex: `new ProdutoDao().guardar(produto);` ou `listProdutos=getProdutosWebService();` é a isto que se está a referir certo?
Comment: @jsantos1991 São conceitos de Orientação a Objetos. Este link que achei agora parece bom: https://www.caelum.com.br/apostila-java-orientacao-objetos/ Esta seção em especial deve dar uma noção da diferença conceitual entre publicar uma variável ou atributo (quem garante as regas de negócio, como validação dos valores setados?): http://www.caelum.com.br/apostila-java-orientacao-objetos/modificadores-de-acesso-e-atributos-de-classe/#6-1-controlando-o-acesso
Comment: @Renan obrigado pelas duas dicas, a do código comentado que vou remover e a do controle de versão, que na verdade não estou a usar(tenho feito a mão), mas este projeto está na fase final logo num próximo sem duvida que vou usar.
Comment: @jsantos1991 que bom que já tem uma luz! Mas em opinião minha, eu faria a parte de acesso ao webservices fora do controller, em um DAO específico, eu inclusive tenho um nome para essa camada, mas não sei se é consenso, eu chamo de Resource. Talvez eu esteja falando besteira, mas é assim que eu faço e eu acho que atende aos princípios da OO.
Comment: @Caffé Obrigado só uma duvida, quando se refere a "variáveis" e sim "atributos" quer dizer que é errado dizer "variáveis" certo? porque o nome correto é "atributos" só para esclarecer o que disseste.
Comment: @Math sim como disseste e muito bem só tenho mesmo uma luz, logo nunca tenho bem a certeza se estou a fazer direito ou não, mas quanto a essa parte acho que está certo e felizmente, neste caso como não tem interface gráfica porque acho que é quando consigo destruir todo o mvc [mas esta parte já é [OFF-TOPIC]]
Comment: @jsantos1991 Não se trata do nome certo, o importante é entendermos que uma classe publica os dados que ela quer expôr, suas características e estado, o que pode ser um pouco mais profundo do que simplesmente aumentar a visibilidade de uma variável. Esse assunto é extenso, rende discussões úteis, mas inadequadas para este espaço.
Comment: @Caffé deu para esclarecer um pouco mais, mas ainda não consegui compreender como gostava, se poderes enviar um link|palavras chaves que fale sobre isso [EN ou PT] ficava agradecido.
Comment: @jsantos1991 não existe certo ou errado, existe apropriado ou inapropriado. O código deve ser apropriado para as suas necessidades, é claro que muitas vezes é difícil enxergar a médio ou longo prazo o que é o mais apropriado para você. Se seu software tiver vida longa provavelmente terá que refatorá-lo mais algumas vezes, e com experiência sofrerá menos nos projetos seguintes. Pra começar, tente nunca fugir dos princípios da OO mais básicos, como o da responsabilidade única (alta coesão) e baixo acoplamento.
Comment: @Math obrigado, no inicio é mesmo complicado saber o que é apropriado ou nao...
Comment: @Caffé obrigado pelos Links...
Answer: Pelas responsabilidades que você citou eu sugeria o seguinte (os nomes das classes são apenas sugestões, podes por ao teu gosto):
1 - Criar classes com a implementação de métodos específicos de acordo com as responsabilidades
GestaoClientes.java : Com a implementação dos métodos para tratamento da gestão dos clientes (Responsabilidade 1 e 3);
AcessoBD.java : Com a implementação de todos os métodos de acesso e interação com o Banco de Dados (Responsabilidade 1b e 4);
AcessoWebServices.java : Com a implementação de todos os métodos que interagem com os WebServices(Responsabilidade 1a).
Nota: Responsabilidade 1a é "buscar dados a um webservice" e a responsabilidade 1b é "a uma base de dados".
2 - Manter os métodos relacionados e específicos do servidor na classe ServidorService e as variáveis (aquelas que devem ser visíveis ou modificadas por todos os métodos).
Se todas as variáveis permanecerem na classe ServidorService então acredito que os métodos nas classes que sugeri podem ser estáticos. E vais apenas invocando os métodos quando precisares.
Notei que tens também uma classe interna e ainda não percebi bem a ideia dela aí, porém não sei o que sugerir ainda quanto a ela.
Desse jeito acredito que já teras diminuido em muito as quantidades de linhas e facilitar a manutenção do código, pois o outro técnico saberia onde olhar de forma específica de acordo com o objectivo.
Answer: A estrutura MVC indica você usar Model onde você irá criar classe referente a suas entidades dentro dela temos os atributos da classe e os get's e set's mais o toString() somente isso, se você estiver usando banco de dados você pode criar classes chamadas DAO(Data Access Object) cada entidade tem seu DAO, o Controller é para a regra de negócio ou seja cada entidade tem seu Controler, e a View é para a exibição das informações.
Você vai separar tudo em pacote exemplo:
br.nomeProjeto.model
br.nomeProjeto.controler
br.nomeProjeto.view
br.nomeProjeto.dao

Boas praticas dizem hoje que um codigo muito comentado não é bem elaborado se necessário crie interfaces e implementes seus metodos.
Espero ter ajudado.
Comment: Espero não te ter confundido, mas a tua resposta nao responde a minha pergunta, a minha duvida NÃO é quanto a estrutura do MVC, eu falei em MVC e só para dizer que esta classe pertence ao controler, a minha duvida é quanto a REFATORAÇÃO...
Comment: @jsantos1991 como vc disse que essa classe já está no Controller, sugiro você seguir o princípio da responsabilidade única e criar um Controller para cada entidade, ou para cada entidade relevante, ou seja, algumas podem ficar de fora se o acesso a elas pode ser feita por uma dependencia de uma outra classe. Além do mais, alguns desses métodos não pertencem ao Controller, crie uma camada apropriada para eles.
Answer: Acho que seria interessante você abstrair métodos que não estejam diretamente ligados a função de sua classe, tais como métodos utilitários, mas respondendo sua pergunta:

Se seus métodos estão ligados/relacionados ao que sua classe propõe não vejo porquê criar uma nova classe, claro cada um tem uma visão diferente do assunto;
É recomendado sempre acessar atributos de outras classes utilizando getters e setters, baixo acoplamento, e alta coesão.

Refatoração de uma classe java


Library or tool to download multiple files in parallel

Question: I'm looking for a python library or a command line tool for downloading multiple files in parallel. My current solution is to download the files sequentially which is slow. I know you can easily write a half-assed threaded solution in python, but I always run into annoying problem when using threading. It is for polling a large number of xml feeds from websites.
My requirements for the solution are:

Should be interruptable. Ctrl+C should immediately terminate all downloads.
There should be no leftover processes that you have to kill manually using kill, even if the main program crashes or an exception is thrown.
It should work on Linux and Windows too.
It should retry downloads, be resilient against network errors and should timeout properly.
It should be smart about not hammering the same server with 100+ simultaneous downloads, but queue them in a sane way.
It should handle important http status codes like 301, 302 and 304. That means that for each file, it should take the Last-Modified value as input and only download if it has changed since last time.
Preferably it should have a progress bar or it should be easy to write a progress bar for it to monitor the download progress of all files.
Preferably it should take advantage of http keep-alive to maximize the transfer speed.

Please don't suggest how I may go about implementing the above requirements. I'm looking for a ready-made, battle-tested solution.
I guess I should describe what I want it for too... I have about 300 different data feeds as xml formatted files served from 50 data providers. Each file is between 100kb and 5mb in size. I need to poll them frequently (as in once every few minutes) to determine if any of them has new data I need to process. So it is important that the downloader uses http caching to minimize the amount of data to fetch. It also uses gzip compression obviously.
Then the big problem is how to use the bandwidth in an as efficient manner as possible without overstepping any boundaries. For example, one data provider may consider it abuse if you open 20 simultaneous connections to their data feeds. Instead it may be better to use one or two connections that are reused for multiple files. Or your own connection may be limited in strange ways.. My isp limits the number of dns lookups you can do so some kind of dns caching would be nice.
Comment: If you want a ready-made, battle-tested command-line tool ... why do you care whether it is written in Python?
Comment: This is what threading is for.
Comment: And what problem specifically are you facing?
Answer: You can try pycurl, though the interface is not easy at first, but once you look at examples, its not hard to understand. I have used it to fetch 1000s of web pages in parallel on meagre linux box.

You don't have to deal with threads, so it terminates gracefully, and there are no processes left behind
It provides options for timeout, and http status handling.
It works on both linux and windows.

The only problem is that it provides a basic infrastructure (basically just a python layer above the excellent curl library). You will have to write few lines to achieve the features as you want.
Answer: There are lots of options but it will be hard to find one which fits all your needs.
In your case, try this approach:

Create a queue.
Put URLs to download into this queue (or "config objects" which contain the URL and other data like the user name, the destination file, etc).
Create a pool of threads
Each thread should try to fetch a URL (or a config object) from the queue and process it.

Use another thread to collect the results (i.e. another queue). When the number of result objects == number of puts in the first queue, then you're finished.
Make sure that all communication goes via the queue or the "config object". Avoid accessing data structures which are shared between threads. This should save you 99% of the problems.
Comment: GaretJax:  If you look at the edit history, the answer above was made ten minutes before that sentence was added by the asker.
Comment: *Please don't suggest how I may go about implementing the above requirements. I'm looking for a ready-made, battle-tested solution.* Seems you're doing exactly that.
Comment: @Peter O.: You're right, sorry about that.
Answer: I don't think such a complete library exists, so you'll probably have to write your own. I suggest taking a look at gevent for this task. They even provide a concurrent_download.py example script. Then you can use urllib2 for most of the other requirements, such as handling HTTP status codes, and displaying download progress.
Comment: gevent works on windows and even has windows builds: http://code.google.com/p/gevent/downloads/list
Comment: I'm afraid that gevent is not available on windows.
Answer: I would suggest Twisted, although it is not a ready made solution, but provides the main building blocks to get every feature you listed in an easy way and it does not use threads.
If you are interested, take a look at the following links:

http://twistedmatrix.com/documents/current/api/twisted.web.client.html#getPage
http://twistedmatrix.com/documents/current/api/twisted.web.client.html#downloadPage

As per your requirements:

Supported out of the box
Supported out of the box
Supported out of the box
Timeout supported out of the box, other error handling done through deferreds
Achieved easily using cooperators (example 7)
Supported out of the box
Not supported, solutions exists (and they are not that hard to implement)
Not supported, it can be implemented (but it will be relatively hard)
Answer: Nowadays there are excellent Python libs you might want to use - urllib3 and requests
Answer: Try using aria2 through simple python subprocess module.
It provide all requirements from your list, except 7, out of the box, and 7 is easy to write.
aria2c has a nice xml-rpc or json-rpc interface to interact with it from your scripts.
Answer: Does urlgrabber fit your requirements?
http://urlgrabber.baseurl.org/
If it doesn't, you could consider volunteering to help finish it.  Contact the authors, Michael Stenner and Ryan Tomayko.
Update: Googling for "parallel wget" yields these, among others:
http://puf.sourceforge.net/
http://www.commandlinefu.com/commands/view/3269/parallel-file-downloading-with-wget
It seems like you have a number of options to choose from.
Comment: Thanks but those links fail on 4, 5, 8 and especially 6. The issue for me is not to throw up a number of processes to do downloads, but to handle http cacheing and have some "global control" system so that the downloads are done in an as efficient was as possible.
Comment: If I were writing this and all these requirements were non-negotiable (as you seem to suggest), I would almost certainly write my own downloader based on Twisted.
Answer: I used the standard libs for that, urllib.urlretrieve to be precise. downloaded podcasts this way, via a simple thread pool, each using its own retrieve. I did about 10 simultanous connections, more should not be a problem. Continue a interrupted download, maybe not. Ctrl-C could be handled, I guess. Worked on Windows, installed a handler for progress bars. All in all 2 screens of code, 2 screens for generating the URLs to retrieve.
Answer: This seems pretty flexible:
http://keramida.wordpress.com/2010/01/19/parallel-downloads-with-python-and-gnu-wget/
Answer: Threading isn't "half-assed" unless you're a bad programmer. The best general approach to this problem is the producer / consumer model. You have one dedicated URL producer, and N dedicated download threads (or even processes if you use the multiprocessing model).
As for all of your requirements, ALL of them CAN be done with the normal python threaded model (yes, even catching Ctrl+C -- I've done it).
Comment: Apparently multiprocessing is better for concurrency in Python due to the global interpreter lock, but assuming the GIL timing is quick enough, threading would probably work fine for this sort of thing, what with the latency you'll generally get and how each thread will be blocking for I/O access anyway as they get more data in. I'm no expert, though, so multiple processes may still be better for this situation in Python.
Comment: Yeah, since everything is IO bound anyways the GIL won't have a noticeable impact.