How to mask raster image (DEM) on QGIS Grass

Question: I am doing research at graduate school. Using QGIS GRASS, I made a raster image (DEM) by spline interpolation from point data. Next, I decided to limit the range to within the vector (within the point data). When I masked it in the raster menu, it was not restricted in the vector, and the resulting raster image became a yellow image (with no value). What is the solution for this?
Comment: I don't understand what you mean by "..limit the range to within the vector (within the point data)"  Please edit your post to include a screen shot depicting what you would like to accomplish.
Answer: Your question is not quite clear.
If you want to mask out values of a DEM that are above or below the minimum and maximum of a vector, using GRASS here's what you can do:
Assuming you have a vector of points "point_elevations" with column "elev", and your DEM raster is called "dem". You can find the minimum and maximum with:
v.univar point_elevations column=elev

This will output all the univariate stats for the elev "column" including minimum and maximum. Let's call them "min_value" and "max_value". Now use these value in a mapcalc expression to create a new, masked dem:
r.mapcalc "dem_masked = if(dem >= min_value && dem <= max_value, dem, null())"

You enter the actual min and max values from the v.univar output. This will result in a new GRASS raster, "dem_masked" with null (no value) in place of the high and low elevations.


How to write equation of state in terms of partition function?

Question: While studying quantum gases (fermions, bosons), equation of state written were $PV = k_B T Z_{gr}$, where $Z_{gr}$ is the partition function of grand canonical ensemble. $P$ and $V$ are pressure and volume, respectively. How can we derive this equation of state? Is it same equation that we use for ideal gases ($PV=NRT$) or is it different?
Comment: Did you find an expression for $Z_{gr}$?
Answer: When dealing with quantum gases you have to apply a correction to your result from the ideal gas!
Let us take a non-relativistic ideal gas, in which each particle has an energy of
$$
\epsilon=\frac{p^2}{2m}.
$$
The ideal gas law follows from
$$
N(T,V,\mu)=\frac{1}{\beta}\frac{\partial\ln Y}{\partial\mu},
$$
so we need an expression for $\ln Y$. For bosons, we can write:
$\begin{align}
Y(T,V,\mu)&=\sum_r\exp\left(-\beta\left(E_r\left(V,N_r\right)-\mu N_r\right)\right)\\
&=\prod_i\sum_{n_{p_i}}\exp(-\beta(\epsilon_{p_i}-\mu)n_{p_i})\\
&=\prod_p\frac{1}{1-\exp(-\beta(\epsilon_p-\mu))}\\
\Rightarrow\ln Y&=-\sum_p\ln(1-\exp(-\beta(\epsilon_p-\mu)))
\end{align}$
A similar calculation yields $$\ln Y=2\sum_p\ln(1+\exp(-\beta(\epsilon_p-\mu)))$$ for fermions.
To find the ideal gas law and its quantum mechanical corrections we need to do a series expansion of those expressions:
$$
\ln Y = (2s+1)\sum_p\left(\exp(-\beta(\epsilon_p-\mu))\pm\frac{1}{2}\exp(-2\beta(\epsilon_p-\mu))+...\right)
$$
with $s=0$ for bosons and $s=1/2$ for fermions.
The possible momentum states are very dense, therefore we can treat the sum as an integral, which yields:
$\begin{align}
\ln Y &= (2s+1)\frac{V}{(2\pi\hbar)^3}\int\!\mathrm{d}^3p\,\exp(\beta\mu)\left(\exp\left(-\frac{p^2}{2mk_BT}\right)\pm\frac{1}{2}\exp(\beta\mu)\exp\left(-\frac{p^2}{mk_BT}\right)+...\right)\\
&=(2s+1)\frac{V}{(2\pi\hbar)^3}\exp(\beta\mu)\frac{1}{2}\sqrt{\pi}\sqrt{mk_BT}\left(2\sqrt{2}+\exp{\beta\mu}+...\right)\\
&:=(2s+1)\frac{V}{\lambda^3}\left(\exp(\beta\mu)\pm2^{-5/2}\exp(2\beta\mu)+...\right)
\end{align}$
in the last step I just summed together some constants, to get the thermal wavelength $\lambda$, as mentioned in another answer.
Now we use $N(T,V,\mu)=\frac{1}{\beta}\frac{\partial\ln Y}{\partial\mu}$. In the zeroth-order approximation we find:
$$
N(T,V,\mu)=\frac{1}{\beta}\frac{\partial\ln Y}{\partial\mu}=\ln Y = \frac{PV}{k_BT},
$$
the ideal gas law. For higher orders we find certain corrections, such that
$$
\ln Y = N\mp\frac{2s+1}{2^{5/2}}\frac{V}{\lambda^3}\exp(2\beta\mu)+...
$$
Answer: When you got the expression of the grand canonical partition function, which depends on whether you are studying bosons or fermions, you can use the link with the grand potential $ \Phi = -k_BT \ln (Z_{gr})$.
From this you can use the relation $$ P = -\frac{\partial \Phi}{\partial V},$$
which will give you the pressure in function of the partition function. From here it should be little calculation to get the form $$PV = k_BTZ_{gr}.$$
Comment: It's true that it generates the same formula, but I think that saying "it doesn't matter" is a bit bold. It can lead to confusion, and other magnitudes might differ. I'm pretty sure that $\Phi=-k_BT\ln(\mathcal{Z}_{gc})$. See Callen, for example.
Comment: Are you sure it's $F$? I think it's the grand canonical potential.
Comment: Yes,I also found a source where that is the grand canonical potential. But it doesn't really matter because it differs by a factor which does not depend on $V$. So when taking the partial derivative to $V$ it vanishes. I could be wrong but in my class it was 'derived' like that.
Comment: I've edited it. I wonder, does the same relation $P = - \frac{\partial \Phi}{\partial V}$ hold? I'm starting to doubt now.
Comment: $\Phi=U-TS-\mu N = F - \mu N$, so yes, it does.
Comment: The grand potential is given by $\Phi=-PV$ and  as @FGSUZ says $\Phi=-kT \ln(Z_{\rm grand})$
Answer: All your questions are related to the choice of the independent variables. The shape of the functions might vary if you work with the canonical ensemble $(T,V,N)$ or the grand-canonical ensemble $(T,V,\mu)$. However, in the thermodynamic limit, all ensembles are equivalent and you should be able to go from one to the other through a change of variables. However, I get to a different expression for the equation of state, are you sure about its validity?

How can we derive this equation of state?

Say you have a very general system in which the canonical partition function in Boltzman's approximation (a.k.a. classical limit) reads:
$Z(T,N,V)=\frac{(V \varphi(T))^N}{N!}$
Where $\varphi$ depends on the system in question, for a three dimensional ideal gas, $\varphi(T)=(\frac{2\pi m k_B T}{h^2})$.
The grand-canonical partition function will then be:
$Z_{gr}(T,\mu,V)=\sum_N e^{\frac{\mu N}{k_B T}}Z(T,N,V)=e^{e^{\frac{\mu }{k_B T}}V \varphi(T)}$.
The associated thermodynamic potential is:
$J(T,\mu,V)=k_BT\log(Z_{gr})=k_B T e^{\frac{\mu }{k_B T}}V \varphi(T)$.
Therefore, pressure is:
$P(T,\mu,V)=(\frac{\partial J}{\partial V})_{\mu,T}= k_B T e^{\frac{\mu }{k_B T}} \varphi(T)$
Now the only thing is to get $\varphi(T)$. From the canonical partition function of one particle: $\varphi(T)=\frac{Z_1(T,V)}{V}$, from the grand-canonical partition function: $\varphi(T)=\frac{\log(Z_{gr}(T,V,\mu))}{V }e^{-\frac{\mu }{k_B T}}$ or from the grand-canonical potential: $\varphi(T)=\frac{J(T,V,\mu)}{k_B T V }e^{-\frac{\mu }{k_B T}}$. The one that approaches the most to your expression is:
$P(T,\mu,V) V=k_B T \log (Z_{gr})$.
One important thing to notice is that all possible ways of getting $\varphi$ lead to the same result (through a different functional expression).

Is it same equation that we use for ideal gases $(PV=NRT)$ or is it different?

Is the same! In order to check it, you have to go from $P(T,\mu,V)$ to $P(T,N,V)$.
$N(T,\mu,V)=(\frac{\partial J(T,\mu,V)}{\partial \mu})_{_{V,T}}=\frac{J(T,\mu,V)}{k_B T} $.
Since $P(T,\mu,V)=\frac{J(T,\mu,V)}{V}$ in one of its forms, combining the last two equations we get:
$P(T,N,V) =\frac{N k_B T}{V}$


Is it possible to get information about all apps installed on iPhone?

Question: Is it possible to get the information (app icon, app name, app location) about all apps that have been installed on iPhone/iPod?
Answer: there is a way to check if an application is installed or not, however, it does violate the Sandbox rules and Apple *may reject your app for using this. But it has been done before by other Apps that are available in the App Store, so feel free to try it
Sometimes you may want to check if a specific app is installed on the device, in case you use custom URL schemes that require some other app to be installed (you could just gray out/disable some buttons then). Unfortunately, Apple apparently does not have any function that checks this for you, so I whipped one up. It does not enumerate every single app, instead it uses the MobileInstallation cache which is always up-to-date with SpringBoard and holds the Info dictionaries of all apps installed. Although you're not "supposed" to access the cache, it's readable by App Store apps. Here is my code which at least works perfectly fine with the Simulator 2.2.1:
Code:
// Declaration
BOOL APCheckIfAppInstalled(NSString *bundleIdentifier); // Bundle identifier (eg. com.apple.mobilesafari) used to track apps

// Implementation

BOOL APCheckIfAppInstalled(NSString *bundleIdentifier)
{
    static NSString *const cacheFileName = @"com.apple.mobile.installation.plist";
    NSString *relativeCachePath = [[@"Library" stringByAppendingPathComponent: @"Caches"] stringByAppendingPathComponent: cacheFileName];
    NSDictionary *cacheDict = nil;
    NSString *path = nil;
    // Loop through all possible paths the cache could be in
    for (short i = 0; 1; i++)
    {

        switch (i) {
    case 0: // Jailbroken apps will find the cache here; their home directory is /var/mobile
        path = [NSHomeDirectory() stringByAppendingPathComponent: relativeCachePath];
        break;
    case 1: // App Store apps and Simulator will find the cache here; home (/var/mobile/) is 2 directories above sandbox folder
        path = [[NSHomeDirectory() stringByAppendingPathComponent: @"../.."] stringByAppendingPathComponent: relativeCachePath];
        break;
    case 2: // If the app is anywhere else, default to hardcoded /var/mobile/
        path = [@"/var/mobile" stringByAppendingPathComponent: relativeCachePath];
        break;
    default: // Cache not found (loop not broken)
        return NO;
        break; }

        BOOL isDir = NO;
        if ([[NSFileManager defaultManager] fileExistsAtPath: path isDirectory: &isDir] && !isDir) // Ensure that file exists
            cacheDict = [NSDictionary dictionaryWithContentsOfFile: path];

        if (cacheDict) // If cache is loaded, then break the loop. If the loop is not "broken," it will return NO later (default: case)
            break;
    }

    NSDictionary *system = [cacheDict objectForKey: @"System"]; // First check all system (jailbroken) apps
    if ([system objectForKey: bundleIdentifier]) return YES;
    NSDictionary *user = [cacheDict objectForKey: @"User"]; // Then all the user (App Store /var/mobile/Applications) apps
    if ([user objectForKey: bundleIdentifier]) return YES;

    // If nothing returned YES already, we'll return NO now
    return NO;
}

Here is an example of this, assuming that your app is named "yourselfmadeapp" and is an app in the app store.
Code:
NSArray *bundles2Check = [NSArray arrayWithObjects: @"com.apple.mobilesafari", @"com.yourcompany.yourselfmadeapp", @"com.blahblah.nonexistent", nil];
for (NSString *identifier in bundles2Check)
    if (APCheckIfAppInstalled(identifier))
        NSLog(@"App installed: %@", identifier);
    else
        NSLog(@"App not installed: %@", identifier);

Log Output:
Code:

2009-01-30 12:19:20.250
  SomeApp[266:20b] App installed:
  com.apple.mobilesafari 2009-01-30
  12:19:20.254 SomeApp[266:20b] App
  installed:
  com.yourcompany.yourselfmadeapp
  2009-01-30 12:19:20.260
  SomeApp[266:20b] App not installed:
  com.blahblah.nonexistent

Try this out before using it, I think Apple changed where the MobileInstallation.plist is located and if you do change it, try it out on an actual device not the simulator. Good Luck!
http://www.iphonedevsdk.com/forum/iphone-sdk-development/37103-finding-out-what-apps-installed.html
PK
Comment: This post is more than 2 years old ,you didn't think it would work right out of the bat with iOS 6 did you?
Also, explain what is not working, simply saying it's not working properly is not enough to help you.
Comment: Pavan, do you know of any apps on app store that use this? Thanks!
Comment: Not that I can think of; but if any application is interested to know whether a specific application is installed on a user's iDevice, the best way to go about doing that is to use the `iPhone URL schemes` and you can do a test if the application is installed by doing something like this: `if ([ourApplication canOpenURL:@"fb://"]) { NSLog(@"facebook installed");};` etc. That will let you know whether facebook is installed.
Comment: for me on iOS7 jailbroken all path return NO why?How can I fix that plz ?I need that urgency
Comment: ion iOS8 the com.apple.mobile.installation.plist is deleted with what it's replaced please?
Answer: Another way to get installed applications in iPhone is just call:
NSString *rootAppPath = @"/Applications";
NSArray *listApp = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:rootAppPath error:nil];

You can access then in each application to read their dictionary Info.plist to know more about these applications.
UPDATE: Obviously, this method does not work anymore (for iOS8) cause our application does not have permission to view the content of /Applications
Comment: This returns the list of default apps not the apps installed via app store.
Comment: There is no private API so it must be accepted in the App Store.
Comment: This returns *a* list of apps, not the correct one.  On the Sim, it returns the apps on the host machine, on real hardware, it looks like a list of the default apps.
Answer: Sure you can do this when the device is Jailbroken, so that you can reach out of the sandBox.
You can get the information you want by analyzing Info.plist in every .app which is located at every path "/var/mobile/Applications/", such as  "/var/mobile/Applications//*.app/Info.plist"
Here is my code.
- (void)scan
{

    NSString *pathOfApplications = @"/var/mobile/Applications";

    NSLog(@"scan begin");

    // all applications
    NSArray *arrayOfApplications = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:pathOfApplications error:nil];
    for (NSString *applicationDir in arrayOfApplications) {
        // path of an application
        NSString *pathOfApplication = [pathOfApplications stringByAppendingPathComponent:applicationDir];
        NSArray *arrayOfSubApplication = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:pathOfApplication error:nil];
        // seek for *.app
        for (NSString *applicationSubDir in arrayOfSubApplication) {
            if ([applicationSubDir hasSuffix:@".app"]) {// *.app
                NSString *path = [pathOfApplication stringByAppendingPathComponent:applicationSubDir];

                path = [path stringByAppendingPathComponent:@"Info.plist"];

                // so you get the Info.plist in the dict
                NSDictionary *dict = [NSDictionary dictionaryWithContentsOfFile:path];
                // code to analyzing the dict.
            }
        }
    }

    NSLog(@"scan end");
}

Here is sample content of an Info.plist. So, get values for any key you care about.
{
BuildMachineOSBuild = 11G63;
CFBundleDevelopmentRegion = "zh_CN";
CFBundleDisplayName = "\U4e50\U89c6\U5f71\U89c6HD";
CFBundleExecutable = LetvIpadClient;
CFBundleIconFile = "icon.png";
CFBundleIconFiles =     (
    "icon.png",
    "email@example.com"
);
CFBundleIdentifier = "com.letv.ipad.hdclient";
CFBundleInfoDictionaryVersion = "6.0";
CFBundleName = LetvIpadClient;
CFBundlePackageType = APPL;
CFBundleResourceSpecification = "ResourceRules.plist";
CFBundleShortVersionString = "3.1";
CFBundleSignature = "????";
CFBundleSupportedPlatforms =     (
    iPhoneOS
);
CFBundleURLTypes =     (
            {
        CFBundleURLName = "m.letv.com";
        CFBundleURLSchemes =             (
            letvIPad
        );
    }
);
CFBundleVersion = "3.1";
DTCompiler = "com.apple.compilers.llvmgcc42";
DTPlatformBuild = 10A403;
DTPlatformName = iphoneos;
DTPlatformVersion = "6.0";
DTSDKBuild = 10A403;
DTSDKName = "iphoneos6.0";
DTXcode = 0450;
DTXcodeBuild = 4G182;
LSRequiresIPhoneOS = 0;
MinimumOSVersion = "4.3";
UIDeviceFamily =     (
    2
);
"UILaunchImageFile~ipad" =     (
    "Default.png",
    "firstname.lastname@example.com"
);
UIPrerenderedIcon = 1;
UIStatusBarHidden = 1;
UISupportedInterfaceOrientations =     (
    UIInterfaceOrientationPortrait,
    UIInterfaceOrientationPortraitUpsideDown,
    UIInterfaceOrientationLandscapeLeft,
    UIInterfaceOrientationLandscapeRight
);
"UISupportedInterfaceOrientations~ipad" =     (
    UIInterfaceOrientationLandscapeRight,
    UIInterfaceOrientationLandscapeLeft
);
}

Answer: try this, it will work even with non-jailbroken devices
#include <objc/runtime.h>
Class LSApplicationWorkspace_class = objc_getClass("LSApplicationWorkspace");
SEL selector=NSSelectorFromString(@"defaultWorkspace");
NSObject* workspace = [LSApplicationWorkspace_class performSelector:selector];

SEL selectorALL = NSSelectorFromString(@"allApplications");
NSLog(@"apps: %@", [workspace performSelector:selectorALL]);


Setup a Simple Domain for Windows and Mac

Question: I have a couple of windows pc's and a mac at home and don't like the idea that i have to have accounts on all the computer for each user at home. Is there a simple domain, no tech-net subscription for win server, that I can put my windows pc's (Win7, win8, WinRT), and mac (OSX) on? I'm cool with running some sort of *nix as I have a plug pc that is not being used at the moment. I'm not looking for the answer "Buy windows serer". I would write a custom GINA for my windows PC's before I do that...
Comment: I'm not sure from the server side of things, but I do know from the Windows side of things you need to have Pro or above in order to connect to a domain, Home and Home Premium versions will not work.
Comment: @Taegost That is a very important thing to mention. Yes, both versions of windows are pro. (I wanted the extra license for a vm just in case, though ill never use it)
Answer: You should look into Windows Home Server 2011. It is based off of Windows Server 2008R2 and can do all of what you're looking for including more; file sharing server, nightly incremental backups of your computers.
I know that you said that you're not looking for buying a Windows Server, but this is only $50 and is packed with a lot of features. (http://www.howtogeek.com/57729/how-to-make-windows-home-server-into-a-domain-controller/)
This would probably be a good method. You would just want to make sure that your versions of windows support joining a domain. Ie. Windows 7 Home doesn't support joining a domain.
Comment: I would like to say that I agree that $50 is not unreasonable, to protect this answer from down votes.
Comment: People say that backups is not important, but I have been in more situations than not where this has saved me or a family member. It's also great if you're wanting to upgrade a platter to SSD with little to no downtime. I recently upgraded my wife's laptop to a SSD/HDD hybrid because she was complaining that it was slow. It allowed me to restore her laptop from the latest backup and zero data loss. I do not use the Domain Controller portion of the server as I do not have a real specific need.


Do shared hosting providers typically let users run make?

Question: I had an ersatz build server running on my website, hosted by Network Solutions.  Recently, I lost permissions to run make via PHP, presumably because of some silent server move or configuration change.  I have had problems with NS in the past, so this gives me another excuse to switch hosts.  Before I do that, though, can I expect other hosts to give me the luxury of running system commands like make?
Comment: I apologize for the single-tag lameness.  There are a few others that might be applicable (like [tag:permissions]), but I don't have enough rep here.
Comment: Wouldn't it be better to ask if specific hosts(that you're presumably considering) allowed this, rather than if arbitrary unknown ones *might*?
Answer: Any decent/competent web host should allow you access to make as there are no reasons not to and it's a pretty commonly needed utility.
Letting users compile their own custom build of PHP or install apps in their home directory poses no additional security risks to a properly configured server, and without access to basic utilities like make, there's not much point to having SSH access.


Plumber wants to replace PRV valve into house at same time he is replacing hot water heater

Question: We are replacing our 16 year old hot water heater that has not be running the last day or so.  The plumber tested the water pressure of the tank at 90 PSI.  He said that is too high.  We had a PRV valve installed a couple of years ago where the water comes into the house.  The plumber is saying that it must be bad and it will need to be replaced with the water heater or it voids the warranty.
Is this true?
Comment: Depends if the plumber is scamming you or not - if the PRV is adjustable, can it be adjusted? Do you have a pressure gauge in place, what does that show? If you don't have a pressure gauge in place, why the heck not? - Without one, you have no idea at any time if the thing works or not... Certainly could be the case that it has failed (if only a couple of years old, was there a warranty on it? Why did it fail, if it failed? Perhaps it needs something like a filter rated for the full supply pressure ahead of it to protect it from junk in the water supply?) Test with your own gauge for a start.
Comment: I agree the PRV should be set no higher than 80 if it is not working it would be a good time to replace it. Normally turning the adjustment CCW counter clockwise will reduce the pressure.
Comment: I had a plumber lie to me about the PRV failing and reported an excessive pressure level. I tested the water pressure and it was normal (60 PSI). I think there are a few unethical actors out there trying to jack up their bids so that they can install new, spendy PRV's. As Reagan once said: "Trust but verify".
Answer: Normal household water pressure is about 40 to 50 PSI. A water heater pressure relief valve is normally set to to open at 150 PSI so 90 PSI voiding a warranty seems like a stretch. If in fact your pressure is at 90 PSI I'd be concerned about your faucets and appliances valves, ice makers, etc.. because 90 is too high.  I'd ask him to check it again and watch him do it (so you'll know how to do it in the future, LOL) if it's high, you might want him to do it. If you have him do it, think about also having him replace your main shut off as those almost always go bad and they're usually right next to the PRV. Good luck and stay safe out there.


When to use Ridge regression and Lasso regression. What can be achieved while using these techniques rather than the linear regression model

Question: I am looking forward to learning more about the regularized regression techniques like Ridge and Lasso regression. I would like to know what can be achieved by using these techniques when compared to linear regression model. Also in what situation we should adopt these techniques. And what makes these two techniques different. I am looking to understand the concept and maths behind these techniques. I would request to share your valuable knowledge.
Comment: The corresponding chapters in this leading (and freely downloadable) textbook offer good insights: http://www-bcf.usc.edu/~gareth/ISL/
Comment: @ChristophHanck, that link doesn't work now. Will you be able to update the msg with a new link?
Comment: This will be https://www.statlearning.com/ (google "introduction to statistical learning" in case of further URL changes).
Answer: In short, ridge regression and lasso are regression techniques optimized for prediction, rather than inference.
Normal regression gives you unbiased regression coefficients (maximum likelihood estimates "as observed in the data-set").
Ridge and lasso regression allow you to regularize ("shrink") coefficients. This means that the estimated coefficients are pushed towards 0, to make them work better on new data-sets ("optimized for prediction"). This allows you to use complex models and avoid over-fitting at the same time.
For both ridge and lasso you have to set a so-called "meta-parameter" that defines how aggressive regularization is performed. Meta-parameters are usually chosen by cross-validation. For Ridge regression the meta-parameter is often called "alpha" or "L2"; it simply defines regularization strength. For LASSO the meta-parameter is often called "lambda", or "L1". In contrast to Ridge, the LASSO regularization will actually set less-important predictors to 0 and help you with choosing the predictors that can be left out of the model. The two methods are combined in "Elastic Net" Regularization. Here, both parameters can be set, with "L2" defining regularization strength and "L1" the desired sparseness of results.
Here you find a nice intro to the topic: http://scikit-learn.org/stable/modules/linear_model.html
Comment: Could you give more details about the 2 meta-parameters LASSO uses? I searched around and it seems that LASSO uses only 1
Comment: Thanks for raising my awareness to this point. I've previously mixed up  "LASSO" with the more general "Elastic Net". See correction above.
Answer: Even though the linear model may be optimal for the data given to create the model, it is not necessarily guaranteed to be the best model for predictions on unseen data
If our underlying data follows a relatively simple model, and the model we use is too complex for the task, what we are essentially doing is we are putting too much weight on any possible change or variance in the data. Our model is overreacting and overcompensating for even the slightest change in our data. People in the field of statistics and machine learning call this phenomenon overfitting. When you have features in your dataset that are highly linearly correlated with other features, turns out linear models will be likely to overfit.
Ridge Regression, avoids over fitting by adding a penalty to models that have too large coefficients.
Comment: Well, yes, but ridge regression is an alternative estimator for a linear model
Answer: It all depends on the kind of problem you are dealing with. On a first glance, deciding upon a model of choice can be tricky.I think, first you should understand the dataset and how features interact with each other and come up to a good representation of your dataset to be used for modelling then comes your model of choice.
suppose you have a high dimensionality and high correlation in your dataset, then you would want to prefer L1(lasso) regularisation since it penalises less important features more and makes them zero which gives you the benefit of algorithmic feature selection and would make robust predictions than L2(ridge) regularisation but sometimes it can remove certain signals from the model even when they have information so it should be used carefully.
L2 regularisation handles the model complexity by focusing more on the important features which contribute more to the overall error than the less important features. But still, it uses information from less important features in the model.
Different features contribute differently to the overall error and naturally our quest is to focus more on the important features which contribute more to the error than less important ones which can be handles with L2(ridge) regularisation.


"Tall, Dark, and Handsome" Character Archetype?

Question: I'm looking a singular word that implies the archetype of "tall dark and handsome". The origin of the word would also help-- I've seen a lot of responses, here and on Google, referencing "The Story of Hester Malpas," but that's not what I'm thinking of. I was once told there was a word that meant that, and it was derived from the first use of the character type, and now I'm totally blanking on it the night before an essay on a character that totally embodies that.
Comment: What, like a *hunk*? Can you clarify what type of word you're looking for by adding an example sentence?
Comment: I don't think there is any definitive "first use" of the stereotype. It was a fortune telling cliché long before romantic literature was a thing ("you will meet a tall, dark stranger", etc.). Some of the precursors of today's stereotypical romantic hero might be the Byronic Hero, Austen's Mr. Darcy, and Heathcliff from Wuthering Heights...
Comment: http://tvtropes.org/pmwiki/pmwiki.php/Main/TallDarkAndHandsome
Comment: Perhaps you are thinking of Adonis? Don't think this is the archetype for tall, dark and handsome, but there is some conceptual overlap.
Answer: The 'Byronic Hero', while not a singular word, is definitely the right kind of area. Examples include:

Heathcliff: "A half-civilized ferocity lurked yet in the depressed brows and eyes full of black fire, but it was subdued; and his manner was even dignified: quite divested of roughness, though stern for grace..." (Wuthering Heights, Emily Bronte, Ch. 10)
Satan: "Satan, with vast and haughty strides advanced,
Came towering, armed in adamant and gold..." (Paradise Lost, John Milton, Book VI)

If that's not quite right, a one-word phrase that might be helpful for you is "antihero". There's a definition here (https://en.oxforddictionaries.com/definition/anti-hero), for starters.


If a file is digitally signed, is posting a hash very useful for security purposes?

Question: In this question I asked about how to handle situations when SHA-256 hashes are not available for a file downloaded from the internet that contains executable code.  Another community member insightfully asked if a hash is really needed to validate the download (in this case, LibreOffice), since it is a digitally signed file.
I don't know the answer to that good question, so I thought it best to ask it here: If a file is digitally signed, is it important to also have a good hash (such as SHA-256) to help validate the file?
One of my concerns is that is is trivially simple to create strings of text that look nearly identical (and in some cases completely identical), but are actually different (often by using various Unicode characters).  For a digital signature to be secure, it would require sufficient safeguards to prevent this type of attack.
Comment: It's not necessary, but it is good form. Some people may only care about integrity because they worry about accidental changes, rather than malicious changes.
Comment: @MechMK1 accidental changes can be detect with Digital Signature as well. How hash can help with extra security? May be just for the sake of self satisfaction or is there any use case I'm missing?
Comment: @saurabh Yes, but they also require me to setup gpg, import a key from somewhere, verify that it's the correct key, set that I trust that key, etc.. A plain hash on the other hand is easier to work with.
Answer: I do not think so. Forging Code signing signature is infeasible whereas hashes can be updated to match the uploaded binary. If the website, which is providing you, the download is compromised than hashes can also be changed. This is not possible with digital signature and you at least need signing key access.
So, Digital Signature is sufficiently secure at least more secure than providing the hashes. Digital Signature uses PKI (Publick Key infrastructure) for verification of signatures. You can read it more over here Digital Verification
For signing binaries like msi you will need code-signing certificates which can be issued by trusted CA's or CA you trust. You will sign the binaries with signing keys (Private Keys). There are measures in place to secure these signing keys like storing it in Hardware Security Module HSM, TPM etc.
Trusted third party CA's do not sign code-signing keys without checking the security of keys or else they provide security hardware themselves before releasing the keys like Tokens (this depends on type of code signing certificate as well for  e.g. EV code signing certs). There are other measures as well like physical verification or organization level verification. The process is quite good.
However, there is no way to find if code-signing keys signed by CA are compromised until the vendor release a statement and inform CA which in turn release the CRL (Certificate revocation list).
Before you verify the digital signature, you can ensure that:

Integrity of the system where you verify the binaries is intact.
System is regularly updated with latest patch specifically certificate/CRL from trusted source such as Microsoft.

Because of the key security measures and other verification measures you only trust the trusted third parties CA like DigiCert, QuoVadis etc.
Comment: Thank you for the helpful answer.  Upvoted!  A follow-up question: https://security.stackexchange.com/questions/251309/how-to-update-certificates-crl-on-windows-7-systems-that-are-without-support-con


How to change delete function on Photoshop?

Question: I just upgraded from CS6 2018 to 2020. Now whenever I select a white background and press delete, a fill popup shows up instead of making the image transparent. How do I change this?
Example
Answer: I found out that the image I was editing was a jpg. Converting it to a .png made the background back to transparent.
Comment: This gets around the issue, but it's a roundabout way to do it. Images open in their initial state. Like when you open a jpg in PS, it will have a `background layer`, because jpg has no transparency, but this doesn't mean much when you're opening it in PS. I'll get back to that. Png on the other hand has transparency and will open without a `background layer`. `Background layer` is never transparent. **The proper solution to the problem is to convert the `background layer` into a normal layer by for example double clicking it.** Then it will behave the same as a png when it's first opened.
Answer: Photoshop inserts background color to the deleted area in 2 situations:
1) Your image has color mode which hasn't transparency. RGB and CMYK have it. Set the mode to RGB  by applying Image > Mode > RGB
2) You have only a background. Open the layers panel to see the situation. Remove the lock icon or double click the layer in the layers panel to get a possibility to convert the background to a layer which can have transparent areas
Photoshop opens JPGs as "background only". I bet most of the people who want to make edits to a JPG in Photoshop either duplicate the background to get a layer or convert it to a layer.
You can flatten a multilayer image to "background only" by applying Layers > Flatten image.  By doing it you throw away all those editing and recomposing possibilities which layered structure provides.


Hypothetically, if our sun was yellow would we still see "white" colours?

Question: Hypothetically,if our sun was yellow would we still be able to see white colour on objects like snow,cotton etc.? Would we have the phenomenon of rainbow?
Comment: https://en.wikipedia.org/wiki/The_dress
Comment: Our Sun *isn't yellow??* https://i.stack.imgur.com/bq60f.jpg Seriously though, I think the question is primarily opinion-based as different people in different fields will have a different working definition of what white or yellow illumination means, and what white or yellow appearance of objects means. It's a slippery-slope question unless you are in a room full of color experts with a copy of  [Billmeyer and Saltzman](https://onlinelibrary.wiley.com/doi/abs/10.1002/col.10083) handy. Color is totally subjective unless you really "go deep" collect a lot of data, and do a lot of analysis.
Answer: Every "normal" star, produces a more or less "black-body" spectrum, at some temperature (more-or-less because there are brighter and darker lines caused by specific elements in the upper layers of the star), so there will always be a range of colours of light (or electromagnetic radiation at least) in its light, which can be spread to give a rainbow.
A white object (like ice crystals, or bleached cotton) reflects all colours of light more or less equally (over some range of wavelengths of interest, anyway).
So, if the Sun suddenly cooled, so that it was emitting a 5000K black body spectrum instead of its current 6000K, with more red light and less blue light, that change would be reflected in the light reflected from a white object, which would look more yellow or orange to our eyes.

There are a couple of caveats though:

As @badjohn pointed out in a simultaneous answer, our brain processes visual information in odd ways. If the colour of sunlight changed, the brain might try and compensate for the overall shift and you might see everything more as you expect it, and less as it now is.
We evolved to perceive sunlight as white. If we had evolved around a cooler star, we would probably perceive a different colour as white.
Comment: Walk indoors on a bright day to a room with no windows and a low power incandescent bulb.  Do you say: " Hey, my shirt looks orange, I was sure that it was white when I was outside"?  Your eyes (1) are clever enough to adjust and figure out that your shirt is still reflecting  all visible frequencies approximately equally.  (1) Well, brain actually.  Most digital cameras have automatic white balance which does something similar (optional on fancier ones).
Comment: Both answers were helpful :)
Answer: We see  white objects as white under a very wide range of lighting conditions.  Photography resources will be quite helpful.  Look at a white piece of paper at dawn, at noon, and under a variety of artificial light sources.  The spectrum of the light hitting your eye will vary a lot but, except in extreme cases, it will stil look white.  The answer is not in astronomy but the visual processing in our brains.
Some extra material.
As I just mentioned in a comment, digital cameras do something similar and call it automatic white balance.  Search for that and you will find loads of discussions in photography websites.
It fails in some extreme cases.  E.g. sodium street lamps which are close to monochromatic and a combination of UV lamps and fluorescent substances.
Answer: You have confused solar spectral output with our neurological processing of collected photons.  While it's true that a careful mix of red,blue, green photons may be perceived as "white" or "off-white,"  whiteness in general occurs when either (1) the cones, which discriminate on the basis of wavelength, saturate and all "output" signals are peak, or (2) the light levels are so low that only the rods, which do not discriminate by wavelength but have higher sensitivity than the cones, respond.  For example, in a dim night locale, everything is black, grey or white.
It gets worse, as our brain reprocesses the incoming signals to make the colors we see "fit expectations."  That's why we tend to observe everything in our backyard garden as having the same coloration all day, but if we take a photo at sunset, the image will show a skew towards the red (due to extreme loss of blue, green thru the atmosphere).
Answer: If the Sun was an early K or late G-type star for example, the white that we see in objects such as snow would still exist - just reflecting yellow light instead of what we are used to. If you look outside at snow in the evening, while the sun lower on the horizon and is yellowed due to Rayleigh scattering (or in other words, at about 4500K in color temperature, congruent with orange-yellow stars), the snow would still appear as white, just reflecting a bit of that color now. So no, an object's color is not directly influenced by the Sun's light. That would be like saying a piece of paper turns orange when it's under a warm lamp.
However, if the Sun was yellow, there would be more noticeable effects on our planet. For example, the sky might be a more pale color, as yellow stars do not contain as much blue light as white stars do, meaning the atmosphere has less of it to scatter. With that in mind, the Sun itself would also appear more orangey in our sky. Redder colors in rainbows would be more prominent, while blues would be faded. Then our planet will frost over because we're out of our habitable zone of our now cold star...


Are SFDC resources older than 2 years still useful or valid for Dev 401 test preperation?

Question: I'm studying for my Dev 401, I've taken all of the video trainings and done all the workbooks. I keep finding all sorts of resources(old dumps, videos, study guides) that are older than 2 years. I'm curious, when it related to Dev 401 are these resources still useful or should I consider them to be out of date and merit no value.
Comment: I used all of the workbooks and the fundamentals book when I studied for the 401 exam, I would trust but verify the older information since a lot has changed in the last 2 years. I know that the fundamentals book has changed a ton since I went through it.
Answer: The fundamentals of Salesforce haven't changed per se. If anything, what's happened is that new features have been added. If you look at and older copy of Force.com Fundamentals and compare it to the latest copy (the on-line version is the only one that will be fully up to date, not the downloadable PDF -> Oct '13), you'll see that most of what's different are that it's been updated to reflect new features which have been added to Salesforce since the book was originally written.
So what that says is that if you go back and view various videos available on YouTube and iTunes that were created a couple of years ago and then review the material that covers new releases to the platform which have occurred since that time, you'll be up to date. Most of the new things that are in the above Document are primarily in the last 2 sections of the book with the exception of the Advanced Setup interface for security, etc on objects that was introduced about 2 years ago.


Research Funding distribution across discliplines

Question: Just curious: how does research funding distributions across different disciplines in EU and US? I would like to see a more detailed chart where different physical sciences and biological sciences are listed. Thanks!
Comment: This question is not so straightforward to answer.  What counts?  In the US, funding is not only provided by the a large number of federal government agencies, but also by state and local governments, various nonprofit foundations, private individuals who want to use some of their fortune to advance knowledge, private companies, sometimes because they think they can profit from the knowledge, sometimes altruistically, sometimes to introduce bias into the journal literature.
Comment: That's the point. What are all the possible funding sources? It is also unclear to me, and it what I would like to learn.
Comment: One can list the major ongoing sources in a particular field, but how do you list Jim-Bob Bachelor Small Businessman who just died and left his entire fortune of $2 million for research on Plutarch because that was his favorite course in college 50 years ago?  People in the US do that sort of thing.  (And Classics is so underfunded that $2 million would be significant.)
Answer: In the UK, the EPSRC (Engineering & Physical Sciences Research Council) has a handy visualisation on their website to see the current funding distribution within their remit.
A similar visualisation has been created for the BBSRC (Biotechnology and Biological Sciences Research Council) - also UK. Although this only appears to be for projects active in 2012.
UberResearch compiles data on research funding, however, their service is not freely available. They do have some samples, which my be interesting. They appear to have data from the UK, US and EU. I've never used this service so cannot comment on its usefulness.
Most of the public funding bodies provide data on funded projects on their websites. However, it would require a lot of processing to make visualisations of it.


Reuse existing objects for immutable objects?

Question: In Python, how is it possible to reuse existing equal immutable objects (like is done for str)? Can this be done just by defining a __hash__ method, or does it require more complicated measures?
Comment: +1 Interesting question.  I know for strings the process is called "interning" but I'm not sure if that term is specifically for strings, as I've never heard of anything else getting interned.
Comment: Often the hash function will be implemented as the address of the object, so you get a chicken/egg problem; the hash won't be equal until you've already decided the objects are the same.
Answer: If you want to create via the class constructor and have it return a previously created object then you will need to provide a __new__ method (because by the time you get to __init__ the object has already been created).
Here is a simple example - if the value used to initialise has been seen before then a previously created object is returned rather than a new one created:
class Cached(object):
    """Simple example of immutable object reuse."""

    def __init__(self, i):
        self.i = i

    def __new__(cls, i, _cache={}):
        try:
            return _cache[i]
        except KeyError:
            # you must call __new__ on the base class
            x = super(Cached, cls).__new__(cls)
            x.__init__(i)
            _cache[i] = x
            return x

Note that for this example you can use anything to initialise as long as it's hashable. And just to show that objects really are being reused:
>>> a = Cached(100)
>>> b = Cached(200)
>>> c = Cached(100)
>>> a is b
False
>>> a is c
True

Comment: Note that `__init__` will be called *twice* when there is a cache miss and once if there's a cache hit. You need to explicitly protect against multiple constructor calls e.g. set a flag at the end of `__init__` and return immediately if the flag is set.
Comment: Thank you. This worked, the only difference I used was like so: `class a(object): _cached = {} `. The reason I did it this way is because I think instantiating  objects in function parameters can be confusing, because in some cases people may not realize that it is being instantiated only once, and that it refers to the same object for all the calls of that function.
Comment: @Abafei: Yes, your change is a good one. A lot of people like to avoid mutable default arguments as they often confuse others - I'm not quite so considerate :)
Answer: There are two 'software engineering' solutions to this that don't require any low-level knowledge of Python. They apply in the following scenarios:
First Scenario: Objects of your class are 'equal' if they are constructed with the same constructor parameters, and equality won't change over time after construction. Solution: Use a factory that hashses the constructor parameters:
class MyClass:
  def __init__(self, someint, someotherint):
    self.a = someint
    self.b = someotherint

cachedict = { }
def construct_myobject(someint, someotherint):
  if (someint, someotherint) not in cachedict:
    cachedict[(someint, someotherint)] = MyClass(someint, someotherint)
  return cachedict[(someint, someotherint)]

This approach essentially limits the instances of your class to one unique object per distinct input pair. There are obvious drawbacks as well: not all types are easily hashable and so on.
Second Scenario: Objects of your class are mutable and their 'equality' may change over time. Solution: define a class-level registry of equal instances:
class MyClass:
  registry = { }

  def __init__(self, someint, someotherint, third):
    MyClass.registry[id(self)] = (someint, someotherint)
    self.someint = someint
    self.someotherint = someotherint
    self.third = third

  def __eq__(self, other):
    return MyClass.registry[id(self)] == MyClass.registry[id(other)]

  def update(self, someint, someotherint):
    MyClass.registry[id(self)] = (someint, someotherint)

In this example, objects with the same someint, someotherint pair are equal, while the third parameter does not factor in. The trick is to keep the parameters in registry in sync. As an alternative to update, you could override getattr and setattr for your class instead; this would ensure that any assignment foo.someint = y would be kept synced with your class-level dictionary. See an example here.
Answer: I believe you would have to keep a dict {args: object} of instances already created, then override the class' __new__ method to check in that dictionary, and return the relevant object if it already existed. Note that I haven't implemented or tested this idea. Of course, strings are handled at the C level.
Comment: Yes this works - I've successfully used this method before. I may post some example code tomorrow if no one beats me to it.


Why is my drone oscillating on position hold after I move it a few meters?

Question: I have built a 5 inch quadcopter using inav 5, mateksys f722, and M8q-5883 gps. When I enable position hold with gps, the drone is relatively stable, but when I move it a few meters forward, it will oscillate a bit (fly forward, back, forward, back, ...) until becoming stable again. Ultimately it moves to the correct position, but I don't know why the oscillation is happening. The faster I move it to a new position, the greater the oscillation effect (although ultimately becoming stable again). Does anyone have any suggestions?
Answer: That sounds like there's too much gain in the GPS position holding functions. It's trying too hard to reach a specified position, and overshooting. I've no idea if that's adjustable though.


Hosting subdomains separately with and without SSL while using a wildcard certificate

Question: I'm having a strange problem with a separately hosted subdomain I have. I'm running an application on Engine Yard, let's call it mysite.com. I have a wildcard SSL certificate installed there which covers all the subdomains (things like api.mysite.com). We recently decided to migrate our blog to be hosted independently (right now it lives on wordpress.com). Because I can't run the blog alongside our Rails app with ease on Engine Yard, I decided to grab some cheap hosting space from Dreamhost to host our Wordpress blog there. I set up the server there to fully host our subdomain (let's call it blog.mysite.com), and updated the DNS A record on Hover (our DNS provider) to point blog.mysite.com to the Dreamhost server. So here's the issue:
If I go to blog.mysite.com via Firefox or Safari on my Mac I see the basic Wordpress install which I set up. However, if I try to view things with Chrome I get the following error:
This webpage is not available
Error 118 (net::ERR_CONNECTION_TIMED_OUT): The operation timed out.
This happens on all Macs running Chrome I could get my hands on. I tried both clearing the cache and flushing the DNS but nothing. The weirdest part is Chrome keeps looking for https://blog.mysite.com instead of http://blog.mysite.com. There is no SSL cert installed on the subdomain for the blog on Dreamhost because it's not necessary.
Has anyone ever come across this before? And in case anyone wants to try the actual address is http://blog.frestyl.com.
Answer: sounds like you have a 301 permanent redirect that Chrome registered http://blog.frestyl.com -> https://blog.frestyl.com. Besides clearing the cache I'm not sure what else can be done.


What is modulo switching, in a nutshell?

Question: Coupled with the terms bootstrapping and relinearization, the term modulo switching appears a lot in the FHE literature. What is it and how does it relate to the other two?
Comment: Modulus switching can be viewed as dropping precision (thinking of the group $\mathbb Z/q\mathbb Z$ as a scaled discretization of the torus $\mathbb R/\mathbb Z$). Basically dropping ``integral'' ticks on the continuous circle.
Comment: See [The second technique is modulus switching](https://www.hindawi.com/journals/tswj/2014/983862/)
Comment: @kelalaka So, in a nutshell: **relinearization is modulo-switching**. Is that correct? Two names for the very same thing?
Comment: AFAIK, [no see 3.3](https://courses.cs.ut.ee/2011/security-seminar-fall/uploads/Main/fauzi-draft.pdf)
Comment: @kelalaka Thanks for the references. I will take a look.
Comment: I was wondering whether rescaling is the same as modulus swtiching?
Answer: The basic idea:
In homomorphic encryption schemes, the ciphertexts are noisy and we want to refrain the noise from growing too fast as we operate homomorphically. So the idea here is to multiply a ciphertext $c$ defined over $\mathbb{Z}_{q_0}$ by $q_1 / q_0$ so that the noise term is also multiplied. Then, by setting $q_1 < q_0$, we go from a noise term $e$ to a smaller noise approximately equal to $\frac{e q_1}{ q_0}$.
Furthermore, the ciphertext we obtain is then defined over $\mathbb{Z}_{q_1}$.
What is the advantage of doing this?
Let's say that fresh ciphertexts have noise term bounded by $B$ and that one homomorphic multiplication increases the noise to (approx.) $B^2$.
Then, two multiplications gives us $(B^2)^2 = B^{4}$, three gives us
$(B^4)^2 = B^8$... A sequence of $L$ multiplications increases the noise to $B^{2^L}$.
Because the final noise must be smaller than the modulus $q$, the bit length of $q$ must be bigger than $\log B^{2^L} = \Theta(2^L)$, that is, exponential in the circuit depth.
Now, imagine that we have $q_L := B^{L+1}$, $q_{L-1} := B^{L}$, $q_{L-2} := B^{L-1}$, ..., $q_{0} := B$. Then, $q_{i-1} / q_{i} = 1/B$.
Now, the first multiplication increases the noise close to $B^2$, but multiplying by $q_L / q_{L-1}$ scales it down to $B$ again.
Then, the second product increases the noise to $B^2$ and multiplying by $q_{L-1} / q_{L-2}$ (modulus switching) reduces the noise to $B$ again.
Proceeding like this, after $L$ multiplications, we have a ciphertext defined over $\mathbb Z_{q_0}$ with noise smaller than $B$, which can be correctly decrypted.
Now, the bit length of the modulus we need to use is $\log q_L = \Theta(L)$, that is, linear in the circuit depth instead of exponential!
But how the modulus switching is really done?
Of course, we can't simply multiply $c$ by $q_0 / q_1$, because we will not obtain a valid ciphertext, thus, each scheme has it is own way of getting a valid ciphertext out of $c\cdot q_0 / q_1$, but this is basically done by rounding.
For instance, let's say that we have the following LWE ciphertext with secret key $\vec s$ and noise term $e$:
$$
\vec c = (\vec a, \, \underbrace{\vec a\cdot \vec s + e + m \cdot q_0/2}_{b}) \in \mathbb{Z}_{q_0}^{n+1}.
$$
Then, over the integers, we have $b = \vec a\cdot \vec s + e + m \cdot q_0/2 + uq_0$
for some integer $u$.
Since the rounding function satisfies $\lceil x \rfloor = x + \epsilon$ with $-1/2 \le \epsilon \le 1/2$,
it is clear that
$$
b' := \lceil b \cdot q_1 / q_0 \rfloor
= \vec a\cdot\vec s \cdot q_1 / q_0 + e \cdot q_1 / q_0 + m\cdot q_1 / 2 + \epsilon + uq_1 \in \mathbb{Z}.
$$
Using this trick again, we define
$\vec a' := \lceil \vec a \cdot q_1 / q_0 \rfloor = \vec a \cdot q_1 / q_0 + \vec \epsilon$, with rounding applied entry-wise, and obtain
$$
b'= \vec a' \vec s -\underbrace{ \vec \epsilon \vec s+ e \cdot q_1 / q_0 + \epsilon}_{e'} + m\cdot q_1 / 2 + uq_1 \in \mathbb{Z}.
$$
Finally, reducing $b'$ modulo $q_1$ removes the term $uq_1$.
Moreover, assuming that $||\vec s||$ is small, we see that the new noise $e'$ is close to original noise $e$ scaled by $q_1/q_0$.
Therefore,
$$\lceil \vec c \cdot q_1 / q_0 \rfloor \bmod q_1 = (\vec a', \, \vec a' \vec s + e' + mq_1/2) \in \mathbb{Z}_{q_1}^{n+1}$$
is a valid ciphertext of the same message, under the same key, but with smaller noise and with respect to a smaller modulus $q_1$.
Comment: I still don't understand how this is useful.  The noise does decrease, but the relative noise is about the same (or increases); i.e. $|e|/q_0\approx |e'|/q_1$.  The smaller modulus tolerates less noise, so it doesn't seem that there's any gain.  My intuition must be wrong, but I can't see my misunderstanding.
Comment: The relative noises are basically the same, that is true. But as I explained in the answer, at the end, you can choose $\log q_L = \Theta(L)$. One explanation for the gain is that if you just multiply ciphertexts, you are "combining" the noises, so the noise grows exponentially fast. But when you switch the modulus, both ciphertexts work mod the new q_i without "combining" the moduli, so the (log of the) modulus decreases linearly instead of exponentially. That is where you gain something.
Comment: Thanks for your response.  As used in [Efficient Fully Homomorphic Encryption from (Standard) LWE](https://eprint.iacr.org/2011/344), modulus-dimension reduction isn't really a "noise reduction" technique, but rather decreases the complexity of decryption to allow bootstrapping.  As you describe modulus switching, it keeps the modulus (much) smaller than naively expected for a given multiplicative depth.  In either case, I think I now understand its utility.
Answer: Hilder Vitor Lima Pereira described modulo switching in a paper entitled Bootstrapping fully homomorphic encryption over the integers in less than one second as "a method to reduce the noise by scaling a ciphertext and switching the modulus of the ciphertext space..."
He has already answered the rest of your question about bootstrapping and relinearization here on crypto.stackexchange.
Comment: Thanks for the answer!


How does the NBA get paid by the networks for playoff games?

Question: How does the NBA get paid by the networks for playoff games?  Do they get more money the more games that are played?
Comment: I don't believe the contract is public record, though perhaps TNT and ESPN/Disney shareholders have some visibility.  The amounts are public record of course.  Personally I think they get paid a lump sum - over the length of the contract the number of playoff games is likely to be fairly predictable, even if one year has more or less.
Comment: This is interesting because if it was a flat fee, networks would want as many games as possible for advertisement and such.  If it is on a per game basis would the NBA charge more for a finals game 7.
Comment: @joe - looks like a flat fee, Disney and TNT are paying for the right to broadcast a few series ("up to X" games). The broadcasters will of course make more money from advertising if each series runs long, hence complaints from Ayesha Curry that the 2016 finals were rigged (https://www.si.com/nba/2016/06/16/ayesha-curry-accuses-nba-rigging-finals-warriors-cavs).
Answer: The NBA included rights to broadcast playoff games in the $24 billion extension signed with Disney/ESPN and TNT. This deal covers both regular season and playoff games:

Under this arrangement, ESPN will pay $1.4 billion per year, while
  Turner will pay $1.2 billion, according to Sports Business Journal.

Per the league's press release:

The National Basketball Association (NBA) has expanded its
  partnerships with Turner Broadcasting System, Inc. and The Walt Disney
  Company through new, nine-year agreements under which ABC, TNT, and
  ESPN will televise NBA games beginning with the 2016-17 season and
  running through the 2024-25 season

Quote regarding TNT playoff games:

TNT will televise up to 45 playoff games during the first two rounds
  and will have exclusive coverage of the conference semifinals games it
  telecasts and one of the conference finals each year.

Quotes regarding ABC/ESPN playoff games:

The postseason will feature the entire best-of-seven NBA Finals in
  primetime on ABC.

and

ABC, ESPN and ESPN2 will also televise up to 30 playoff games in the
  first two rounds, while ESPN will feature exclusive coverage of the
  conference semifinals games it televises and one of the conference
  finals each year.

Note that there is no accounting for additional "per-game" payments, so the two broadcast partners of the NBA are paying a flat fee every year for the entire package of games (both regular season and playoffs).
Comment: While accurate, this in no way answers the question, which asked about how the networks were paid.
Comment: @joe everything is paid as a flat fee - i'll update the question if unclear.
Comment: I don't think your article is at all clear on that, particularly as it's talking about the entire package, and is vague on the totals. It is likely there are significant terms relating to the playoff games that could involve payments per game or reimbursement if games fall below a certain number per year - who knows. I don't think this information is knowable without being a lawyer for one of the parties ...


Clay block effectiveness

Question: Just wondering if anyone has tried any of these commercial clay block paint oxide remover things... If so; do they work? Do they work on all paint types? Can I just use regular $1 a pound sculpting/pottery clay, or do I need the magic of the $20/150gram block at my local auto store?
Answer: Yes, they do work for what they're intended (removing contamination on the paint, overspray etc) and no, as far as I know they're not the same as pottery clay but have some other ingredients in them. Rubbing pottery clay on your car will more or less give you the same result as rubbing sand on it...
Keep in mind that using a clay bar only make sense as part of a full detail. You wash the car thoroughly, then you clay it, polish it, wax & seal it. Clay will not remove oxidised paint (that's not what it's intended use is) but it'll remove contaminants on the paint reasonably safely that so you don't end up either trying to remove it using polish or worse, scratch the paint while polishing. Clay also pretty much strips any existing wax and sealant off the paint so if you're not going through the full routine you're likely to make things worse, not better.
Comment: So you are saying that cleaning an claying a car alone is useless? Even if it is a 9year old car who defenitly doesn't have any way on it anymore? I have a claybay, but I cannot polish myself, and shops charge way to much for my liking..
Comment: Yes, just using a claybar can actually be detrimental to your paint's health as you strip off all the protective layers like waxes and sealants. If your 9 year old car hasn't seen wax in months or years, it's probably a good time to put some on anyway.
Comment: Thanks! Yes, as I bought mine 2 years ago off an elderly lady, so I am 99% sure she didn't bother to wax or polish the car. Now the next dilemma is to learn to do it myself or spend the bigbucks on professionals...
Answer: I have tried it many years ago on an Imron paint job. The finish felt smoother than before and the sliding rag test was better. I didn't see any visual difference. In after thought I wondered about the grit that the clay is picking up. Since you reuse the clay wouldn't the clay hold the grit and eventually act as an abrasive? It appeared to do no harm but I didn't think the results justified the effort.
Comment: The Clay should only be used a certain amount of times. I don't remember if one claybar was designed to last for something like 4 fully clayed cars (depending on the state of the cars). And as you probably know, if you drop it on the ground it's more or less unusable.

Clay block effectiveness


Introductory text on nets

Question: I have learned point-set topology using filters. Now I do functional analysis where we are using nets to do topological stuff. Therefore I search an introductory text on nets that is suitable for this purpose, i.e. to lay the foundations for usage in FA and maybe that the text assumes some knowledge in point-set topology, I don't want to start from scratch.
Answer: One of the very best references that I've seen is a PDF, Translating Between Nets and Filters, by Saitulaa Naranong that can still be found here. (I'm aware of one typo: $\Phi$ and $\Psi$ have been interchanged in the displayed implication at the top of page $11$. The one-sentence paragraph two lines down ('In other words ...') is correct.)
Answer: Two sources:

Kelley, General Topology, which popularized nets and the terminology, and
Pete Clark, Convergence, a paper developing the theory of sequences, filters and nets, and proving implications and equivalences between them.
Answer: A caveat to my answer: I don't know functional analysis, so this is just what has helped me in understanding point set topology better. I find nets build intuition in point set topology.
In no particular order:

Try the section in Engelking's General Topology starting on p. 49.
Munkres's Topology has a little section on nets with some good exercises.
Willard's General Topology chapter 4 is good.

I recommend learning the canonical way to translate nets into filters, and vice-versa. If you still want more practice, I recommend going through some of the proofs in point set topology (e.g. tube lemma) in the context of nets, and then the corresponding one in terms of filters. It's fun!
Answer: I don't know a reference that explains nets for functional analysis, but here is an introductory book on nets:
Limits - A New Approach to Real Analysis by Alan F. Beardon.

Introductory text on nets


How to add "Family Member Information" Columns to Event Attendee List report?

Question:
Is it possible to add Family Member Information columns to this screen? I run a youth program and it would be helpful to be able to pull emergency contact information when running the report. Very new to Civi.
Answer: All the checkboxes in the screenshot you posted refer to built-in CiviCRM fields. "Family Member Information" doesn't refer to any built-in CiviCRM field, so it's a bit hard to know exactly what you're asking to do.  Here are some options:

If "Family Member Information" is the name of a custom field group, then you can add those checkboxes to the report by going to Administer menu » Customize Data and Screens » Custom Fields.  Find the Family Member Information custom group and click View and Edit Custom Fields.  Change any field you'd like to appear on this report such that Searchable is set to Yes.
If "Family Member Information" is set via a relationship, you have a few options.  I won't detail them here unless you update the question to indicate this is the case, but:

Add folks to a group, use the Relationship Report.
The Extended Reports extension has an Event Attendee List report with relationships.
If you're using Drupal, this is easy to implement using Views.


JS code for web template only working for full control user

Question: I deployed the control list with paging and item template for using the with Content Search Web Parts, we have realized that users with no Full Control are not able to run the logic in these templates. SP admin has told me that this is not correct because the users won't be configured with Full Control, so my question is, what else should be configured in order to get the scripts running? My understanding is that all the users that have access to the folder will be able to run the scripts?
Answer: Yes, in most cases they should be able to, unless running scripts has been disabled by an admin or they do not have access to the place where you placed the script file. Because the file has been published, right?
Comment: Yes, the files have been published, and the users have access (Read) to the location but the issue remains unless they are Full Control which is not allowed by SP Admin. Any idea? Thanks.
Comment: Is this a self service created site, as per https://support.shortpoint.com/support/solutions/articles/1000188451-prerequisite-1-enabling-scripting-capabilities?
Comment: I'm working with SP 2013 on-premises I think this does not apply to this version, do you know if there is any similar feature in 2013?
Comment: You have tested it in different browser? Does the Chrome dev tools show anything?
Comment: Same behavior in IE and Chrome, no errors in the console for dev tools, I have been debugging the code through the Dev tools and the code is reachable (I can show alerts or console.log) but just not doing the change on values per code, it is odd.


Custom post type permalink tag 404

Question: I'm using wp-types to build a wordpress site. I'm happy with the types and their fields, I just wanted to created a function that shows parents in the url - see blow function that achieves this - also using the Custom Post Type Permalinks plugin to add my custom tag %parents% to the url.
The rewrite seems to work fine, displays in the back end etc. But when you click view document - 404.php.
I've tried rewrite flushing - mod rewrite definitely works, as friendly permalinks work for pages and don't 404, it's only on the custom post types.
My ideal url structure is: %sector%/%parents%/%postname% - Sector being a custom taxonomy. The custom taxonomy also displays fine because of the CPTP plugin but 404s as well.
wp-types does have rewrite functions but they are all turned off. it's really got me stumped, the code below is included in my functions.php - I am using bones starter theme.
Should I include this code as a plugin instead? Or is there something i've missed? The only way I can get access the items is by setting permalinks to default.
<code>    add_filter('post_type_link', 'parents_post_rewrite_function', 10, 3);

  function parents_post_rewrite_function( $post_link, $id = 0, $leavename = FALSE ) {
    $post = get_post($id);
    $postID = $post->ID;

    // Always globalize $wpdb
    global $wpdb;

    // Get current posts parent type
    $parentType = $wpdb->get_var("SELECT  `meta_key` FROM  `wp_postmeta` WHERE  `post_id` ={$postID} AND `meta_key` LIKE '%belongs%'");

    // Get current posts parentID if project
    $parentID = $wpdb->get_var("SELECT  `meta_value` FROM  `wp_postmeta` WHERE  `post_id` ={$postID} AND  `meta_key` =  '{$parentType}'" );

    if(!empty($parentID)){

    // Get parent parent posts type
    $parentParentType = $wpdb->get_var("SELECT  `meta_key` FROM  `wp_postmeta` WHERE  `post_id` ={$parentID} AND `meta_key` LIKE '%belongs%'");

    // Try and get same value from parent ID, if fails then no parent of parent
    $parentParentID = $wpdb->get_var("SELECT  `meta_value` FROM  `wp_postmeta` WHERE  `post_id` ={$parentID} AND  `meta_key` =  '{$parentParentType}'" );

    }

    if(!empty($parentParentID)){
            $parentslug = get_post($parentID)->post_name;
            $parentparentslug = get_post($parentParentID)->post_name;
            $parents = $parentparentslug.'/'.$parentslug;
    } else {
        $parents = get_post($parentID)->post_name;
    }

    // Get parents parent post ID & continue until none found
    // Convert IDs to names and render slug

    return str_replace('%parents%', $parents, $post_link);
  }
</code>
Comment: the `post_type_link` filter is just for functions that output permalinks, it has nothing to do with generating rewrite rules for incoming requests for those permalinks.
Comment: I see. I'm new to wordpress, what should I be using to get it working?
Answer: I had this same problem when I cloned my site to set up a development server - for me, it working by setting permalinks to ugly, saving, then setting them to pretty again, which rebuilt whatever had gone wrong in htaccess.

Custom post type permalink tag 404


Port mirroring as a Windows service

Question: Is anyone familiar with a port mirroring or port translation tool for Windows (2k8 if it makes a difference)? I have an installation of Openfire that needs to run over 443, however, it doesn't support client connections on that port (currently is on port 5222).
Currently the firewall is handling this port translation, however, this doesn't work for internal users as DNS is pointed to a local address and not the firewall.
Answer: I am not sure if this works using the same local IP address as remote adress, but you could try this free tool: RelayTCP

RelayTCP allows to redirect TCP/IP
  connections from a local port to a
  remote IP and port.
There is two versions of the software:

RelayTCP command line: is a command line executable to make one
  socket redirection.
RelayTCP service: is a NT service that allows make multiple
  redirections.
Answer: If you don't mind having your XMPP sessions encrypted (and why would you?!) you should be able to use stunnel to proxy between the two ports.
There are Windows binaries available, although I don't know if it can run as a service "out of the box".
Answer: You could use rinetd which is a similar solution to that of the other answers, or you could run Openfire as an administrator which will allow it to listen on ports <1024, such as 443, without any port mirroring.
Comment: I'm going to try changing the svc account to a higher privileged one. I'm interested to see if that works.
Answer: You could try socat under cygwin, it is a "swiss army knife" like netcat, but works for bidrectional connections:
http://www.dest-unreach.org/socat/
one example from the man-page:
socat -d -d -lmlocal2 \
TCP4-LISTEN:80,bind=myaddr1,su=nobody,fork,range=10.0.0.0/8,reuseaddr \
TCP4:www.domain.org:80,bind=myaddr2

TCP port forwarder, each side bound to another local IP address (bind). This example handles an almost arbitrary number of parallel or consecutive connections by fork'ing a new process after each accept() . It provides a little security by su'ing to user nobody after forking; it only permits connections from the private 10 network (range); due to reuseaddr, it allows immediate restart after master process's termination, even if some child sockets are not completely shut down. With -lmlocal2, socat logs to stderr until successfully reaching the accept loop. Further logging is directed to syslog with facility local2.
Answer: I would look first at the configuration file for Openfire, which is pretty flexible. It's been a while since I have worked with it, but the port it listens on should be a configuration option in the XML-based config file, and some Google searches and/or a review of the documentation should turn up some more details if the config file isn't self-documenting. No use making the solution more complex if you can solve it in your application, and it may (I'm not sure) be capable of listening on both ports at once even.

Port mirroring as a Windows service


How do i make all torches have fire aspect?

Question: I wanted to make all torches (Crafted and found) have fire aspect using commands. I tried some different things but they didn't work out. Does anyone else have any ideas?
Comment: Your idea is good and I will "steal" it for my modpack. xD Because with it I exactly know how to do it, however it is not easily possible (maybe some workarounds) in vanilla MC. If you are interested in modded answer, just write me and will post it here. (It involves MineTweaker)
Answer: Default items have no data of which to detect in order to separate them from custom items. Unfortunately this greatly restricts methods of replacing default items since, for example, /clear would not be able to differentiate your custom torches with default ones.
One work-around is to detect each individual slot in the player's inventory and use /replaceitem, but that requires 64 command blocks for each of the 36 slots just for detection alone.
As a quick run-down:

Find 1 torch in slot 0.
 /scoreboard players set @a TorchSlot0 1 {Inventory:[{id:"minecraft:torch",Count:1b,Slot:0b}]}

Replace that torch with 1 custom torch.
 /replaceitem entity @a[score_TorchSlot0_min=1,score_TorchSlot0=1] slot.container.0 minecraft:torch 1 0 {ench:[{id:20s,lvl:1s}]}

Which would need to be repeated for every number of torches possible in every slot possible.

If using 1.9, you can customize the loot tables so that the torches that are at least found in chests will have the desired enchantment.
For example, the chests/abandoned_mineshaft.json loot table has the following entry:
{
    "type": "item",
    "name": "minecraft:torch",
    "functions": [
        {
            "function": "set_count",
            "count": {
                "min": 1,
                "max": 16
            }
        }
    ],
    "weight": 15
}

If you want it to be enchanted with fire aspect, you'd add the enchant_randomly function while only specifying minecraft:fire_aspect:
{
    "type": "item",
    "name": "minecraft:torch",
    "functions": [
        {
            "function": "set_count",
            "count": {
                "min": 1,
                "max": 16
            }
        },
        {
            "function": "enchant_randomly",
            "enchantments":[
                "minecraft:fire_aspect"
            ]
        }
    ],
    "weight": 15
}

Comment: You are aware that clear takes a data tag, right?  You can easily count the total as well as the custom torches, and the difference gives you the default number of torches. Mind you, this doesn't do you much good since you can't clear just the default ones, unless you also set another tag to a non-default value for the custom torches.
Comment: @MBraedley `/clear` takes dataTags, but you cannot cause it to remove non-custom items because they don't **have** any extra NBT data to detect. You'd just as well be clearing the custom items provided since you cannot use a NOT operator within dataTags. Counting custom and default items is easy, the replacement part (as OP requested) is just not feasible due to that limitation, which is why I stated `/clear` is not helpful for the end result.
Comment: "unless you also set another tag to a non-default value for the custom torches"  Change the damage value and then check for the default (0).  I don't know for sure if that'll work, and currently have no way to test it, but I have a feeling it'll do exactly what's desired.  Stick a `stats` command on it to say how many torches were cleared.  Then it's just a matter of giving that many torches back, which admittedly isn't easy, but is certainly doable.
Comment: @MBraedley Unfortunately any invalid Damage values will be given the "not found" texture (black & purple boxes), which is the case with torches. I'm not sure if that can be remedied with resource packs; at the time the invalid textures were implemented for invalid Damage values, it wasn't possible to use resource packs to fix that. I haven't kept up with resource packs, though.
Answer: Are you currently playing Minecraft without mods, or with mods? Without mods, you are unable to add an enchant to items such as torches, even with commands or command blocks. There are ways however, to use command blocks to set your target that you hit with a torch on fire but it involves some more complicated programming.
An easier method of setting mobs on fire when you hit them with a torch is to download a mod. One example of a mod that works for you is this one:
http://dev.bukkit.org/bukkit-plugins/fire_aspect-torches/
Comment: It should be clarified that enchantments *can* be applied to *any* item in the game (for example, fire aspect will work just fine on a torch: `/give @p torch 1 0 {ench:[{id:20s,lvl:1s}]}`).


can i add a pre-set subject line in email links

Question: I've recently figured out that I can add email links to a SharePoint page. I was wondering if it's possible to add a default subject line or body text?
Comment: This is so clever, thank you. I am finding though that when you click on the email link, its showing the % signs in the email subject header. Is there anyway to have it so they show as spaces? I can get the subject to show but if I try at the &body= bit, it doesn't like it at all. Could I be doing something wrong, or does Sharepoint just not like it? Thanks Jen
Answer: Add ?subject= at the end of the mailto link and then add the subject line, replacing spaces with %20. So if you want an email link to go to email@example.com with a subject line that says "this is important", you would format the link as follows:
mailto:firstname.lastname@example.com?subject=this%20is%20important

you can also add body text by appending &body= and following the above formatting.
Reference the following: https://knowledge.hubspot.com/articles/kcs_article/cos-pages-editor/how-do-i-create-a-mailto-link


What is the best way of adding Kitbash 3d asset packs to the asset browser?

Question: All of the individual assets are parented to an empty. When I mark it as an asset it looks like this:

I would like to add all of the emptys as individual entries in the asset browser.
Answer: At the moment, Asset Browser still lacks the functionality to use object groups(Collections) as assets. As this functionality is essential it would be reasonable to expect it added in the future and it is in the goals as we can see here: https://developer.blender.org/T94031
So at the moment full functionality needed for what you want is simply not there. You could use Python to make collections out objects parented to the empties and then make instances of those collections that can be made into assets, however it seems that you would need to somehow generate thumbnails as well.


How can I stop the loud banging sound in my water pipes when using my clothes washer?

Question: We just bought a townhome and every time we use the LG Front loading washing machine there is a loud bang throughout the pipe system.  I checked the back and there are already 2 water hammer arrestors (Size AA) installed on the inlet to the washing machine's hot and colt water intakes.  When I checked the water pressure of the cold water it was at 80psi.  Does the banging have something to do with the water pressure or could I need bigger Water Hammer Arrestors?  We are on a municipal water supply.
Comment: The water hammer is caused by the fast acting valves in the washer.  Water hammer arrestors don't always stop the noise, so you'll likely have to try and locate any and all points in the plumbing that are banging.  Once you've found where the pipe is banging, you'll have to support and/or isolate the pipe to prevent it moving.  Then you'll have to locate and support/isolate the next area, and repeat until the bang is silenced.
Comment: How long is the pipe to the washer faucets?  Does it happen with cold or hot equally?  What kind of hose connection is there from the washer to the plumbing?
Comment: It looks like around 6ft of a braided cable between the utility sink where the cable connects in, to the intake for the washer.  This happens with both hot and cold water.
Answer: You may need to replace the arrestors.. they do fail eventually.  Also even with them in place there may be poorly secured pipes that bang around.  Try to find them and secure them properly.
Answer: You could try swapping the inlet hoses for longer ones.  The theory would be that the longer rubber hoses might absorb some of the shock (by expanding slightly) as the valves close.  If possible grab some 6 foot ones or similar and curl them around behind the machine.
Answer: Temporarily reduce the inlet pressure slightly by turning down the main inlet facet and see if the hammer goes away.


Code Coverage is including system.debug and comments too

Question: I have a weird situation, in one of my class code coverage is calculated including system.debug statements and comments too. Any clue what might be causing this?
Answer: Sometimes your code coverage will be out of sync with the source. The reason for this is unknown, but fixing it is usually as simple as using Clear Code Coverage and then using Run All Tests.
Comment: This has always frustrated me.  Didn't know there was a such an easy potential work around! :)
Answer: I've seen this happen before when you've run tests multiple times. I think it is getting confused with previous test runs. Try clearing out your test history and running the test again.
Setup > Develop > Apex Test Execution > View Test History and click 'Clear Test Results'
Comment: Thank you.. I cleared code coverage and the issue disappeared.


Design advice when daisy-chaining multiple JTAG devices

Question: I know that JTAG was specifically developed to support daisy-chaining of multiple devices, but I am still worried that my board will show up and I won't be able to talk to one of the devices because I did something wrong.  Does anyone have any advice on what the common pitfalls are or what precautions should be taken when implementing more than 1 device on the JTAG bus?
Comment: Look at voltage! Different voltages - like 3.3V and 5V - can not be chained. Some cores may have even lower voltage on JTAG lines.
Answer: 1) Don't put very many devices on one bus.
2) If there's something for which jtag is really critical, and you need jtag extensions for firmware download or debug, put it on it's own separate bus so electrical issues on a longer one don't force you to run at a painfully slow clock rate.
3) See (instead or as well) if you can design end-to-end functional tests that would detect assembly or component errors, thus including coverage of the parts that don't have jtag capability.


How to generate 3D clusters that are really only 2D?

Question: How to construct a data set in three dimensional space, containing five classes (small overlapping is allowed but not advisable) clusters  such that only two dimensions possess significant discriminatory power?
Comment: Questions solely about how software works are
[off-topic](https://stats.stackexchange.com/help/on-topic) here, but
you may have a real statistical question buried here.
You may want to edit your question to clarify the underlying
statistical issue. You may find that when you understand
the statistical concepts involved, the software-specific elements are
self-evident or at least easy to get from the documentation.
Answer: Does this count as a solution? I define the clusters by defining their centers in two dimensions and randomly make points around them. The third dimension consists of random numbers and therefore should not add anything to the cluster structure.
<code>#centers of 5 clusters

center.x <- c(10, 20, 30, 40, 50)
center.y <- c(10, 20, 30, 40, 50)

x <- c(replicate(50, jitter(center.x[1])),
       replicate(50, jitter(center.x[2])),
       replicate(50, jitter(center.x[3])),
       replicate(50, jitter(center.x[4])),
       replicate(50, jitter(center.x[5])))

y <- c(replicate(50, jitter(center.y[1])),
       replicate(50, jitter(center.y[2])),
       replicate(50, jitter(center.y[3])),
       replicate(50, jitter(center.y[4])),
       replicate(50, jitter(center.y[5])))

z <- runif(5*50)

collected <- data.frame(x=x, y=y, z=z)

# 2D plot x and y
plot(x,y)

# cluster structure in all 3 dimensions
plot(hclust(dist(collected)))

# cluster structure in x and y dimension
plot(hclust(dist(collected[,c("x", "y")])))
</code>
You could make the 5 cluster structure even more pronounced by choosing centers that are not in a row but on a circle of the x-y-plane.
Comment: Although implementation is often mixed w/ substantive content in questions,
we are supposed to be a site for providing information about statistics, machine
learning, etc, not code. It can be good to provide code as well, but please
elaborate your substantive answer in text for people who don't read this language
well enough to recognize & extract the answer from the code.
Comment: I was just about to add some more text, when I saw your comment. Added a short explanation.
Comment: @Bernhard Yes it did work out now I am trying to compute the Kmeans so that these cluster look even nicer. After that I want to compute **Fisher Score** Could you help me how I can compute the Fisher Score on clusters?


What is the percentage of energy recovery in Kinetic Energy Recovery Systems(KERS) in cars?

Question: Kinetic Energy Recovery Systems (KERS) use flywheels to recover energy from the kinetic motion of cars. They use a rotating flywheel that generates energy as it rotates- this  generates the electric energy for recovery. Do these systems use the flywheel on braking - so that when the car brakes the rotating flywheel generates electricity? Or, energy recovery is done even when the car is in motion, when the flywheel rotates with the cars' axles? Also, what is the percentage of energy recovery (ie, energy recovered from the flywheel vs the energy needed to rotate the flywheel - if this is a good measure!) of these mechanisms?
Comment: Two small clarifications of the text: 1) a KERS system can be *any* method to store the braking energy, not limited to a flywheel (could be a spring, compression of air etc.). 2) the flywheel doesn't *generate* but only *stores* energy while rotating. The rotation is proof that some energy is stored, but it has been *transfered* to the flywheel, and not *generated* within it.
Answer: KERS is also known as regenerative braking. It is a system found on the latest hybrid cars.
When a vehicle is stopping,  the brakes are applied to the discs to generate friction to slow it down. This generates heat which is energy that is lost or wasted.
Regenerative braking creates friction in addition to the brakes, however, some of the energy is converted to electrical potential instead of heat.
Using the flywheel at all times during the journey/race would result in lower performance. It is only used when braking, however, in KERS the energy is used as a boost in the straights in a race for example.
The efficiency would depend on the generator you were using. More coils, better efficiency however more coils means more weight. You need to find the balance.
Answer: KERS systems (as used in F1 and leMans cars) rely on a flywheel to store kinetic energy. When braking, the transmition is clutched trough a 1:n gearbox to the flywheel. That is, one full revolution of the car's wheels will make the flywheel turn n revolutions. The flywheel system will then store the energy until it is needed. Since a flywheel system rotates on ball-bearings and inside a vacuum, the losses are minimum. When the energy is needed on acceleration, there are two ways to use it:
a) Clutch the flywheel to the transmition whith a reduction gearbox n:1 and recover the stored kinetic energy directly
b) Store the flywheel energy as electricity in a  battery, by means of connecting the flyWheel to an alternator. That energy is then transformed back into kinectic by electric motors.
As for efficiency, of course the a) method is vastly more efficient, as there are no transformations. just losses as heat and drag through the gearboxes.
The b) method is as efficient as the alternator, the battery and the electric motor are. And also has the gearboxes in between.
In practical terms, for a racecar, that is either accelerating or braking hard, the a) method is preferred as more efficient and direct.
However, for a street car, it is different. As it is always stopping, the long term storage option of a battery is much more useful.
Nowadays, even racecars are forced, by rules, to rely exclusively on method b), as a means to further develop the technology for everyday use.


Is it OK to paint MDF only with primer?

Question: I am working on a garage cabinet project and I used MDF as the material. I used a white primer for 2 coats and I like the result already. Can I leave it like this or I have to paint it with a proper paint? Basically can primer be used as paint?
Comment: it's not as durable or easy to clean.  It's not going to protect your MDF from swelling if you get water on it.  So really it's up to what your expectations are of the cabinet.  I have bare MDF shelves in my garage.
Comment: I will paint a small piece and run tests on it (dirt, water,  scratch, etc) and decide later. I will update this post afterward.
Answer: There's no reason you can't leave it as-is as long as you are happy with the appearance.  A lot of new homes in large developments don't have much more on the walls either.
You could have also used a primer/paint combo product and not have to apply any additional layers at all.
Answer: I don't think you'll have a problem with it. I built cabinets using mdf from Home Depot recently and was soaking small pieces of it in water for days to see what would happen to it and nothing ever did.

Is it OK to paint MDF only with primer?


I want to log GPS coordinates using my Arduino and other sensors

Question: I want to log GPS coordinates using my Arduino and other sensors.  The difficulty is that I will be in forested conditions.  Would it be better to hardwire a gps device to the Arduino or to bluetooth an external device (a much more accurate, Trimble device that I know will give me the accuracy I need)?  Can such devices pair to a bluetooth receiver that would hardwire to my Arduino, or do I need an intermediate device like my iPhone?
Answer: The GPS receiver I've always used is the Adafruit Ultimate GPS:
https://www.adafruit.com/products/746?gclid=CjwKEAjwoPG8BRCSi5uu6d6N5WcSJABHzD8FmldH1vIvTt8h-S-948CJmDR6fR_BcF20UZMS4I-AEBoCt9Tw_wcB
It comes with a spot for you to plug in an external antenna (with an adapter), so you can get more precision and presumably be able to receive data much more accurately in places like forests. However, I'm not sure of this, so an external device may be the way to go. Bluetooth would be possible, but it can be a little difficult to set up with Arduino if you lack the experience, so if you can I'd go with a wired connection (with an RS232 or other).
Answer: What's more important, GPS accuracy or GPS yield?
Normally it's a trade off between the two. Under difficult conditions such as lots of trees a high accuracy GPS may simply stop outputting locations because it can't guarantee the accuracy, it depends upon the target market for the GPS.e.g. a survey system will be very conservative, survey people don't like getting bad data. A cheaper lower accuracy GPS won't care and will still output a position no matter how dubious the signals are.
I've seen $100 GPS units output a position when a $20,000 Trimble system refuses to.
The GPS should support NMEA output over bluetooth serial port protocol which would then work fine for a direct connection to your device however you may need to change the receiver settings to enable it, the default is likely to be outputting Trimbles binary TSIP format over the bluetooth.
Answer: If the Trimble has a serial output then you could connect via that.
If the Trimble supports the same Bluetooth spec as your receiver, then there is no reason why you shouldn't be able to connect them directly. You will need to find out if the Trimble is a Master, Slave or both and buy the appropriate receiver for you Arduino.


Proof of the Arzela-Ascoli theorem - where is the assumption that $X$ is compactly generated used?

Question: I'm learning the proof of the following version of Arzela-Ascoli's theorem (Willard, General Topology, page 287):
Let $X$ be a Hausdorff, or regular, k-space, $(Y,\mathcal D)$ a Hausdorff uniform space, and $\mathcal F$ a family of continuous functions from $X$ to $Y$. Then $\mathcal F$ is compact in the compact-open topology iff
(a) $\mathcal F$ is pointwise closed,
(b) for each $x\in X$, $\pi_x(\mathcal F)$ has compact closure,
(c) $\mathcal F$ is equicontinuous on each compact subset of $X$.
($\mathcal D$ is a diagonal uniformity on $Y$. k-space := a topological space in which a set is closed iff its intersection with any compact set is closed. $f:X\to Y$ is said to be equicontinuous on $K\subseteq X$ iff $f|_K$ is equicontinuous.)
$\\$
Here is a sketch of Willard's proof:
Necessity:
$\mathcal F$'s compactness in the compact-open topology implies its compactness in the pointwise topology, which is (under Hausdorff assumption on $Y$) equivalent to (a) and (b).
(c), i.e. equicontinuity of $\mathcal F$ on every compact $K\subseteq X$, is shown by taking, for a given $x\in K$ and $E\in\mathcal D$, a symmetric $D\in\mathcal D$ with $D\circ D\subseteq E$ and finding (using regularity of $K$ and continuity of functions in $\mathcal F$) a finite cover of $\mathcal F_K:=\{f|_K:f\in\mathcal F\}$ of the form
$$\left(U_{f_1}, D\left[f_1(x)\right]\right),\ldots,\left(U_{f_n}, D\left[f_n(x)\right]\right),$$
where
$$(K, U):=\left\{f\in\mathcal F_K:f(K)\subseteq U\right\},\qquad D[y]:=\{z:(y,z)\in D\},$$
$f_i$ are elements of $\mathcal F_K$, and $U_{f_i}$ are closed neighbourhoods of $x$ in $K$ with $f_i(U_{f_i})\subseteq D[f_i(x)]$. Then $f(U_{f_1}\cap\ldots\cap U_{f_n})\subseteq E([f(x)])$ for all $f\in \mathcal F_K$.
Sufficiency can be shown like this (I think):
It is enough to show that (c) forces the compact-open topology on $\mathcal F$ to reduce to the pointwise topology (for we can then use the upper-mentioned equivalence of conditions (a)-(b) and compactness in the pointwise topology), i.e. to show that the pointwise convergence of nets in $\mathcal F$ implies their convergence in the compact-open topology on $\mathcal F$ = the topology of uniform convergence on $\mathcal F$ (as these two coincide on spaces of continuous functions).
Indeed, if a net $(f_\lambda)$ in $\mathcal F$ converges pointwise to $f\in\mathcal F$, then for every compact $K$ in $X$
$f_\lambda|_K\to f|_K$ pointwise, but, as on equicontinuous families the pointwise topology concides with the topology of uniform convergence on compacta, (c) implies that $f_\lambda|_K\to f|_K$ uniformly, so $f_\lambda$ converges to $f$ in the topology of uniform convergence on compacta.
$\\$
My question is: in which part of the proof is the assumption that $X$ is a k-space used? I think the property "$f:X\to Y$ is continuous iff its restriction on any compact $K\subset X$ is continuous" is meant to be used somewhere, but I don't see where. I should also mention that it is entirely possible that there is a hole in the above sketch of proof, as it is not word-by-word transcribed from Willard.
Thank you for your help!
Comment: Hi, I was wondering if you came up with a solution to this problem. I'm reading Williard's book at the moment, and I was puzzled for the same reason you were.
Answer: You say $(f_{i})$ is a net in $\cal{F}$ converging pointwise to $f$. But $f$ does not belong necessarily to $\cal{F}$. So, the proof is going like this: If $K$ is compact, then $f_{i}|_{K}\rightarrow f|_{K}$ pointwise, so by equicontinuity of $\cal{F}$ on $K$, $f_{i}|_{K}\rightarrow f|_{K}$ in $\tau_{c}$, the compact-open topology. Thus, $f_{i}\rightarrow f$ in the topology of uniform convergence on compacta, by 43.6. Moreover, $(f_{i}|_{K})$ is an equicontinuous family, so $f|_{K}$ is continuous by 43.13 of Willard (: if $\cal{A}$ is equicontinuous family, the same holds for $\bar{\cal{A}}$). Thus, since $X$ is a k-space, $f$ is continuous. So, $\cal{F}\cup \{f\}$ is a family of continuous functions and by 43.7 (: for spaces of continuous functions the topology of compact convergence is the compact open topology $\tau_{c}$), $f_{i}\rightarrow f$ in $\tau_{c}$.