Write random data to random sectors of Hard Drive

Question: I have a hard drive that went bad and before I send it for RMA I want to wipe as much as possible from it. I tried using windows utilities and also did dd of /dev/random. The problem is I can't wait for either of those solutions to finish as the reason for the RMA is that it writes at a max speed of 1 MB/Sec. It will take over 140 hours just to do a single pass of the 500GB hard drive.
I have been looking for a utility (or even a bash script on linux) that would pick sectors at random and write random data (or even zeros) to those sectors. I'm hoping that if I run this for 24 hours, that will wipe approximately 80 GB of data. Since it will be randomly chosen, all the bigger files will be impossible to recover and the smaller ones will either be wiped, will be missing chunks, or will possibly be recoverable. This is unfortunately the optimal solution for me at this point.

SOLUTION

Thanks to "evildead" I was able to get a lot of the data on the drive to be randomly filled with junk from /dev/urandom. The bash script, in case someone ever needs it, is below:
#!/bin/bash
while (true); do
    randnum=${RANDOM}${RANDOM}
    number=$((10#$randnum%976773168))
    echo -e $number >> progress.txt
    dd if=/dev/urandom of=/dev/sdx1 skip=$number count=1
done

You will need to replace 976773168 with the number of blocks on your drive. I originally tried $RANDOM in bash, but it is only a 16-bit int and therefore is only 32k. As I needed a number that is over 900 Million I combined two $RANDOM values, so for instance if my random numbers are 22,861 and 11,111 I get 2,286,111,111 then fitting it to my block size I get a pretty random value in my range. It doesn't have perfect entropy but then again, what is really random on a computer? ;) The 10# is there in case the first random number is a 0, it forces bash to use base 10, not base 8, which is what it uses if it thinks the number is an octal (leading zero). I also write the random numbers to a file for analysis later to see what the spread was. If you don't need this you can just take out
echo -e $number >> progress.txt

and it will run fine. Also dont forget to replace sdx1 with the actual drive or partition you want to work on. I hope this is helpful for someone, I know it was really helpful for me.
Comment: Had this recently, so i improved Marcins solution a little ( automatic size detection , initial 8mb wipe .. etc ) https://github.com/benchonaut/disk-randomizer
Answer: you can use a shellscript in combination with dd.
e.g.
 while (true); do
   number=$((($RANDOM * 32768 + $RANDOM)))
   dd if=/dev/urandom of=/dev/sdx1 seek=$number count=1
 done

You only have to modify the number which is generatet from $RANDOM to fit to your Blocks.
EDIT 2016: Please note, the old solution was incorrect. Because we want to overwrite a byte at a random position in the output stream, we have to use seek instead of skip, as mentioned in the comments.
Comment: Did you ever hear of software that does this? Cause I haven't and it seems like at least some people would find it useful.
Comment: The accepted solution is plain wrong. You have to use "dd seek" instead of "dd skip", otherwise it make no sense: you will be writing to the same block while you need to write to random block. Here's an excerpt from documentation: > seek=n Seek n blocks from the beginning of the output before copying. On non-tape devices, an lseek(2) operation is used. Otherwise, existing blocks are read and the data discarded. If the user does not have read permission for the tape, it is positioned using the tape ioctl(2) function calls. If the seek operation is past the end of file, space from the current en
Comment: Worked great, I made some changes to the script for my situation, I added them to my question in case someone else had to use it in the future. Thanks for your help.
Comment: Hi, I never had such a problem and so I never heard about any software to solves such a problen. I knew dd can skip blocks and supports limit wirtes with count. So the only problem was to get a random number and putting everything together. :)
Comment: wow, it seems you are absolutely right. I will change that. Thank you.
Answer: My system isn't broken and writing from /dev/random is about 650Kb/s. Writing from /dev/urandom is 7Kb/s.
I've worked this issue for myself and for a purely bash solution you have to be realistic about what your goal is because unless your goal is to learn how to do it in bash because the point is doing it ALL in bash, the objective of wiping a disk securely is better accomplished in other ways though bash is useful in the solution.
evildead's bash algorythm to randomly select where dd will write to the drive works however, /dev/urandom is significantly slower than /dev/random and I think it is less secure to pepper the drive with chunks of random data and I think it will be harder to retrieve data from a wiped drive if you quickly do two passes writing one's then zeros to it.
For myself I wrote ones to the drive with:
tr '\0' '\377' < /dev/zero | pv > /dev/sdz

then I wrote zeros to the drive with:
pv < /dev/zero > /dev/sdz

Note the use of pv. It's a great program though you have to install it on most systems. Easy to find, doesn't seem to come with a man page but there is one on-line. It shows the progress of data passing through a pipe and with some programs you can even set it to stop after a certain amount of data is passed.
THEN purely because I was irritated that no bash solution would write random numbers as fast as either of those writes above would do, I wrote a small C program called svrandom to generate random numbers QUICKLY which I called from bash as follows:
while :; do ./svrandom; done  | pv > /dev/sdz

On my system that fills drive /dev/sdz with random numbers as fast as writing /dev/zero does. Goal of wiping drive with random numbers achieved, and bash was used in the solution.
/* svrandom
a small program from shadowvision.com September 2015 to generate strings of random numbers.
the default setting of 10000000 gives about 453 megs, adjust as
needed */

#include <stdio.h>
#include <stdlib.h>
int main()
{
int r,a,b;
 /* adjust following number as needed to suit your string length */
 for(a=0;a<10000000;a++)
 {
 for(b=0;b<5;b++)
 {
 r=rand();
 printf("%d",r);
 }

 }
 return(0);
}

In the source I mention that you can control the length of the random number string by adjusting a number. If you call it from a while loop you wont have to although there may be a carriage return at the end of each loop, but if you want one continuous string from just the program you will have to change the number.
snip!
looking around I found this solution on https://superuser.com/questions/19326/how-to-wipe-free-disk-space-in-linux
openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero |pv > /dev/sd{x}
and like the guy there said, I can't believe how fast this is. It's writing to the disk faster than I thought the disks maximum write speed was.
It looks to me that this command is using openssl to encrypt a string of zeros from /dev/zero using /dev/urandom as the seed but im not 100% sure.
my solution with the C program to write random numbers writes to my system at 82mbps, this openssl solution is writing at 100mbps.  wow!
Comment: I think your system is a bit broken if `/dev/urandom` can only supply data at 7kB/s.  On my system it gives a nice, steady 4.8MB/s, according to `pv`.  But +1 from me for a nice first answer - welcome to Server Fault!  (And I hope you'll forgive me for marking it up a bit.)
Answer: A Python 3 script, with:

awareness of block device size
verification of block device size, by writing bytes at the boundaries
guarantee of first and last byte destroy
random data overwrites
random offset
random batch sizes

If you need Python 2 then it should be easy to covert by using "".format() instead of f"" strings.
import subprocess
import random

REPS = 100
CRASH_ON_FIRST_FAILURE = True  # change to False if the device is not reliable

def run(*cmd, assert_returncode=None, print_stderr=False):
    completed = subprocess.run(cmd, capture_output=True)
    if assert_returncode:
        assert completed.returncode == assert_returncode, f"Unexpected return code (got={assert_returncode}, expected={assert_returncode})"
    if print_stderr:
        print(str(completed.stderr))
    return completed.stdout

def part_size(part):
    """
    Partition size in bytes
    """
    return int(run("blockdev", "--getsize64", part))

def destroy_block(part, bs, seek, assert_returncode=None, print_result=False):
    run(
        "dd", f"bs={bs}", "if=/dev/urandom", f"of={part}", f"seek={seek}", "count=1",
        assert_returncode=assert_returncode
    )
    if print_result:
        print(f"Destroyed bs={bs} of={part} seek={seek}")

def destory_random_block(part, part_size, bs):
    """
    bs=1 destroys bytes sized block
    bs=1024 destroys KB sized block
    etc.
    """
    seek_max = int(part_size / bs)
    seek = random.randint(0, seek_max)
    if CRASH_ON_FIRST_FAILURE:
        assert_returncode = 0
    else:
        assert_returncode = None
    destroy_block(part, bs=bs, seek=seek, assert_returncode=assert_returncode)

def destroy(part):
    """
    part - partition to be destroyed
    """
    s = part_size(part)
    destroy_block(part, bs=1, seek=s, assert_returncode=1)  # "test" destroying 1 byte at size boundary, should fail
    destroy_block(part, bs=1, seek=(s - 1), assert_returncode=0, print_result=True)  # "test" destroying 1 bytes before boundary, should pass
    destroy_block(part, bs=1, seek=0, assert_returncode=0, print_result=True)  # "test" destroying first 1 byte
    while True:
        print(f"Destroying {REPS} byte-sized blocks")
        for _ in range(REPS):
            destory_random_block(part, part_size=s, bs=1)
        print(f"Destroying {REPS} KB-sized blocks")
        for _ in range(REPS):
            destory_random_block(part, part_size=s, bs=1024)
        print(f"Destroying {REPS} MB-sized blocks")
        for _ in range(REPS):
            destory_random_block(part, part_size=s, bs=(1024 * 1024))
        rand_size = random.randint(1, 1024 * 1024)
        print(f"Destroying {REPS} {rand_size}-sized blocks")
        for _ in range(REPS):
            destory_random_block(part, part_size=s, bs=rand_size)
        print("\nRise and repeat\n\n")

destroy("/dev/sdb1")

Example output:
# "test" destroying 1 byte at size boundary, should fail
# "test" destroying 1 bytes before boundary, should pass
Destroyed bs=1 of=/dev/sdb1 seek=10736369663
# "test" destroying first 1 byte
Destroyed bs=1 of=/dev/sdb1 seek=0

Destroying 100 byte-sized blocks
Destroying 100 KB-sized blocks
Destroying 100 MB-sized blocks
Destroying 100 875091-sized blocks

Rise and repeat

Destroying 100 byte-sized blocks
Destroying 100 KB-sized blocks
Destroying 100 MB-sized blocks
Destroying 100 1028370-sized blocks

Rise and repeat

To run:

Copy the source code all but the last line
On the victim host log-in as root
On the victim host run: python3
Paste the code from step 1
Type destroy("/dev/XYZ") and return key twice
After a few hours, hit Ctrl-c

NOTE: "/dev/XYZ" is the partition name that will lose data.
WARNING: Watch out for Fat-finger errors. The data will be gone forever so double-check what partition you are typing in.
WARNING: Running this script for days in cloud services may cost if you are charged for disk write IOPs.
Answer: If you have a powerful magnet, you can physically wipe it.
I take apart old, dead hard drives and get the voice coil magnets from the head positioning assembly.  These are rare earth magnets, and from a typical 3.5" drive they're powerful enough to erase enough of the servo tracks on a hard drive so it'll be completely unusable.
One way or another, get the magnet, wipe it back and forth over the top cover of the drive and in less than a minute you'll have a dead drive.  Since it's being RMA'd anyway, they shouldn't care.
Answer: The best software that will automate this process is Darik's Boot and Nuke (aka DBAN)
It's a boot CD with a comprehensive range of drive-wiping mechanisms, ranging in aggresiveness (And time taken to wipe the drive)
It is available from http://www.dban.org/
Answer: Thanks for the solution on top!
Since I am not allowed to comment the answer with the script I have to add my tiny enhancement myself.
I just tuned it a little for more convenience so that one only has to define the device. fdisk will deliver the number of sectors:
#!/bin/bash
# add device to mangle
# example:
# disk=/dev/sde
disk=
# get number of sectors
sectors=$(fdisk $disk -l| head -1| awk '{print $7}')
# run
while (true); do
    randnum=${RANDOM}${RANDOM}${RANDOM}
    number=$((10#$randnum%$sectors))
    echo -e $number >> progress.txt
    dd if=/dev/urandom of=$disk seek=$number count=1
done

Answer: The dban option is the correct answer to this problem.
Writing "random" data from /dev/random will run into problems on most systems because you'll exhaust the entropy on the system and wind up writing the data very slowly.
Almost as "random" as from /dev/random is to encrypt a string of zeros with a random passphrase.  You can use dd and openssl to do this, and it allows you to nuke the disks as fast as the system is able to write to disk (instead of waiting on /dev/random).  You'll be able to run this 3-5 times in the same time you'd run something that's just dding from /dev/random.
dd if=/dev/zero | openssl des > /dev/hda13

Comment: I don't understand how people constantly ignore part of the question on here. I am writing at 1 MB/Sec because the drive is broken. /dev/urandom provides data at a higher rate than that. dban is not the correct answer because it doesn't write to random sectors, it writes linearly sector by sector. It might write random DATA but it doesn't write it in a random ORDER.
Comment: you can use /dev/urandom or /dev/zero, doesn´t really matter. Only thing not to use is definatly /dev/random, cause it blocks if entropy is exhausted.


Generating DDL from entity class in intellij

Question: Could you please let me know whether we have an option to generate ddl from entity class in intellij?
I could do the entity class generation from the table? I am looking to change the definition of a entity class and i need to re-generate the DDL for table?
I am using eclipse links for JPA.
Answer: I would recommend to use eclipse link (not intellij) to create the ddl script.
With the ddl-generation.output-mode in persistence.xml you can instruct eclipse link to create an ddl script.
<persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence persistence_2_0.xsd" version="2.0">
    <persistence-unit name="test" transaction-type="RESOURCE_LOCAL">
        <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
        <properties>
...
            <property name="eclipselink.ddl-generation.output-mode" value="both"/>

            <property name="eclipselink.ddl-generation" value="drop-and-create-tables"/>
            <property name="eclipselink.create-ddl-jdbc-file-name" value="createDDL_ddlGeneration.jdbc"/>
            <property name="eclipselink.drop-ddl-jdbc-file-name" value="dropDDL_ddlGeneration.jdbc"/>

        </properties>
    </persistence-unit>
</persistence>

@see:

http://eclipse.org/eclipselink/documentation/2.4/jpa/extensions/p_ddl_generation_output_mode.htm
http://eclipse.org/eclipselink/documentation/2.4/jpa/extensions/p_ddl_generation.htm
Comment: I couldn't  find any generated files. Links to Eclipse are useless. Can you please clarify step-by-step ?
Comment: @alery: first of all it is "Eclipse Link" not "Eclipse"! - But there are much more deteils needed to help you, so please write a new question  (that describe your problem and what you have done so far, as well as the name and version of JPA Provider you use (Hibernate/EclipseLink./...) - If you want me to notice the question, then add an comment below this one that point at your question.


Difficult Integration Question (Reverse Chain Rule)

Question: My friend gave me this integration question, after after a while on it, I am not sure how to go about solving it:
$$\int\frac{dx}{x(\ln{x} + 1)^2}$$
Any suggests are welcome, I'm really lost at this point. Reverse Quotient Rule?
Comment: Try a substitution: $u = \ln x + 1$ with $du/dx = 1/x$.
Answer: Let $u = \ln x \Rightarrow du = \dfrac{1}{x}dx \Rightarrow I = \displaystyle \int \dfrac{du}{(1+u)^2}= -\dfrac{1}{1+u}+C=-\dfrac{1}{1+\ln x}+C$


Interpretation of variable in a probit model

Question: I am struggling interpreting the coefficient of a variable which is expressed as a proportion in a probit model. As it currently stands , I am interpreting the average partial effects of this variable (0.14) as a one unit increase in share of debt raises the probability of innovation by 14 percentage points.
However, if I multiply this variable by 100 and turn it into a percentage, the average average partial effect is (.0014556 ), does it mean that a 1 percentage point change in the percentage of debt increases the probability of innovation by 0.0014556.
Questions : a) are my interpretations correct
b) Instead of saying a one percentage change increases can I write in terms of a 10 percentage point increase and if so what should I do .
Regards,
naveed
Answer: if 0.14 is your coefficient, then you should exponentiate it [exp(0.14) = 1.15], which says that one unit increase of whatever your original variable, you will get 0.15 (1.15-1=0.15) or 15% increase in your response.
Comment: This makes sense in a logit model for multiplicative effects on the odd ratio (which is not the same as the probability of success), but is not meaningful for a probit model. Sometimes people will multiply the probit coefficient by 1.6 or 1.8 to get approximate logit coefficients, and then exponentiate, but this is may not work well in the tails (near 0 or 1).


Best practices for Moneydance extension development

Question: Are there any best practices when developing Java-based extensions for Moneydance?
Comment: This sounds like a good question for the moneydance support/forum http://help.infinitekind.com/
Answer: As the founder and main developer of Moneydance I can admit to using several different environments to write extension, from emacs+terminal+ant to Xcode, eclipse and (currently) IntelliJ's IDEA.
The only thing that I'd really recommend is to use ANT.  That is mainly because it makes my life easier when you submit an extension for review and to be signed.  The sample extension available from moneydance.com/developer also provides a sample ant build file.  Other than ANT, just use your favorite java environment.


Recording digital piano sound

Question: I want to record (sound, not MIDI) from my Yamaha P115. I am just wondering what is the optimal way.
I have tried doing it simply with my phone, without a cable, and as expected it sucks. I live in noisy Kolkata, making the ambient noise impossible during the day. The phone (OnePlus 3) mic cracks even when it is relatively quiet.
So I was thinking of connecting it with cable to the phone. Or connecting the piano to a PC and record through a software (it seems Audacity will serve the purpose). But then someone said "audio interface". I looked into the stuff and got daunted by TRS, XLR & RCA. P115 has two 1/8" headphone o/p and two Aux Out (L & L/R). There's also USB to Host, but I guess that's for MIDI.
Given this, what are my setup options?
(I feel this isn't a perfect question. So, apologies.)
Comment: What level of audio quality are you looking for?
Comment: Start with your PC and Audacity. Just connect the audio out from the keyboard to the "aux" audio input on the PC (not the "mic" input). If that doesn't give you good enough quality, *then* consider spending money on something more "optimal".
Comment: alephzero, I don't have an aux input on my laptop, just a mic input. and what do you mean by "audio out" from the keyboard - the 1/8" headphone out, or the "aux out" (which I believe uses the RCA cable)?
And if latter, should I use only the Mono (L/R), or both L & R,, in which case what kind of cable/ jack required?

ScottM, I haven't thought much about quality. I am no performer. But a late beginner. Will record for my own listening for improvement. What do YOU suggest?
Comment: realised that my laptop does not have mic in :-((
what do I do now? :-0
Answer: My suggestions here, slightly shortened discussion.
Electrical pianos sound a lot better in stereo than in mono in my experience. So, when possible try to record both left and right signal. More about that later.
The piano has aux out-s that are perfect for recording. And you possibly have a sound input on your computer. This is wastly better than using the microphone in a noise surrounding.
So, possibly, you could get a cable and be done with it. But there are actually two problems (or even more) with that approach.
.1 The computer most probably wants a microphone level signal (read very weak). The piano outputs line level signal on the aux out (read medium strong). I have never tried doing this kind of thing, so I cannot really tell, but I would expect the result using only a cable to be mostly distorsion.
.2 The computer most probably only has a mono input (one signal), and as I said, the piano sound better using stereo (ie two channels).
My suggestion is to get a really cheap USB audio interface. It should cost you about 30 to 50 USD or so for an interface with two instrument inputs ( 1/4 inch ). You will need a usb cable and two 1/4 mono cables. We tend not to recommend specific products here, but a quick search on internet should give you a lot to choose from, simply pick one. Even better, when possible, visit your local friendly music shop and spend a few dollars extra and getting help.
Comment: Check the manual. The UM2 has two line in. Line 1 in is in the middle of the left kombination contact.  Line 2 has its own contact. Use two mono cables, 1/4 inch*  mono contacts at both ends. Connect from Aux L and Aux R on the piano to Line 1 and Line 2 on the UM2.   *) 1/4 inch ( quarter inch ) mono cables is the same type of cable used for electrical guitars. No RCA contacts here.
Comment: Right. Then two cables, both ends 1/4 inch (6,3 mm) mono between the UM2 and the piano.
Comment: Thank you gheliquist!  I was looking at Behringer UM2. Not sure where can I fit two RCA cables (from the L&R Aux Out) in that one. Looks like it has only one XLR line 1. Or should I take the 1/8" headphone out? Would appreciate any advice.
Comment: It has the following inputs: 1x XLR/TRS combo jack mic/line input
1x 1/4" (6.3 mm) instrument input
Comment: realised that my laptop does not have mic in :-(( what do I do now? :-0
Comment: The UM2 will work perfectly.
Comment: So I take the UM2 USB output to one of the laptop USB ports?
Comment: I have recorded with my PC's combination jack set to "line in" (i.e. "mic in", but stereo) and got full range recordings that passed a blind A/B test with other producers.  It's worth trying the mic in on your PC if there is one.

Recording digital piano sound


Saving the file 'web.config' requires you to stop your debug session error

Question: I'm trying to make changes to web.config after running the application. It gives the following error.

Saving the file 'web.config' requires you to stop your debug session.
  Would you like to stop debugging now?

I'm using Visual Studio 2012.

I thought it has to do something with Enable Edit and Continue feature in visual studio. Still doesn't work.
Comment: @WeTTTT - we recently switched to VS2012 from VS2010. I used to be able to change web.config without this error.
Comment: I was wrong. Both VS2012 and VS2010 are able to keep debugger attached when directly save web.config as I tested on VS2010 & VS2012 on IIS7. Are you using IIS7/7.5/8 or IIS Express for debugging?
Comment: That's quite strange. I cannot recreate this here. Is it possible that you(or someone else) was holding a breakpoint when you were trying to save the web.config?
Comment: @WeTTTT - No. There aren't any breakpoints.
Comment: Are you sure you are using IIS and visual studio internal development web server?
Comment: Why did you need that? I'm just curious
Comment: @Kuzgun - we use web.config to add users to our internal Web Application (not Website) manually by using custom tags ``. I used to be able to add/edit/delete them without stopping debug mode.
Comment: I was having the exact same issue after upgrading to VS2013, however disabling `Enable Edit and Continue` **did** do the trick
Comment: Different option is to edit the file outside of Visual Studio. Really no reason at all it does this.
Answer: I just want to add something to the comment that janv8000 made about Edit and Continue, because I've just been having the same problem and managed to sort it.
To give some background, I had an existing web application, created in VS 2012. When I debug this project in VS 2013, I can edit the web.config file happily without restarting the debugger. I then created a new web application in VS 2013, and found that I got the error message in the original question when I tried to edit the web.config file.
The solution to this was to disable Edit and Continue for the individual project. The reason I highlight this is because there's a global debugger setting in Tools->Options->Debugger for Edit and Continue. This is not the option that causes this problem.
To solve it:

Go to the project preferences for your web application (right click on it in the solution explorer and go to "Properties").
Go to the "Web" tab.
At the bottom of this tab is a section called "Debuggers".
Uncheck "Enable Edit and Continue".
Answer: First off let me point out a quick fact. There are two different types of projects here.

A WebSite which is created by select New WebSite from the Visual Studio menu.
A WebApplication which is created by selecting New Project and selecting ASP.Net Web Forms Appplication

Now the reason to point this is out is simple.
The first option allows you to change your web.config without performing an Application reset (only while Debugging). This is because the WebSite project files are compiled as the pages \ files are requested. Therefore everytime you request a page the code is compiled. This is also why you can change code files while debugging with no problems.
The second type is a compiled application. Therefore all the code is compiled before runtime. In this scenario the AppPool (IIS, IISExpress) watches for changes to this file. If changes happen to this file the application MUST restart in order for changes to take affect. Now there is no way to stop this behavior as its part of the core AppPool.
Here is an excerpt from MSDN:

Configuration Changes Cause a Restart of the Application Domain
Changes to configuration settings in Web.config files indirectly cause
  the application domain to restart. This behavior occurs by design. You
  can optionally use the configSource attribute to reference external
  configuration files that do not cause a restart when a change is made.
  For more information, see configSource in General Attributes Inherited
  by Section Elements.
Attempts to change a configuration file by
  someone who does not have permission to edit the file will not cause a
  restart of the application domain.

Full MSDN page. http://msdn.microsoft.com/en-us/library/ackhksh7.aspx
As it states here there is one way around it. By using the configSource to reference external configuration files which you may change without a problem.
I hope this helps.
Comment: While i understand the explanation, my question is: Why this behaviour started only AFTER switching from VS2012 to VS2013? I was able to edit the web.config when using VS2012 without the need to restart the dubugging session.
Comment: We still don't have the real explanation of why this behaviour have changed in VS2013. I didnt had this problem with VS2008, VS2010 and VS2012!
Comment: thanks for your answer. Ours is a web application. I'm guessing there are certain config values which can be edited without restarting app domain but some are not(may be I'm completely wrong !!)
Comment: I don't see how this is possible.. Not a single test across multiple VS versions and projects have yielded the result. No parts of the web.config may be changed with out an app restart.
Comment: Yes, I noticed this issue too after I switched from VS2012 to VS2013. I have two projects in a solution: a ASP.NET MVC application (created in VS2012) and ASP.NET Web API application (created in VS2013). For some reason, when I'm debugging the MVC app, Visual Studio does not complain if I edit web.config file for the MVC app, but it complains if do the same for Web API app while debugging it.
Comment: –1 this post's conclusion is false. using external source configs just means the host doesn't know it *should* restart. You can edit the file outside of VS and it can restart with the debugger attached, without issue.
Answer: Actually I had the same problem on a Web Application using Visual Studio 2013. Before then I was using VS2010 and had no problem as you mentioned.
The solution I found is not to use IIS Express (: (I couldn't figure a way with using IIS Express)
1- Just right click on your project and select Properties.
2- On the "Web" tab, choose "Local IIS" under "Servers" section on the page.
3- Click on "Create Virtual Directory" button.
Viola, now you've deployed it on iis with port 80. (So it will be under "Default Web Site" section when you run inetmgr (IIS Manager), of course if you dont have any other site running on port 80)
If you want a custom port (i.e. 1234):
Just create new site on IIS Manager with the port 1234, then edit your url in project property in VS2013 as: "http://localhost:1234/Your.Project.Name"
If anyone shows how to edit and continue web.config using IIS Express, it will also be appreciated.
Comment: Take a look at the answer I just added if this is still an issue for you. I'm using IIS Express and can edit the web.config file now that I've disabled Edit and Continue.
Answer: Got the same error in VS 2019. the following fixed it for me:
Go to the Debugging Browsers list (as in the screen shot), open the dropdown > script debugging, and disable it:


Should i use an index on a column with limited number of values?

Question: Reading this I now understand when to use indexes and when not to use them. But i have a question; would using an index on a column with a limited number of possible values help speedup queries (SELECT-ing) ? Consider the following:
Table "companies": id, district_id, name
Table "districts": id, name

The number of districts would never pass 5 entries. Should i use an index on companies.district_id then or not? I read somewhere (can't find the link :( ) that it wont help since the values are not that many and it would actually slow down the query in many cases.
PS: both tables are MyISAM
Comment: Good indexes helps a lot to speed up queries. Bad designed indexes bloats the DB and can slow down it. In your case, since the table just gets 5 records it ill probably not impact performance in any measurable way. In this case better not to create the index (from the maintenace point of view).
Comment: I thought so. Thanks
Answer: Almost never is an INDEX on a low-cardinality column used by the optimizer.
On the other hand, a "compound index" may be useful.  For example, does INDEX(district_id, name) have any use?
Having INDEX(district_id) will slow down INSERTs because the index is added to whenever a row is inserted.  It will not slow down SELECTs, other than the minor amount of time for the Optimizer to notice the index and reject it.
(My statements apply to both MyISAM and InnoDB.)
More discussion of this answer:
MySQL: Building the best INDEX for a given SELECT: Flags and Low Cardinality


What does \0 mean in sed?

Question: I have the following code:
echo "12. Chapter Name" | sed -n -E "s/([0-9]{2})\.[[:space:]].*/\1/p"

It prints 12 as expected, since \1 refers to the first capturing group.
However, if \0 is used instead of \1, the output is 12. Chapter Name, the entire input string is printed.
It seems that as long as the regex found a match, \0 prints the entire input string. Is this correct?
I'm running Debian 10.2.
Comment: I think you have `sed -n -E "s/([0-9]{2})\.[[:space:]].*/\1/p"`
Comment: `\0` or (`$0` for the tools that use `$n` to identify the capturing groups) is the string that matches the entire `regex`. Not only in `sed`.
Comment: @Wiktor Stribiżew Indeed I mistyped. Thanks for pointing that out.
Answer: The \0 is a placeholder for the whole match value.
Note that capturing group indices start with 1 and go up until 9 in POSIX regex patterns. The zeroth group is the whole match.
Also, instead of \0, it is more common - and probably more portable - to use & in sed.
echo "12. Chapter Name" | sed -n -E "s/([0-9]{2})\.[[:space:]].*/\0/p"

is equal to
echo "12. Chapter Name" | sed -n -E "s/([0-9]{2})\.[[:space:]].*/&/p"

You may read more about & at Using & as the matched string.

What does \0 mean in sed?


How do I post an input whitout a form?

Question: So, I'm using Bootstrap as my framework, and I don't like what happens with my column if I put it in a <form> tag. So, can I post my input to a php file without a form?
HTML
<form role="form-horizontal" id="usr_reg_in" name="usr_reg_in" action="php/usr_time_reg.php" method="post">
    <input type="text" class="form-control" id="reg_in" name="reg_in" placeholder="Var god ange anst. nummer">

            <button class="btn btn-info" type="submit">In</button>

</form>

Comment: you can with ajax yes, you'd need to do a http post request in javascript. OR use formdata
Comment: So that's the only way? i can't use like data-target in the input, or grab the values with the button or something else fancy?
Comment: "grab the values with the button" then what? you'll still need a http request or use the formdata api
Comment: As the form is posting to another page, when is your form getting messed up? is it being redirected back?
Comment: *What* happens with your column? Please post code that actually demonstrates the problem.
Comment: There is no really problem. It´s just a question. Can i some how post the input without a form tag...
Answer: The short answer is yes. For example, you could use jQuery's Post method.
But form tags are here for a reason, and they should be used accordingly. Layout shouldn't be an issue for dropping a form.
Ask a question about the form breaking your layout here in SO. I'm sure it's quite simple to fix it.

How do I post an input whitout a form?


ggplot2: nonlinearly divide scale into equally-wide sections

Question: I am trying to reproduce this style of plot (Bayesian sequential analysis), but I can't manage to get the y axis right. This is some simple example data:
library(ggplot2)
testdata <- data.frame(logbf=c(1/1000,1/300,1/100,1/30,1/10,1/3,1,3,10,30,100,300,1000),n=1:13)
ggplot(testdata,aes(n,logbf)) + geom_line()

The goal is to get this to plot as a straight line. To do this, I need to divide the y-axis into 'sections' ranging from 1 to 3, 3 to 10, etc. I thought that this could be implemented by using scale_y_continuous(trans=X) where X is a custom nonlinear scale, as done here.
The problem is I can't get the scaling algorithm right. Using the value 38 as an example, I thought of finding the lowest matching section break (30), and decomposing each value into an integer part representing the section number (e.g. 3, counting sections from -5 to 5) and a fractional part representing the distance to the next section (8/(100-30) = 0.1143), resulting in a scaled value of 3.1143.
I do not know if this algorithm is correct, but when I tried to code it the results I got were not as intended. The bigger question is: isn't there an easier way to achieve this?
Comment: Maybe I don't know enough about the subject matter, but can you spell out the rule for creating this sequence?
Comment: I believe that these values go back to an old paper by Jeffreys which proposes guidelines for interpreting magnitudes of Bayes Factors (which are basically likelihood ratios), but I am not aware of any rule for generating this exact sequence, other than that it starts out from a base-10 logarithm. Specifically, I have no idea where the (to me) non-obvious breaks at 3, 30, and 300 (and their inverses) come from. That's why I feared this would be non-trivial to code. The accepted answer shows I was wrong :)
Answer: The original example looks like a log scale to me, with smaller distances between 10 and 30 (x3) than between 30 and 100 (x3.33). This can be replicated using scale_y_log10 (or scale_y_continuous(trans = "log10") and then specifying custom breaks and labels for those breaks:
library(ggplot2)
testdata <- data.frame(logbf=c(1/1000,1/300,1/100,1/30,1/10,1/3,1,3,10,30,100,300,1000),n=1:13)
break_labels <- c("1/1000","1/300","1/100","1/30",
                  "1/10","1/3","1","3","10",
                  "30","100","300","1000")
ggplot(testdata,aes(n,logbf)) + geom_line() +
  scale_y_log10(breaks = testdata$logbf,
                labels = break_labels)


Why date field is picking the user's timezone?

Question: I have a date field with default value set to now. I have selected the site's timezone in the date field settings.
So for every user the date field value should be according to the site's timezone. But on node forms I can see the date field value is different for every user according to the timezone set on that user profile form.
Why the field value is not according to the site's timezone. it should be same for all the users.
If I disable this settings Users may set their own time zone in the regional settings then every thing works fine.
Is there any configuration I'm missing?
Answer: In Drupal, dates stored in the DB do not have any timezone fixed to them; it presumes all date fields are consistent in timezone and relies on the system's timezone setting to offset everything to its correct date for displaying.
There are two different offset settings: the field's timezone settings (which adjust from user's timezone input to the site's timezone) & the user's timezone setting which adjust display from the site's timezone to the user's timezone.
Needless to say, the various offset settings can make for a confusing situation to adjust to your needs.
If you want to keep the field adjusted to the site's settings and have the users still adjust for their own time zone, you should customize the display of your field so the time zone is presented in an unambiguous format.
Comment: I want the input should be same for all users according to the timezone set for the site. I did not want the input should vary according to the user's timezone.
Comment: What you're asking is kind of vague, but the main reply gets to the heart of the problem (what is being offset and where). I updated the conclusion bas on what you said; if you look at your DB, you'll see all dates are fixed when setting the field to the site's timezone. The issue is how to display the offsetting.
Comment: My requirements are exactly the same according to your answer's last paragraph. so you mean we should just play with the display in the input?


Normalize infinite range into finite one

Question: First of all; I'm a programmer, not a mathematician so please excuse the informality of my math-vocabulary.
I have a series of slopes, calculated out of random angles (their tangents). These angles will always be located in the I and IV quadrants i.e. there are no angles "pointing to the left".
As you probably know, the tangent curve goes from -infinity to +infinity in this range.
I need to normalize the slopes into a range from -1 to 1. This means that the slope of an angle approaching (pi/2) should get closer and closer to 1 instead of infinity.
I would like to know how to do this in two ways:
1) Preserving a smaller tangent curve in my new range
2) Making it a linear range
Comment: It isn't clear to me what exactly you want to do. Why not just normalize the angles to be from -1 to 1?
Answer: I'm not sure if this is what you're looking for, but you could first normalize your point so that it is at distance $1$ from the origin, then take the resulting $y$-coördinate.
In other words, given $(x,y)$, you would be calculating $\frac{y}{\sqrt{x^2+y^2}}$, which is the sine of the angle from the $x$-axis.
Comment: To put it another way, if you are not given the point $(x,y)$ but only the tangent $t$, you can calculate $\frac{t}{\sqrt{t^2+1}}$.
Answer: Normalisation as such implies linearity.  If you want to include tan(90) you will simply see values of unity at +/-90 and zero elsewhere.
You can perform a non-linear function on tan (generally called companding rather than normalisation), but unless the non-linear function relates in some way to the desired solution you may just as well use the original angle [same as atan(tan(theta))].


Access CPUIDs (cpu serial numbers) WITHOUT using WMI using VB.net or C#

Question:
Possible Duplicate:
x86/x64 CPUID in C#

I need to retrieve CPU's serial numbers. When using WMI, it is called CPUID.
My goal is to access that information WITHOUT using WMI.
Unmanaged is code is fine
I really need some sample code and preferably .NET implementation

Comment: Why did you tag C++ if you want to use VB.net or C#?
Comment: that will be the last resort option:)
Answer: You can get a CPU's CPUID with the __cpuid() function from <intrin.h>. Is that what you're trying to do?
Comment: do you have sample code? And possibly .NET implementation? That's what I am looking for


INNER Tag Avoid, Android webservices

Question: i have one webservices(in PHP) i need to parse it so simply i need tag name. so when i viewsource it in browser i got ... some span tag are there and it stuch to display data so how can i avoid this?
<weeklyparties>

<status>
<weeklypartyid>13</weeklypartyid>
<weeklypartytitle>Thursdays</weeklypartytitle>
<weeklypartydesc>Rare Rooftop - Chelsea
</weeklypartydesc></status><status><weeklypartyid>14</weeklypartyid><weeklypartytitle>Fridays</weeklypartytitle><weeklypartydesc>Bar Basque
Forum
Ganzevort Park
Hudson Terrace

</weeklypartydesc></status><status><weeklypartyid>15</weeklypartyid><weeklypartytitle>Saturdays</weeklypartytitle><weeklypartydesc>Aspen Social Club
Bar Basque
Crimson
Forum
Lexicon

Lounge 16 (New Highbar)
Hudson Bar
Hudson Terrace
<span style=\"font-family: Tahoma; color:

Comment: Do you control the web service?
Comment: no its already build and may be if i am send request then i can., but fist priority at my side.
Answer: I asume you want to develop a Java or Android application. In this case you can use a third party library for HTML parsing,like

jsoup (see http://jsoup.org/)
HTML Parser (see http://htmlparser.sourceforge.net/)

To find all the weeklypartydesc with jsoup, you would use the following code fragment:
Document doc = Jsoup.connect("http://your.url/").get();
Elements partyDesc = doc.select("weeklypartydesc");

for (Element entry : partyDesc)
{
    // all the weeklypartydesc as string
    Log.i(entry.text().toString());
}
This code also removes the br and span elements. You can easily find additional infos on using jsoup on StackOverflow (even in combination with Android). Do not forget to give your application the internet permission.
Answer: You have not written in your question which tag name you need. I assume you're looking for <weeklypartydesc> and you want to get the plain text out of it there.
Additionally as you use the view source from your browser, I assume that there is some CDATA element you don't see (as otherwise the XML would just be invalid).
You can do this with DOMDocument. First obtain the element you're interested in, then convert the HTML data inside that element into plain-text by stripping tags:
# obtain the element in question
$doc = new DOMDocument();
$doc->loadXML($xml);
$xp = new DOMXPath($doc);

foreach($xp->query('/weeklyparties/details/status/weeklypartydesc') as $weeklypartydesc)
{
    # obtain the HTML inside that element and convert it into plain text
    $text = strip_tags($weeklypartydesc->nodeValue); # Rare Rooftop - Chelsea
}

Demo
If strip_tags does not fit your needs, you can create a DOMDocument out of the HTML fragment and use XPath to obtain all text nodes which will give you the HTML as plain text as well.

INNER Tag Avoid, Android webservices


How to monitor Segate HDD health under Linux?

Question: I'd need to monitor the health of several Seagate ST16000NM002G SAS HDDs hosted in a disk server running CentOS 7. As far as I understand, Seagate disks do not expose S.M.A.R.T. attributes due to a precise management decision (see this page), and the company suggests to use their SeaTool software which according to them is more reliable than S.M.A.R.T. Sadly, it seems that only the SSD version of SeaTool is available for Linux (see this page).
Since I'd say that Segate+Linux should be a fairly common case in modern data centers I'm pretty sure that some reliable monitoring tool for Seagate disks must be available for Linux. Can anybody provide some insight, please?
Edit: this is what I get with smartctl for the Seagate disks:
$ sudo smartctl -A /dev/sda
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1160.53.1.el7.x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
Current Drive Temperature:     33 C
Drive Trip Temperature:        60 C

Manufactured in week 42 of year 2020
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  20
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  3324
Elements in grown defect list: 0

while for a Toshiba HDD on another machine:
$ sudo smartctl -A /dev/sdb
smartctl 7.1 2020-04-05 r5049 [x86_64-linux-4.18.0-348.12.2.el8_5.x86_64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       7019
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       34
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   050    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   050    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   062   062   000    Old_age   Always       -       15428
 10 Spin_Retry_Count        0x0033   100   100   030    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       34
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       32
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       39
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       31 (Min/Max 15/39)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       0
220 Disk_Shift              0x0002   100   100   000    Old_age   Always       -       0
222 Loaded_Hours            0x0032   062   062   000    Old_age   Always       -       15427
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
224 Load_Friction           0x0022   100   100   000    Old_age   Always       -       0
226 Load-in_Time            0x0026   100   100   000    Old_age   Always       -       648
240 Head_Flying_Hours       0x0001   100   100   001    Pre-fail  Offline      -       0

I would expect something like the latter in order to be able to set up a proper (even if not accurate or reliable) monitoring.
Answer: This article says they expose SMART attributes normally, but for handwavy reasons only SeaTool knows how to interpret them beyond pass/fail.
To some extent, that is true for the SMART attributes of any disk as only the computed value is machine-readable, and the interpretation of the "raw" value is somewhat undefined. Temperature_Celsius is obvious, but the integration time for the various "error rate" attributes is vendor dependent, and so are the thresholds. SeaTools knows how to interpret raw values, that's it basically.
I doubt they'd be selling many harddisks if SMART support was missing or inaccurate, the vast majority of server disks go into RAID arrays where SMART is the only monitoring standard available.
They might be able to stack their own analysis software on top of a RAID controller, but if it doesn't integrate with minimum effort into existing monitoring solutions that provide a dashboard for the entire datacenter, it will be a niche solution for the hobbyist market.
This is one instance of the class of problems I call "top-of-the-foodchain" problems, where multiple software components are written to be the primary user interface, while the user requires them to be integrated into a larger system.
Comment: I understand, but smartctl -A reports no vendor specific attributes with worst and threshold values; it just reports the current and trip temperatures and some figures like accumulated start-stop cycles and accumulated load-unload cycles without any reference range. So how can S.M.A.R.T. infer something (even not precise) about the disk health status? In other words, I'm afraid that setting up an automated health check system based on S.M.A.R.T. might be useless since the disks do not provide enough information.
Comment: @NicolaMori, SMART expresses the reference range by normalizing the values, so for vendor-independent monitoring, all you need to check is whether the current and worst values are above the threshold, and whether they are moving towards it and how fast.
Comment: FWIW, I don't really bother with most of that monitoring beyond drawing pretty graphs. I have several disks that got ejected from my RAID for failing to hold data, but that look absolutely fine in SMART. All you get from SMART is an advance warning, sometimes.
Comment: The issue is just that with smartctl -A I don't get any value, just the temperature, so there's nothing to be monitored. See the edit on my original post. The disk S.M.A.R.T. support is advertised as available and enabled.
Comment: @NicolaMori, I have researched this a bit more -- the `-A` attribute dump is ATA/SATA specific and will not work for SAS drives. The `-x` extensive dump should show a bit more information, but it works a bit differently there. The `smartctl` manual page has a few comments where it says `[ATA]` or `[SCSI]` to highlight the differences.
Comment: Thank you very much for your help. I tried with `-x` and also with `-d scsi` but in the end no more info is printed in the SMART DATA SECTION. I guess that smart just behaves differently for SAS drives, and that the only available metrics are start-stop cycles, load-unload cycles and elements in the defect list as reported in my initial post. I guess I'll simply look at the synthetic health status (smartctl  -H), it seems the result is not worth the effort here. Thanks again!


Como aumentar a espessura de um text-shadow

Question: É possível aumentar a espessura de um text-shadow ?
Olhem esse EXEMPLO no JSFiddle
Simplesmente, é como se eu não tivesse colocado um text-shadow, fica invisível.
Answer: Duas sugestões para resolver o problema:

Utilizar um contraste melhor. A sombra branca se difunde no azul por trás e não fica muito perceptível;
Usar múltiplas sombras. As sombras via CSS para texto são realmente pequenas, e quando você aumenta o raio do amaciamento, elas ficam tão tênues que fica cada vez mais difícil vê-las. Felizmente a sintaxe da sombra permite múltiplas sombras por elemento, bastando separar as sombras por vírgulas.

Experimenta usar:
text-shadow: 0 0 30px #FFF, 0 0 30px #FFF, 0 0 30px #FFF, 0 0 30px #FFF, 0 0 30px #FFF, 0 0 30px #FFF, 0 0 30px #FFF;

No seu Fiddle. Você também pode variar os offsets de cada sombra para obter efeitos interessantes ;)
Comment: Ajustando os offsets você consegue melhorar o efeito. Olha: http://jsfiddle.net/Q8dX9/3/ Para conseguir um efeito mais artístico, você pode também combinar cores próximas entre si.
Comment: Vou testar agora.
Comment: Quando se utiliza várias sombras ele fica sem resolução, meio borrado. Veja: http://jsfiddle.net/AlexandreBR/Q8dX9/2/
Comment: Eu ajustei e ficou perfeito! eu utilizei `text-shadow: 0 0 15px #FFF, 0 0 30px #FFF, 0 0 40px #FFF, 0 0 40px #FFF, 0 0 50px #FFF;` Eu vou aumentando os pixeis, tipo 10, 20, 30. Assim ele vai desparecendo suavemente! Veja: http://jsfiddle.net/AlexandreBR/Q8dX9/4/


Solving a linear equation in an extension field

Question: Let $\mathbb K$ a commutative field and $\mathbb L$ an extension field of $\mathbb K$ (that is to say a field that contains $\mathbb K$).
Let $A$ be a $n \times p$ matrix over $\mathbb K$.
Let $B$ be a $n \times 1$ matrix over $\mathbb K$.
Let $S_k=\{X,X \; \text{is a} \;p \times 1 \; \text{matrix over } \; \mathbb K \; \text{such that} \; AX=B \}$
Let $S_l=\{X,X \; \text{is a} \;p \times 1 \; \text{matrix over } \; \mathbb L \; \text{such that} \; AX=B \}$
Prove that $$S_k \neq \emptyset \Leftrightarrow S_l \neq \emptyset$$
Clearly the left-to-right implication is true. I have no clue how to do the converse though.
Any help appreciated.
Answer: Let $S_l\ne\varnothing$. It is sufficiently to consider the case when
$n<p$ and ${\rm rank} A=n$. One can to choose values of $n$ unknowns arbitrary from $\mathbb{K}$ (see wikipedia). Then others are expressed through them and coefficients of $A$, so they also belong to $\mathbb{K}$.


Are Epochs, Learning rate and Hidden units related to each other?

Question: I'm working on an LSTM NN,  I wanted to know if there is any relationship between the learning rate, epochs and the hidden units that will/might affect my classification output?
The MATLAB version I'm using is 2018a
Thank you.
Answer: In itself, this is kind of an odd question since we do not know what kind of relationship you are looking for.
Epochs are the number iterations over the whole training set. By the typical definition, a neural network sees each training sample one time per epoch. Some people like to speak of steps instead, which are the number of training batches that the system needs to see in total.
Learning rate indicates how big or small the changes in weights are after each optimisation step. If you choose a large learning rate, the weights in the neural network will change drastically (see below).
Hidden units are the neurons in your network, typically those between the input and output layer. They are, of course, in their own layer(s). In other words, you can have an input layer with 120 features, one hidden layer with 256 hidden units, and a probabilistic output of one unit.
Now, to discuss the relationship between the three, I'll limit myself to performance and time consumption. When you specify the number of epochs, your system will run for n epochs and you have to wait until it is finished. This of course is not ideal, since it might be that the system has already reached its best performance and it keeps training for n-m epochs. To combat this, you can look into early stopping which stops training if the neural network has not improved on the validation set for a given amount of epochs. How quickly a network is optimised (and thus, how many epochs are needed) also depends on the learning rate. Usually a high learning rate will get to improved results rather quickly, but then it overshoots the optimal solution. When the learning rate is too small, it will just take too much computation time (and too many epochs) to find a good solution. It is important to find a good learning rate. Hidden units, then are not specifically related to the other two. They are not specifically influenced by them.


I am trying to rewrite a few links with htaccess

Question: I have a few URLs and I need them to be rewrite'd to the ones below:
http://domain.net/blog/posts
http://domain.net/blog/posts/index
http://domain.net/blog/posts/view/uri/non-working-holiday
http://domain.net/blog/posts/view/uri/we-no-longer-offer
http://domain.net/blog/posts/view/uri/festivals
http://domain.net/blog/posts/view/uri/christmas-is-just-around-the-corner

http://domain.net/posts/
http://domain.net/posts/index
http://domain.net/posts/view/uri/non-working-holiday
http://domain.net/posts/view/uri/we-no-longer-offer
http://domain.net/posts/view/uri/festivals
http://domain.net/posts/view/uri/christmas-is-just-around-the-corner

I was hoping that my .htaccess will fix this but it doesn't:
Options +FollowSymLinks
IndexIgnore */*
RewriteEngine on

RewriteRule ^blog\/(.*)$ posts\/$1  [NC]

# if a directory or a file exists, use it directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

# otherwise forward it to index.php
RewriteRule . index.php

Answer: Options +FollowSymLinks
IndexIgnore */*
RewriteEngine on

RewriteRule ^blog/(.*)$ /posts/$1  [R=301,L]

# if a directory or a file exists, use it directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# otherwise forward it to index.php
RewriteRule ^(.*)$ index.php [QSA,L]

Comment: It may need to be RewriteRule ^blog/posts/(.*)$ /posts/$1  [R=301,L]


A Coin Flip Problem Reconsidered

Question: In an earlier post, I asked:
"Suppose I have $n$ fair coins, and I mark one of them for identification. Next I flip the n coins without looking. My friend, who is looking on, now informs me that there were at least $k$ heads flipped.
What is the probability that my marked coin was heads?"
To that, @Łukasz Grad answered:
$$P(X_1 = 1| S \ge k) = \frac{\sum_{i = k}^{n} \binom{n-1}{i-1}}{\sum_{i=k}^{n}\binom{n}{i}}$$
But suppose the question was instead:
$P(S \ge k | X_1 = 1)$
and
$P(S \ge k | X_1 = 0)$
How does the answer change?
Answer: Hint: You can combine the answer of Łukasz Grad and Bayes' theorem. For instance,
$$P(S\geq k|X_1=1)=\frac{P(X_1=1|S\geq k)\cdot P(S\geq k)}{P(X_1=1)}$$
Obviously $P(S\geq k)=\sum\limits_{i=k}^{n}\binom{n}{i} \cdot 0.5^i\cdot 0.5^{n-i}$ and $P(X_1=1)=0.5$
Comment: I think you mean $P(S \ge k)$ and ${n \choose i}$
Comment: Yes. Good catch. Thanks
Comment: @Callculus, should $P(S>k)$ in the numerator not be $P(S>=k)$
Comment: @user120911 Sure. I have forgotten it at my first edit.
Answer: For $X_1 = 1$, just take the probability of $n-1$ coins landing on at least $k-1$ heads:
$$
\begin{split}
P(S \ge k \mid X_1 = 1) &= \sum_{i=k-1}^{n-1} \text{Binomial}(i; n-1, 1/2) \\
&= 0.5^{n-1} \sum_{i=k-1}^{n-1} {n-1 \choose i}.
\end{split}
$$
Similarly, for $X_1 = 0$, get the probability of $n-1$ coins landing on at least $k$ heads.
Comment: @user120911 Actually, if you work out callculus' answer, you'll find that it turns out to be the same as mine
Comment: your answer and Callculus' answer seem to be different in the summation. Why do you subtract 1 from n and 1 from k?
Comment: Is you answer for X = 0 ?
Comment: "For `$X_1 = 1$`, take the probability of `$n-1$` coins landing on at least `$k-1$` heads"

That makes intuitive sense. It seems like a general answer. Does it not also hold even if coins are not fair? I guess in that case we must keep track of the marked coin, by removing it and computing the probability of the other `$n-1$` coins landing on at least `$k-1$` heads". Here we would be dealing with the Poisson Binomial. Do you agree with that @Bridgeburner?

A Coin Flip Problem Reconsidered


Correct way to interpret sklearn's calibration error and generate a numeric calibration loss/score

Question: I have been using sklearn's CalibrationDisplay and think it is pretty cool. One thing I am wondering, though, is how I could potentially take that curve and make it an interpretable score. For example, here is a simple model and calibration curve:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.calibration import CalibrationDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=100000, n_features=10, n_classes=2, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

lr = LogisticRegression()

lr.fit(X_train, y_train)

display = CalibrationDisplay.from_estimator(estimator=lr, X=X_test, y=y_test, n_bins=10)

This curve is great for showing calibration, but I want to take it one step further. Using the prob_pred and prob_true attributes returned by from_estimator(), could I take the absolute value of the mean difference between the bins to get an overall calibration percentage error?
prob_pred
array([0.03257269, 0.1443675 , 0.24642089, 0.34935094, 0.45096792,
       0.55000066, 0.65085682, 0.75263477, 0.8547215 , 0.96749284])
prob_true
array([0.03235031, 0.15408416, 0.24418605, 0.34750911, 0.43198091,
       0.56077016, 0.67623421, 0.75992063, 0.85426829, 0.96590106])
np.absolute((display.prob_pred - display.prob_true).mean())
0.002781836006191922

And could this value be interpreted as something similar to: "on average there is an expected 0.278% difference between a given positive prediction's predicted probability and the expected true probability of a positive outcome for said observation", or something similar?
Again, I am very curious as to how I can generate a numeric calibration score from sklearn's CalibrationDisplay and I think what I came up with is a decent proxy? Would love to know more. Thanks!
Edit: my idea came from this excerpt in the documentation: "Well calibrated classifiers are probabilistic classifiers for which the output of predict_proba can be directly interpreted as a confidence level. For instance, a well calibrated (binary) classifier should classify the samples such that for the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class."
Comment: Brier score can be decomposed to give a calibration component. I wonder if that might be what you seek, though +1 for the interesting question!
Answer: In the graph produced by the code in the original question, the hope is for the plotted points to follow the line $y=x$. That is, we want the probabilities predicted by the model to match up with the true probabilities of event occurrence (a reasonable desire, I believe).
A standard way to check the deviation of predictions $\hat y$ and true values $y$ is with the $R^2$. Therefore, such a statistic might prove useful here.
$$
R^2=1-\left(\dfrac{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\hat y_i
\right)^2
}{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\bar y
\right)^2
}\right)
$$
If the predicted and actual probabilities match perfectly, your score will be a perfect $1$. As predictions deviate more and more from the true probability, the score will get worse and worse.
Another approach, granted, not derived from the graph, is that Brier score can be decomposed into measures of model calibration and discrimination (ability to distinguish between the categories), the latter of which might be of interest.


Input programmatically text in fields in webview after load page

Question: I have some issue about WebView.
I have some page. This page have two input fields.
We can input some text in field when the page is loading. But, can I set this fields after loading?
For example, page opened and I choose some text in toolbar menu. After this text pasted in webView fields.
It's really to realize? Thanks.
Answer: First let's break down the problem.

A webview will just load some HTML like a browser will. If you want to programmatically insert text into an input element in a webpage then we need to use javascript.
To Android, the webview is a view that handles its own drawing, we can interface with its methods and that's about it on a general level.

However, where these two meet is if we want to trigger some javascript to happen inside the webpage as a result of performing some action within the Android scope.
We still have to obey all of the rules that we described in our breaking down of the problem. Let's talk about it in those terms then: We need to interface with the webview's methods in such a way that it will use javascript to insert text into an input element.
So, now we know what we need:

JavaScript to insert text into an input element
A WebView method to use our JavaScript

The JavaScript is easy. Something like this will do the trick.
document.findElementById('my_text').value = 'My Text';

<input type='text' id='my_text' />

And thankfully there is such a WebView method we can use for this. It's fairly lengthy but quite straight forward and you can read all about it here.
Hope this helps.
Comment: I have an issue with this. After setting the input values of username and password, I am clicking the login button, but nothing happens. I need to manually click on the username field and delete and rewrite a letter manually and then click on login button works. I think some event also needs to fired programmatically to make sure the webview knows that input value has been set. Could you please help me in solving this?
Comment: I am just changing the input value, I need to change the state value.
Answer: The feedback from @DanielLaneDC helped me a lot just needed a little more info to get it working on my side. Hope it helps the next persone.
If your website has input fiels like so:
<input type="password" id="activeInput" autoaccept="true" value="">

I added a javascript function to my webserver:
function getFromAndroid(str){
    document.getElementById('activeInput').value = str;
}

In Android webview i called the javascript function:
webView.loadUrl("javascript:getFromAndroid('"+yourtext+"')");

if you didnt already make sure to add:
webSettings.setJavaScriptEnabled(true);


How is this combinatorial relation called?

Question: I was trying to learn some set theory and came up with the following relation between the union and intersection:
For any set consisting of sets $A$, we have:
$$\left\vert\bigcup_{a\in A}a\right\vert=\sum_{i=1}^{|A|}\bigg((-1)^{i+1}\sum_{B\subseteq A, |B|=i}\left\vert\bigcap_{b\in B}b\right\vert\bigg)$$
To me this is pretty obvious. We take the sum of all the cardinalities and then subtract those that were added twice. We then add the ones that were substracted twice back in et cetera.
The thing is, I have no idea how one would prove this (induction maybe?) or whether it's already proven and has some sort of name.
Is there anyone who can help me to either prove/disprove this relation or, if it already exists, give its name?
Comment: This is the [inclusion-exclusion principle](https://en.wikipedia.org/wiki/Inclusion%E2%80%93exclusion_principle).
Comment: Thank you. Would you please consider putting this into an answer that I can then accept?
Comment: Not every thing revolving sets is of set theoretic interest. If the question, or relation, or whatever, revolves around "how many" it is more likely to be about combinatorics, than it is about set theory.
Comment: I am not really sure link-only answers are allowed, I don't provide any insight by posting this Wikipedia page.
Comment: @C.Falcon Yes, you do. I agree with the OP,  your comment should be the accepted answer
Comment: For a proof of the principle have a look at [this answer](http://math.stackexchange.com/a/1674481/75923). It is a proof indeed based on induction.
Answer: This is the inclusion-exclusion principle.
Comment: Note to reviewers: added as an answer at request of OP.


RadioButton: How to change the IsChecked state?

Question: If a RadioButton is in the InChecked state, then the button has a black toggle. How can I change that so that is is white instead of black?
Answer: You will probably have to re-template your radiobutton control either by editing the default template found here, or by using Blend (if you have it available).  Blend will likely be a lot easier if you're familiar with it.
Answer: <code>namespace WpfApplication1
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();

        }

        private void button1_Click(object sender, RoutedEventArgs e)
        {
            radioButton1.IsChecked = false;
        }
    }
}
</code>


How do I insert Excel cells without creating a corrupt file?

Question: I'm using the OpenXML SDK to update the contents of an Excel spreadsheet. When inserting cells into an Excel row they must be inserted in the correct order or the file will not open properly in Excel. I'm using the following code to find the first cell that will be after the cell I am inserting. This code comes almost directly from the OpenXML SDK documentation
public static Cell GetFirstFollowingCell(Row row, string newCellReference)
{
    Cell refCell = null;
    foreach (Cell cell in row.Elements<Cell>())
    {
        if (string.Compare(cell.CellReference.Value, newCellReference, true) > 0)
        {
            refCell = cell;
            break;
        }
    }

    return refCell;
}

When I edit files with this code and then open them in Excel, Excel reports that the file is corrupted. Excel is able to repair the file, but most of the data is removed from the workbook. Why does this result in file corruption?
Side note: I tried two different .NET Excel libraries before turning to the painfully low-level OpenXML SDK. NPOI created spreadsheets with corruption and EPPlus threw an exception whenever I tried to save. I was using the most recent version of each.
Answer: The code you are using is seriously flawed. This is very unfortunate, seeing as it comes from the documentation. It may work acceptably for spreadsheets that only use the first 26 columns but will fail miserably when confronted with "wider" spreadsheets. The first 26 columns are named alphabetically, A-Z. Columns 27-52 are named AA-AZ. Column 53-78 are named BA-BZ. (You should notice the pattern.)
Cell "AA1" should come after all cells with a single character column name (i.e. "A1" - "Z1"). Let's examine the current code comparing cell "AA1" with cell "B1".

string.Compare("B1", "AA1", true) returns the value 1
The code interprets this to mean that "AA1" should be placed before cell "B1".
The calling code will insert "AA1" before "B1" in the XML.

At this point the cells will be out of order and the Excel file is corrupted. Clearly, string.Compare by itself is not a sufficient test to determine the proper order of cells in a row. A more sophisticated comparison is required.
public static bool IsNewCellAfterCurrentCell(string currentCellReference, string newCellReference)
{
    var columnNameRegex = new Regex("[A-Za-z]+");
    var currentCellColumn = columnNameRegex.Match(currentCellReference).Value;
    var newCellColumn = columnNameRegex.Match(newCellReference).Value;
    var currentCellColumnLength = currentCellColumn.Length;
    var newCellColumnLength = newCellColumn.Length;
    if (currentCellColumnLength == newCellColumnLength)
    {
        var comparisonValue = string.Compare(currentCellColumn, newCellColumn, StringComparison.OrdinalIgnoreCase);
        return comparisonValue > 0;
    }

    return currentCellColumnLength < newCellColumnLength;
}

If you wanted to place a new cell in column "BC" and you were comparing to cell "D5" you would use IsCellAfterColumn("D5", "BC5"). Substituting the new comparison function into the original code and simplifying with LINQ:
public static Cell GetFirstFollowingCell(Row row, string newCellReference)
{
    var rowCells = row.Elements<Cell>();
    return rowCells.FirstOrDefault(c => IsNewCellAfterCurrentCell(c.CellReference.Value, newCellReference));
}


Pearson correlation critical level values formula

Question: As statistics is not my domain, I have got hard time to find and understand all missing pieces. I have seen many tables with critical levels for pearson-r but I can't find any formula I can trust. What I need is to generate this kind of table but for any value I wish:
df\p  0,1    0,05    0,02    0,01    0,001
-------------------------------------------
1   0,98769 0,99692 0,99950 0,9998  0,99999
2   0,90000 0,95000 0,98000 0,9900  0,99900
3   0,8054  0,8783  0,93433 0,95873 0,99116
4   0,7293  0,8114  0,8822  0,91720 0,97406
5   0,6694  0,7545  0,8329  0,8745  0,95074
6   0,6215  0,7067  0,7887  0,8343  0,92493
7   0,5822  0,6664  0,7498  0,7977  0,8982
8   0,5494  0,6319  0,7155  0,7646  0,8721
9   0,5214  0,6021  0,6851  0,7348  0,8471
10  0,4973  0,5760  0,6581  0,7079  0,8233
11  0,4762  0,5529  0,6339  0,6835  0,8010
12  0,4575  0,5324  0,6120  0,6614  0,7800
13  0,4409  0,5139  0,5923  0,6411  0,7603
14  0,4259  0,4973  0,5742  0,6226  0,7420
15  0,4124  0,4821  0,5577  0,6055  0,7246
16  0,4000  0,4683  0,5425  0,5897  0,7084
17  0,3887  0,4555  0,5285  0,5751  0,6932
18  0,3783  0,4438  0,5155  0,5614  0,6787
19  0,3687  0,4329  0,5034  0,5487  0,6652
20  0,3598  0,4227  0,4921  0,5368  0,6524
25  0,3233  0,3809  0,4451  0,4869  0,5974
30  0,2960  0,3494  0,4093  0,4487  0,5541
35  0,2746  0,3246  0,3810  0,4182  0,5189
40  0,2573  0,3044  0,3578  0,3932  0,4896
45  0,2428  0,2875  0,3384  0,3721  0,4648
50  0,2306  0,2732  0,3218  0,3541  0,4433
60  0,2108  0,2500  0,2948  0,3248  0,4078
70  0,1954  0,2319  0,2737  0,3017  0,3799
80  0,1829  0,2172  0,2565  0,2830  0,3568
90  0,1726  0,2050  0,2422  0,2673  0,3375
100 0,1638  0,1946  0,2301  0,2540  0,3211

More precisely I need piece of code (but not in R) to generate value for any N I need. For example for N = 79, p = 0.01, one-tailed (with steps N-2).
It would be great if someone explain how to calculate it step by step, post formula or just write piece of code (or pseudocode) easy to understand.
function criticalR(N, alpha){
    return // this part I need;
}

console.log( criticalR(50, 0.02) ); // prints 0,5155

Comment: Close voters (as off topic): please indicate why you believe this doesn't require statistical expertise to answer (as described in the [help/on-topic] under *Programming*) -- I find myself at a loss to guess why.
Answer: The exact null distribution for a Pearson correlation where the variables are assumed to be bivariate normal and the null is $\rho=0$ will have the form of a shifted and scaled symmetric beta distribution - specifically, linearly transformed to (-1,1).
The explicit form of the density is given in the Wikipedia article on the Pearson correlation coefficient:
$$p(r) = \frac{(1-r^2)^{(n-4)/2}}{\operatorname{B}(1/2, (n-2)/2)}$$
where $r$ is the random variable (the sample correlation), $n$ is the sample size (number of observation-pairs) and $B$ is the (complete) beta function.
We can readily use the beta cdf and the inverse beta cdf (common in many stats packages) to give critical values and p-values by taking care of the linear transformation ourselves. Or if we're writing code from scratch many numerical libraries offer the incomplete beta function and its inverse, which can be used to do the same thing.
As an alternative, given the null hypothesis of population correlation of 0, a particular transformation of the sample correlation will have a t-distribution; if you're stuck using tables this will be particularly convenient, since t-tables are often more extensive than tables for the correlation coefficient.
This is also given in the same Wikipedia article linked above, in the section titled Testing using Student's t-distribution:
$$t=r{\sqrt {\frac {n-2}{1-r^{2}}}}$$
with $n-2$ degrees of freedom.
This relationship may be used to obtain p-values directly or it can be used to back out critical values if you want to build a larger table (though I question the need for tables at all, since t-distribution calculations are nearly ubiquitous in statistical software and numerical packages for languages) -- it is straightforward to make $r^2$ the subject: $r^2=1-1/[1+t^2/(n-2)]$, and hence to obtain critical values for $r$ from critical values for $t$.
As a check, the 1% two tailed critical value for a t with 80 df (i.e. with n=82) is  2.6387 and 1-1/[1+2.6387^2/(80)] is about 0.080065; the square root of that is 0.28296, which rounds off to the value in your table above (0.2830).
For moderate to large $n$ (say $n>80$ or so, depending on your tolerance for approximation) the normal approximation is quite good at typical significant levels (say near 5%). If you're using very small significance levels (perhaps to maintain a low familywise type I error rate), then it may take considerably larger sample size to have a good approximation from the normal, but if you're using a computer there's little reason not to calculate exactly using either the beta distribution or the transformation to a t-distribution (both of which are exact when the assumptions are true).
Comment: You'll either need a function for the inverse incomplete beta, or the inverse cdf for a t distribution (followed by the formula to convert t to r); As I already stated, many numerical libraries have the first. Stats libraries often have the second. An F inverse cdf would also work.
Comment: I still analyse your answer, I think second part with example is quite clear, the question and problem I can see is that N is unknown input number - it can be 50, 101 or 183 (thats why I want to avoid big table and generate it instead) so I would need to store again table for t. I am not sure I understood first part - do I need correlation calculated for first formula? I am not sure how exactly the first formula can be used - I mean the `r` there isalready calculated correlation or the factor I need or just completely something else?
Comment: I can't follow the second half of your comment, sorry. If you want tables of critical values you won't have r values at all. If you have sample r's you'll presumably want to compute p-values. Which is it?
Comment: I need critical values, thats why 'r' in the first formula was confusing for me but if I understand correctly it's for reverse situation. I could find some library to get criticals but I need to understand also how it was made - someone generated big table of numbers and I had no idea from where it comes from.
Anyway if I am correct I just need beta function to generate t values of normal  distribution and then use the second formula, yes?


Do I have perfect pitch

Question: I noticed that when a certain note is played I can immediately attach to it songs that begin with that note.
I was wondering if:
1 - I have the potential for perfect pitch
2 - Is there a way to check if you have the potential for perfect pitch, rather than perfect pitch itself?
Comment: It *may* be possible, without any musical training at all, to have perfect pitch; yet be completely unable to do the blind piano test - because you don't know the names of the notes.
Comment: You say 'a certain note'. Do you mean that there is just one note that this works with?
Comment: What if the song is played in different keys by different groups?  Anyway, it's easy to test perfect pitch.  face away from the piano while a friend plays single keys.  If there's any doubt in your mind, you don't have perfect pitch.
Comment: @JaySkyler That does not seem correct - hearing and instantly abstracting any pitch to the note A would be the ability to suppress perfect pitch and impose a system of relative pitch.  If you can hear any frequency and tell what name it has (along with the system you are using : i,e, equal tempered A=440), then THAT is perfect pitch.  If you are working from a reference tone, then you are not using perfect pitch at all.  Where we define the cutoff for 'when you last heard a note' is rather arbitrary and leads me to believe tinnitus makes perfect pitch much easier.
Comment: @tetsujin I disagree, perfect pitch is specifically the ability to consistently match tones with their corresponding symbols and names in music even if the name may be different based on language. In addition to the ability to sing the corresponding tone when presented with the name or symbol.
Comment: @JaySkyler Take an example; someone with no musical training whatsoever. Challenge, 'Sing me the opening line of [known reference] songs x, y & z' - a task they may eminently be capable of, without any knowledge of what those notes are called. To them, the symbols & names are just they 'know' what those notes ought to be, because that's how they heard them. Personally, I cannot sing a note I'm handed on a sheet of paper without first playing it on some instrument; but I know for certain if a known song is in the 'right' or 'wrong' key, within a semitone.
Comment: @Tetsujin For instance, I certainly have the ability to begin a song singing without any instrumental backing and I have many of the notes memorized, but I do not have perfect pitch. There are countless frequencies we will accept as being an A. Perfect pitch is the ability to immediately hear and instantly abstract any of them to the note A. Many have good ears, perfect pitch is rare because it specifically requires this instant symbolic abstraction.
Comment: I was pretty certain that having perfect pitch also entailed the ability of recognizing "out-of-tune"-ness to a very sharp degree regardless of tuning system (though most people associate this with a standard tuning such as A=440Hz). For example, being able to tell, with just your ears, whether a played A is sharp or flat, within a degree of a few cents. This, in essence, allows the person to be able to recall and produce accurately, a given note, on an instrument of choice (voice included)
Comment: I have 2 friends, both of whom have "perfect pitch"... yet both have a tolerance to pitches being "imperfect" We used to go to a pub with an old 45rpm jukebox... which played at slightly the wrong speed'. They would irritate the rest of us by yelling the chord structure of every song at each other [we were young, we were drunk]... a semitone out from each other; because each was referencing a slightly different 'absolute' & the juke box was just about a quarter-tone out ;)
Comment: [contd...] One of them is famous in local circles for once being at a premiere of a new classical choral piece. he was given the score to read whilst listening. About ten minutes in, he got up & left, to read the score outside... claiming loud enough for the conductor to hear, "I can either listen to this or read it. I cannot do both at once!"
Answer: Many musicians can tell intervals, i.e. guess what a note is once they have been told the name of another note.  That's relative pitch.
A smaller group can often start some songs on the correct note.  This happens with songs they hear or practice often typically.  And they will occasionally make mistakes.  (I am in that category: I will get the start note of some songs right more than half of the time; when I am wrong, I am usually off by a tone or so, and sometimes I am completely off.)
Very few can systematically start a song on the correct note, or, given a note, tell instantly which song starts with that note.  In my opinion, the last step of telling the name of a note as opposed to naming correctly, effortlessly, which song starts with that note is not a requirement to perfect pitch: knowing the note name is a detail that you can master easily if you already have associations to each note without thinking about it, and that last skill isn't related to your ear per say.
For example, if you can almost always sing a C because your favorite song starts with a C and from there you derive the relative interval to, say, an A, and can then sing that A pretty accurately, you have a good ear, but not perfect pitch.
In your case, you say you associate "certain" notes to its song.  If you never get that wrong, you are closer than most to perfect pitch.  But until you have an association of each note sound to a correct representation of that note, be it a song name or note name, you would not have what is called perfect pitch as far as goes my understanding of perfect pitch.
People with perfect pitch will be able to name quickly all the notes of a melody they hear effortlessly.  They don't have to think about intervals between any of these notes.  You may still have a way to go for that; most are :-)
By the way, there are some methods to help people acquire perfect pitch, and they do start exactly with what you do: being able to associate sounds to concepts: songs you know, colors, textures, etc. so all of this is a continuous spectrum...  A friend of mine got very near perfect pitch with that approach so people have a margin for improvement: it doesn't have to be a gift though it can be just that.
Answer: Yes.  Your potential for perfect pitch is very high.
Take care of your ears (you can go to loud shows but if you are close to huge speakers be sure to have earplugs).  The little fibres/hairs in your ears are lifetime friends so take good care of them.
Calling out songs that blend well is a great skill and should you ever want to become a DJ you'll probably master the crossfade.
However, understanding the interval is more vital to musical success than "absolute pitch" recall, and honestly there is probably not such a thing as "absolute" pitch thanks to relativity.
Study how string length and octaves are related and your musical intuition shall steadily improve.
Comment: Why was this made community wiki?

Do I have perfect pitch


How to hide a view if it has more than 1 row and also show some block if this is true?

Question: I have found a solution in "Hide a view if less than N results" for hiding the view by using the template.php file.
However I need to show a certain block also, the block is already in the correct region it would just need 'switching on' for this page.
Would it be possible to provide a callback inside the following code from the link posted above
<code>function MY_THEME_NAME_preprocess_views_view(&$vars) {
  if ($vars['display_id'] == 'MY_DISPLAY_ID' && count($vars['view']->result) < 1) {
    $vars['view']->result = NULL;
  }
}
</code>
To a function which turned the block on
Note: I am just starting module development.
edit: I'm now thinking that this would need to be done inside a custom module as it might be too late to turn the block on from the template.php file. If someone could clarify this.
Comment: Is the `block` that you're talking about a `view block` or a `regular/normal block`?
Comment: yes it is a views block.. ahh I think I see where this is going, nice thinking
Answer: Part 1
Enable the Rules block visibility module. Here is a quote from its project page:

The Rules block visibility module allows Rules components to be used to control block visibility. This provides Drupal administrators and developers extreme flexibility in controlling when blocks should be displayed on their websites, in addition to the default visibility options provided by Drupal.
The general idea is that if you can do it with Rules, you can use it to control block visibility, so the possibilities are limitless:

Need to show a block only for users registered more than a month ago?
Perhaps you have a block that must be shown only between 8am-5pm on weekdays?
What about displaying or hiding a block based on current weather conditions?

All of this can be done by using Rules block visibility.

For an illustration of how to use this module, refer to my answer to "How to stop a Views block from displaying to admins, such as user/1?".
Part 2
As per the "if you can do it with Rules, you can use it to control block visibility" above, you've reduced your question to making Rules "check the number of rows returned by your view" (so that the block is only shown if the view has more then 1 row).
For that, perform these steps:

Use the Views Bulk Operations module to create a variation of your view in which you add a VBO field to it.
Create a Rules Component with a Rules condition "Check number of results returned by a VBO view". And select the VBO view created in the previous step.


java to matlab conversion

Question: I am a newbie at matlab . As a part of a larger problem, I need to find maximum number of occurrences of a string in an array of Strings.
Since I have some experience in java I have written the partial code in java ( only until the number of occurences of the string within the string array can be computed I can sort the hashmap depending on the values and extract this)
int incr = 0;
    String[] c = { "c1", "c2", "c3", "c1", "c2", "c2", "c2","c1","c2" };
    Map<String, Integer> classRegistry = new HashMap<String, Integer>();
    for (int i = 0; i < c.length; i++) {
        String classes = c[i];
        if (!(classRegistry.containsKey(classes))) {
            for (int j = i + 1; j < c.length; j++) {
                if (classes.equals(c[j])) {
                    incr++;
                }
            }
            classRegistry.put(classes, incr+1);
            incr = 0;
        }
    }

Any idea how i can use something like a hashMap in MATLAB to calculate the number of occurrences of all the strings in an array
Thanks,
Bhavya
Answer: MATLAB has a function TABULATE available in the Statistics Toolbox:
c = {'c1' 'c2' 'c3' 'c1' 'c2' 'c2' 'c2' 'c1' 'c2'};
t = tabulate(c)
t = t(:,1:2)

The result:
t =
    'c1'    [3]
    'c2'    [5]
    'c3'    [1]

Alternatively, you can do the same using the UNIQUE and ACCUMARRAY functions:
c = {'c1' 'c2' 'c3' 'c1' 'c2' 'c2' 'c2' 'c1' 'c2'};
[classes,~,subs] = unique(c);
counts = accumarray(subs(:),1);

Again the result as before:
>> t = [classes(:) num2cell(counts)]
t =
    'c1'    [3]
    'c2'    [5]
    'c3'    [1]

Then to find the string that occurred the most, use:
>> [~,idx] = max(counts);
>> classes(idx)
ans =
    'c2'

Comment: I used the tabulate function and was able to obtain the solution quite easily.
Answer: You didn't specify exaclty how you would want your input and output data types to be, but I wrote this quick script you might find usefull.
c = {'c1' 'c2' 'c3' 'c1' 'c2' 'c2' 'c2' 'c1' 'c2'};
count = struct();
for ic=1:length(c)
    field = c{ic};
    if isfield(count, field)
        count = setfield(count, field, getfield(count, field) + 1);
    else
        count = setfield(count, field, 1);
    end
end

The output for this specific c would be
count =

    c1: 3
    c2: 5
    c3: 1

Answer: Since you have experience with Java, you could just write your code in Java and call it from MATLAB.  This techdoc article should help you get started if you decide to go this route.  But it would probably be more useful to learn how to do it in m-script (see jomb87's answer)
Incidentally, you could improve the performance of your Java algorithm if you take further advantage of the hash map:
    for (int i = 0; i < c.length; i++) {
        String classes = c[i];
        if (classRegistry.containsKey(classes)) {
            classRegistry.put(classes, classRegistry.get(classes) + 1);
        } else {
            classRegistry.put(classes, 1);
        }
    }

java to matlab conversion


How setInterval works in the background of React Re-Rendering

Question: I'm creating a simple countdown timer app with React and I am having hard time understanding how setInterval works while React re-renders the component.
For example, this following code had timer continue to run even though I had used clearInterval onPause().
    let startTimer;

    const onStart = () => {
        startTimer = setInterval( ()=>{
            if ( timeRemaining === 0 ) {
              clearInterval(startTimer);
              setIsCounting(false)
              return
            }
            updateTimer()
          }, 1000)
          setIsCounting( (prev) => !prev )
     } // end of onStart

    const onPause = () => {
        setIsCounting( (prev) => !prev )
        clearInterval(startTimer)
    }

    return (
             { props.isCounting ?
               <button onClick={props.onPause}> Pause </button>
             : <button onClick={props.onStart}> Start </button> }
    )

However, the timer successfully pauses when I simply change
let starter;

to
let startTimer = useRef(null)

const onStart = () => {
        startTimer.current = setInterval( ()=>{
            if ( timeRemaining === 0 ) {
              clearInterval(startTimer);
              setIsCounting(false)
              return
            }
            updateTimer()
          }, 1000)
          setIsCounting( (prev) => !prev )
     } // end of onStart

    const onPause = () => {
        setIsCounting( (prev) => !prev )
        clearInterval(startTimer.current)
    }

What's happening to setInterval when React re-renders its component? Why did my timer continue to run when I didn't use useRef()?
Answer: A ref provides what's essentially an instance variable over the lifetime of a component. Without that, all that you have inside an asynchronous React function is references to variables as they were at a certain render. They're not persistent over different renders, unless explicitly done through the call of a state setter or through the assignment to a ref, or something like that.
Doing
let startTimer;

const onStart = () => {
    startTimer = setInterval( ()=>{

could only even possibly work if the code that eventually calls clearInterval is created at the same render that this setInterval is created.
If you create a variable local to a given render:
let startTimer;

and then call a state setter, causing a re-render:
setIsCounting( (prev) => !prev )

Then, due to the re-render, the whole component's function will run again, resulting in the let startTimer; line running again - so it'll have a value of undefined then (and not the value to which it was reassigned on the previous render).
So, you need a ref or state to make sure a value persists through multiple renders. No matter the problem, reassigning a variable declared at the top level of a component is almost never the right choice in React.
Comment: During the re-rendering, you mentioned that startTimer gets a new reference. How come setInterval is still running when its assigned variable received a new reference? @CertainPerformance
Comment: A periodic function scheduled with `setInterval` will only ever be stopped if the same timeout ID is passed to `clearInterval`. Other than that, the timeout ID returned by `setInterval` doesn't have to be used at all - if it gets ignored and eventually garbage collected, that will have no effect on the running interval, which will continue to run. The timeout ID returned by `setInterval` isn't a timer object, it's only a number identifier used to clear a timer. If the identifier is no longer accessible, the timer keeps running.


How to avoid first byte timeout with nginx

Question: I have an application in RubyOnRails, proxied by a NGINX server that is behind a CDN service. The CDN has a limitation that if the response first byte does not arrive in 60 seconds, the CDN server responds:
Error 503 first byte timeout
The problem is that I do have some requests that take more than 60 seconds to be processed on the Rails side. Is there a way to hack the response, sending a few bytes before the response finish its processing? How?
Edit 1:
I know 60s is a long time and we are thinking about ways to avoid this. Still, I need a temporary solution that allows those 60s requests work. Even if I use Ajax, some ajax calls would take more than 60s and would fall into the same issue. The tcp_nopush or tcp_nodelay options did not work, I think it is because nginx is waiting the rails application response to know what headers it should send. So, maybe the solution is something that I need to hack inside the RubyOnRails request/response cycle
Answer: A request taking more than 60 seconds is a significant amount of time. You may want to review your application. A page load time that long could have a lot of visitors exiting your app. You may want to consider loading the data asynchronously such as with AJAX or websockets.
The time to first byte is usually applied to the headers, not the response body.  The client should be receiving the headers early on. Are you sure that it is your CDN timing out the requests? NGinx will timeout your requests also. A good way to check is with curl:
curl -I -H "Host: <domain>" http://<ip of proxy>/request/uri

(Use the real IP of your proxy, not the CDN IP)
The -I option will show you the response headers, that way you can get a good indication of what error code (if any) is being returned by nginx. You can also see if you are receiving request headers and how long it takes, as this is what tcp first byte refers to.
If you are not receiving the headers quickly, try using:
http {
tcp_nopush off;
tcp_nodelay on; # force socket to send buffer
}

If you are getting a gateway timeout error or similar, please try the following configuration options:
http {
keepalive_timeout 300;
proxy_connect_timeout 300;
proxy_read_timeout    300;
proxy_send_timeout    300;
}

Comment: This is good advice, especially around rethinking the architecture. If something is going to take more than a couple of seconds of computation you probably should make it asynchronous. Fir example with a message queue or similar, with the results available from a web page later. 60 seconds is longer than virtually everyone will wait for a public website, so unless this an internal only site you'll likely lose visitors.
Comment: I know 60s is a long time and we are thinking about ways to avoid this. Still, I need a temporary solution that allows those 60s requests work. Even if I use Ajax, some ajax calls would take more than 60s and would fall into the same issue.
The tcp_nopush or tcp_nodelay options did not work, I think it is because nginx is waiting the rails application response to know what headers it should send. So, maybe the solution is something that I need to hack inside the RubyOnRails request/response cycle