Cannot Open Images With PyGame in Atom

Question: I am learning python using this website
I am trying to load an image like so:
background_image = pygame.image.load("background.jpg").convert()

I have found several other posts asking the same question, and all the answers are telling the poster to make sure the image is in the same folder as the .py file. Well, it is. I double checked.
I also tried putting it into a subfolder called 'images' and using this.
background_image = pygame.image.load(os.path.join("images","background.jpg")).convert()

In both cases I get the following error message.
pygame.error: Couldn't open saturn_family1.jpg

or
pygame.error: Couldn't open images\saturn_family1.jpg

When using the absolute path, it works. But shouldn't I be able to use a relative path like this?
Also, this happens when using the script package to execute in Atom. But when executing the actual python file, it works.
I am very frustrated, this doesn't seem to make any sense! What's going on?
The full error message is:
C:\Users\Manu\Dropbox\Python\ProgramArcadeGames
Traceback (most recent call last):
  File "C:\Users\Manu\Dropbox\Python\ProgramArcadeGames\Ch11\11_graphicsAndSound.py", line 26, in <module>
    background_image = pygame.image.load(os.path.join("images","saturn_family1.jpg")).convert()
pygame.error: Couldn't open images\saturn_family1.jpg
[Finished in 0.815s]

This is the .py directory:

UPDATE:
I ran the exact same code in sublime and it worked fine. I am assuming this is a problem with Atom, so I will just use Sublime from now on. Thanks to everyone who tried to help!
Comment: Are you sure the extension is `.jpg` or something else?
Comment: @MalikBrahimi Yes, it's .jpg
Comment: Try adding `print(os.getcwd())` just before loading the file.  This is just a sanity check to make sure the current working directory is what you think it is.
Comment: Add a screenshot of your project directory.
Comment: I have added the full error message to my original post.

@saulspatz As you can see, `print(os.getcwd())` gets me `C:\Users\Manu\Dropbox\Python\ProgramArcadeGames` whereas my .py file is actually in `C:\Users\Manu\Dropbox\Python\ProgramArcadeGames\Ch11`.

Why is this?
Comment: Just guessing: Did you double check capital letters? There might be a difference between MyFamily.jpg and myfamily.jpg on plattform running your code
Comment: @Manu Do you do `os.chdir(..)` anywhere in your file?  Do you call a function that might do this?
Comment: @saulspatz No, I don't believe so.
Comment: @frlan Yes, I literally copy/pasted the file name just to make sure.
Comment: @MalikBrahimi I added the screenshots as requested.
Answer: Putting the file in the same directory as your python program will work provided that is the current working directory.  Since we know this is not the case, either you are somehow changing the working directory while the script is executing, or you are starting in some other directory.  You could do this on the command line with
python ../myscript.py

but I don't suppose you're doing this.  You say something about running the script under Atom, which may explain it.  I don't have any experience with this, so I can't say offhand.  Please put
print os.getcwd()

as the very first line in your program.  Then you'll know where you're starting and whether the current directory is changing.  If it's not changing, try running the script under Atom and from the command line to see if you get different results.  We won't be able to solve the problem until we know exactly what is happening.
Comment: I made a brand new python file in this directory: `C:\Users\Manu\Dropbox\Python\ProgramArcadeGames\Ch11\test.py` The file contains only two lines: `import os` and `print(os.getcwd())`
If I run the script in Atom, I get `C:\Users\Manu\Dropbox\Python\ProgramArcadeGames`
If I run the exact same script in Sublime Text 2, I get `C:\Users\Manu\Dropbox\Python\ProgramArcadeGames\Ch11` which is the correct path. What is going on here??
Comment: Weird.  I tried this out on atom and I can't figure out what it's doing.  I wrote a two-line script like yours, ran it atom, and it worked.  Then I opened a file in another folder, then ran the original script, and it printed the name of the second folder.  That makes sense, but I can't get it to switch back to the first folder, except by shutting down atom and restarting it.
Answer: Just use the absolute path for the picture.
For example:
self.image = pygame.image.load('C:/Users/denis/Documents/python_practice/git_practice/Python_lessons/alien_invasion/images/ship.bmp')

Answer: Same error happened when I was coding with VS.
I fixed that by changing the path of the image (use an absolute path here) .
Change pygame.image.load('/images/xxx') to
pygame.image.load('/Users/xxx/..../images/xxxx')

I am using the mac but it should be the same on other OS.
Comment: You should never use absolute paths for resources in programs, it carries the very very high probability of breaking when anyone else with a slightly different directory tree structure comes to try to run it. Always use relative paths if possible.

Cannot Open Images With PyGame in Atom


How to filter a list, remove lowest probability value, using python

Question: I have the image in which I want to detect the text, I am using easyocr to detect the text. The OCR gives the output bounding box value and probability as shown in the output image. I want to remove the probability which is less than 0.4 of any text detected How can I change that?
Image1

Image 2

The results element gives the output probability of the first text and second text 'AA' as shown in figure. i want to remove the lowest probability text detected.
Output of image1

Output of image2

Requirements
pip install pytesseract
pip install easyocr
Run the code using python main.py -i image1.jpg
<code># import the necessary packages
from pytesseract import Output
import pytesseract
import argparse
import cv2
from matplotlib import pyplot as plt
import numpy as np
import os
import easyocr
from PIL import ImageDraw, Image

def remove_lines(image):
    result = image.copy()
    gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
    thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

    # Remove horizontal lines
    horizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (40,1))
    remove_horizontal = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, horizontal_kernel, iterations=2)
    cnts = cv2.findContours(remove_horizontal, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    cnts = cnts[0] if len(cnts) == 2 else cnts[1]
    for c in cnts:
        cv2.drawContours(result, [c], -1, (255,255,255), 5)

    # Remove vertical lines
    vertical_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1,40))
    remove_vertical = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, vertical_kernel, iterations=2)
    cnts = cv2.findContours(remove_vertical, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    cnts = cnts[0] if len(cnts) == 2 else cnts[1]
    for c in cnts:
        cv2.drawContours(result, [c], -1, (255,255,255), 5)

    plt.imshow(result)
    plt.show()

    return result

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
    help="path to input image to be OCR'd")
ap.add_argument("-c", "--min-conf", type=int, default=0,
    help="mininum confidence value to filter weak text detection")
args = vars(ap.parse_args())

reader = easyocr.Reader(['ch_sim','en']) # need to run only once to load model into memory

# load the input image, convert it from BGR to RGB channel ordering,
# and use Tesseract to localize each area of text in the input image
image = cv2.imread(args["image"])
# image = remove_lines(image)

results = reader.readtext(image)

print(results)

</code>
Answer: <code>results=[([[5, 0],[233, 0],[233, 15],[5, 15]],' Ꮎ ]TC T III3 UᎪCU 3', 0.011015821004953916),
 ([[241, 0], [390, 0], [390, 15], [241, 15] ] , 'ᎠA[ [ C 0lᎢ', 0.0023567583563770737),
 ([[2, 16], [46, 16], [46, 42], [2, 42]], 'MM', 0.9965182566504686),
 ([[98, 16], [140, 16], [140, 46], [98, 46]], 'D', 0.9973547096148511),
 ([[182, 16], [220, 16],[220, 44], [182, 44]], 'Y', 0.9971791823074896),
 ([[24, 46], [62, 46], [62, 74], [24, 74]], '62', 0.9999828941291119),
 ([[94, 46], [130, 46], [130, 74], [94, 74]], '26', 0.9997197349619524),
 ([[180, 46], [242, 46], [242, 74], [180, 74]],'1970', 0.999931275844574)]

low_precision = []
for text in results:
    if text[2]<0.5: # precision here
        low_precision.append(text)
for i in low_precision:
    results.remove(i) # remove low precision
print(results)
</code>
result:
<code>[([[2, 16], [46, 16], [46, 42], [2, 42]], 'MM', 0.9965182566504686),
 ([[98, 16], [140, 16], [140, 46], [98, 46]], 'D', 0.9973547096148511),
 ([[182, 16], [220, 16], [220, 44], [182, 44]], 'Y', 0.9971791823074896),
 ([[24, 46], [62, 46], [62, 74], [24, 74]], '62', 0.9999828941291119),
 ([[94, 46], [130, 46], [130, 74], [94, 74]], '26', 0.9997197349619524),
 ([[180, 46], [242, 46], [242, 74], [180, 74]], '1970', 0.999931275844574)]
</code>
Comment: Thanks for your answer but I have lots of images and the text with low probability at different indexes like in image2 the text[2]<0.5 doesn't work on them, I want to get this thing to be general
Comment: It works, try it. The index is for `([[1, 2], [2, 9]], 'one', 0.556548)` which is always the probability .
Comment: It does not work in image 2 case after removal [([[241, 0], [390, 0], [390, 15], [241, 15]], 'DA| [ C OlI', 0.0023567583563770737), ([[2, 16], [46, 16], [46, 42], [2, 42]], 'MM', 0.9965182566504686),
Comment: Updated my answer. Try again.
Comment: this answer amounts to a **list comprehension** that filters for the confidence value... `results = [row for row in results if row[2] >= 0.5]`
Comment: Nice hit @ChristophRackwitz !!
Comment: @Nima thanks for your answer, Can we remove the part with white pixels values which threshold is less than 0.5? Only show the Text which is detected
Comment: Do I need to post that in another question?
Comment: I would recommend that you think about that issue some more. maybe you can figure it out on your own. you have the four corners of the box. there are multiple ways to draw.


Second normal form : Artificial key vs Composite Primary Key

Question: I recently read about the second normal form that: "If your table has an artificial primary key and no composite primary key, it's in second normal form".
What if my table has an artificial key as well as a composite primary key in my table?
Will it be in second normal form, then also?
Comment: Where is this statement from?
Comment: @stark  Reference: HeadFirst SQL
Answer: A relation in which every candidate key consists of a single column is in 2NF. (The relational model always considers every candidate key.)

element_name  atomic_number  chem_symbol
--
Hydrogen      1              H
Helium        2              He
Lithium       3              Li
...

Each attribute here is a candidate key. This table is in 2NF. It's also in 3NF, BCNF, 4NF, and 5NF.

What if my table has an artificial key as well as a composite primary key in my table? Will it be in second normal form, then also?

It depends on the non-prime attributes and their dependencies. If you have a non-prime attribute that's functionally dependent on part of that composite primary key, then the relation is not in 2NF.
Comment: The first sentence is wrong. All CKs simple but {} determining a non-prime attribute, ie only 1 value allowed for it, does not satisfy 2NF. (As I expect you know. So I'm surprised you don't say that.)


Variations in "Il Gardellino" - Cantabile

Question: Listening to renditions of Vivaldi's Concerto in D Major, Op 10 No 3, Second Movement - Cantabile, it seems that when it's played as solo piece, the repeats include ornaments that are not in the score. In a recording by Sir James Galway of the entire concerto, there was none of this ornamentation on the repeats.
So, my questions are: Is there a tradition about this? Are the ornaments individual to each player? Is there a written set of standards?
Pahud plays the Cantabile:

Galway plays the Concerto (Cantabilie/Largo) begins at 3:55.
Answer: In Baroque music it was traditional to embellish on the repeat.  Many performers these days like to respect these traditions in their modern-day performances.  They are free to do whatever they think appropriate.
Comment: I'm sure there will be other contributors who could answer your last question more accurately than I.
Comment: Thx. I rather suspected that the answer was as you explained. I'm asking because I may play the Cantabile for in church as a prelude, and I'd rather like to have some variation in the repeats. If I tried to create embellishments myself, they'd probably sound rather 20th century.


Should I be concerned of Over-Optimism in Nested Cross Validation with Multiple Scoring Criteria?

Question: Suppose I am using Nested CV to estimate the generalized error of a model and a set of hyperparameters.  I have chosen two scoring metrics of interest: Brier Score and AUC, because I want to get an idea of how discriminative and calibrative my model is.

Because my associated cost requires good calibration, I want to choose/estimate a model with the best brier score. This of course means that within the inner loop I am going to return the hyperparameter configuration that corresponds to the highest brier score, do this for all outer folds as a validation step, and compute a generalized error.  I save this error, and I will then perform vanilla cross validation on the entire data set to 'recover' the optimal hyperparameters (I report the general error as the error of my model and discard the cv scores here
Now, I want to return the hyperparameter configuration within the inner cv fold that maximizes the brier score. But because I care about AUC also, I collect all the hyperparameters configurations that correspond to brier scores within one standard deviation of the maximum brier score and choose the one with the highest auc.
I am familiar with the concept of nested cv for a particular socring rule.  However, my concern is that I am introducing selection bias and biasing the reported scores

My Question is: are my reported errors optimistic by including a second scoring statistic within my model building process ?
Should I do something like:

Triple cv?, where within the inner loop of a standard cv I nest
another single loop that selects the best auc configuration?
Independant Nested CVs for my two scoring rules to estimate generalized
error wrt to a single error metric.  I then refit on the entire data,
perform vanilla cv, return all hyperconfigurations within one std
deviation of the max brier score, and then choose the one with the
highest auc

I should disclaim that most of my confusion is coming from the 'refit' parameter in sklearn, I am not sure if the one standard error rule is being applied within the nested cv structure.
Thank you
Answer: Nested CV is not a model-selection procedure. It does not return hyperparameters. It is a method to estimate a generalization error of a model-selection procedure. It does not matter how you decide to choose your model/tune your hyperparameters in the inner loop.
Just run it once to get an estimate of your generalization error, and then run exactly the same inner loop on the whole dataset to get your final model.
You will need to decide in advance if you would like to use the Brier Score, the AUC, or a combination (you could go for a weighted average or use a model with the highest Brie in a certain quantile of AUC or anything you can dream up) to tune the hyperparameters and then do it exactly in the same way when fitting the final model.


Hosmer-Lemeshow goodness of fit test, P=1?

Question: I'm trying to perform a Hosmer-Lemeshow goodness of fit test on a bunch of logistic regression models but for some reason the p-value that is returned is 1 on all of the models. Does anybody have a clue on why that is happening?
Here's a piece of my R-code:
model1 <- glm(c(variable1,variable2)~x+y+z, family="binomial")
model2 <- glm(c(variable1,variable2)~x+y,   family="binomial")
model3 <- glm(c(variable1,variable2)~x,     family="binomial")

hoslem.test(model1$y, model1$fitted)
hoslem.test(model2$y, model1$fitted)
hoslem.test(model2$y, model1$fitted)

and the output of the tests are:
X-squared = 0.35263, df = 8, p-value = 1
X-squared = 0.035221, df = 8, p-value = 1
X-squared = 0.29097, df = 8, p-value = 1

Comment: We need more information for this to be answerable. What are your variables? How much data do you have? Etc. Can you post your data?
Comment: @gung my data is about 300 rows and 10 columns. The explanaotry variables are numerical data between 0 and 1, and the response variables are numerical data from the natural numbers.
Comment: @MichaelChernick hehe well the library im using in R calls it hoslem.test :P
Answer:
Does anybody have a clue on why that is happening?

I think the test is working as it should. The p values are just being rounded up to 1. E.g.:
> pchisq(0.35263, 8, lower.tail = F)
[1] 0.999965

Whether this test meets your needs is another matter.
Answer: @Matthew's answer is spot-on.  Adding a bit of detail: I found this function in the ResourceSelection package (using library("sos"); findFn("hoslem.test")).  The relevant bit of code reads
PVAL = 1 - pchisq(chisq, g - 2)

the authors should probably be using pchisq(chisq, g-2, lower.tail=FALSE) for greater precision, but that's not really the issue here (we would only really start to run into trouble when $1-p < 10^{-16}$).
The part of the stats:::print.htest function responsible for printing the results reads:
print.htest(..., digits = getOption("digits") , ...) {
  ## ...
  fp <- format.pval(x$p.value, digits = max(1L, digits - 3L))
  ## ...
}

The default value of the "digits" option is 7, so R is printing the result of
format.pval(0.999965, digits=4)

which is "1".
For your most extreme example ($\chi^2=0.035221$), $p=1-3.9 10^{-9}$, so you would need even more digits of precision to see the difference from 1.


User added to a specific group should be blocked

Question: I'm trying to accomplish the task -" If the User is added to a specific CiviCRM group manually by admin/'users with permissions to add to group', The added user should be blocked.I'm using Rules module with CiviCRM entities.
Here is my Rules export:
{ "rules_block_unblock_user" : {
    "LABEL" : "Block\/unblock user",
    "PLUGIN" : "reaction rule",
    "OWNER" : "rules",
    "REQUIRES" : [ "civicrm_entity", "rules", "civicrm" ],
    "ON" : { "civicrm_group_edit" : [] },
    "IF" : [
      { "civicrm_entity_contact_is_in_group" : {
          "civicrm_contact" : [ "site:current-user:civicrm-contact" ],
          "group_id" : "2"
        }
      }
    ],
    "DO" : [ { "user_block" : { "account" : [ "site:current-user" ] } } ]
  }
}

I guess I'm not selecting the right data selectors. Any help?
Answer: Not sure which tokens you have available to select from, but the token site:current-user refers to the user performing the add to group, i.e. some "admin/'users with permissions to add to group". So your Rules Condition seems to be wrong. Due to which your Rules Action is not performed either of course.
And if the list of available tokens doesn't seem to have any relevant token, it may be because you're using the wrong Rules Event.


request.POST.get() gives None in django

Question: I have an ajax call and I caught the request using pdb. request.POST gives me the following value:
<QueryDict: {u'{"layer_id":1,"status":"True"}': [u'']}>

But request.POST.get('status') gives None
Answer: You are not sending form-encoded data, so request.POST is useless. Use request.body.
Comment: but request.body is a string @Daniel
Comment: Yes, it's a JSON string. So use `json.loads()`.

request.POST.get() gives None in django


SonarQube plugin generator for StyleCop not working with SonarC# v6 onwards

Question: I'm currently having an issue getting StyleCop.Analyzers to work with SonarQube. I'm using v6.7 of SonarQube, and I've tried this with multiple versions of the SonarC# plugin.
The recommended way to get the custom rules into SonarQube seems to be to use the SonarQube Roslyn SDK tool (https://github.com/SonarSource/sonarqube-roslyn-sdk) to generate a plugin from a Roslyn analyzer, in this case the StyleCop.Analyers NuGet package.
This seemed to work OK for both v1.0.0 and v1.1.0-beta of the package, and the rules show up in SonarQube for both - however using the latest version of SonarC# the project won't be scanned properly, and the number of C# lines of code & issues found is 0. However if you downgrade to v5.11 of the plugin, the project is scanned properly, it finds the ~2k lines of code, and any StyleCop warnings are shown properly as far as I can tell.
In the VSTS build it looks as though the plugin is working, because the StyleCop warnings do appear in the MSBuild stage (my understanding is that the SonarQube rules override any code analysis rules in the build), but they don't show at all in the summary on the project page in SonarQube.
The SQ Roslyn SDK project says it's compatible with v4.5 or higher of the C# plugin - but it doesn't look like it's being actively maintained.
I can't see any errors in the SonarQube logs, but there are a few warnings in the "Complete SonarQube analysis" task in VSTS:
<code>2018-01-03T15:07:50.0734421Z WARNING: WARN: Protobuf file not found: null\output-cs\encoding.pb
2018-01-03T15:07:50.0734783Z WARNING: WARN: Protobuf file not found: null\output-cs\metrics.pb
2018-01-03T15:07:50.0735152Z WARNING: WARN: Protobuf file not found: null\output-cs\issues.pb
2018-01-03T15:07:50.0735516Z WARNING: WARN: Protobuf file not found: null\output-cs\token-type.pb
2018-01-03T15:07:50.0735888Z WARNING: WARN: Protobuf file not found: null\output-cs\symrefs.pb
2018-01-03T15:07:50.0736289Z WARNING: WARN: Protobuf file not found: null\output-cs\token-cpd.pb
</code>
Not really sure if this is an issue with the C# plugin or the SDK tool - has anyone else had any similar issues? Are there any workarounds other than using an older version of the C# plugin?
Comment: Have a look [here](https://github.com/SonarSource/sonar-csharp/issues/492), It looks like an old issue is returned.
Answer: The SonarQube Roslyn SDK tool is still the recommended way to connect third-party Roslyn analysers to SonarQube. There hasn't been any need to change the SDK recently as the way the generated plugins are handled by the SonarC# plugin has not changed.
I've just tried generating a new plugin for StyleCop, configured the Quality Profile, assigned it to project etc and the analysis worked as expected and uploaded StyleCop issues to SonarQube. I was using the following versions (although I was running the analysis locally rather than using VSTS): SonarQube 6.7; Scanner for MSBuild v22.214.171.1241; C# plugin v6.7; StyleCop 1.0
I think I can explain why the issues appear when you are using v5.11 of the SonarC# plugin but not later versions. v5.11 was the last version had the ability to run the Roslyn analysers itself (the SonarC# plugin had its own exe that would open the C# files, run the Roslyn analyzers against them, and upload the issues). This feature enabled Roslyn analyzers to be used even when using MSBuild v12, which doesn't support running Roslyn analyzers as part of the build. Support for that feature was dropped in version 6.0 of the SonarC# plugin (https://github.com/SonarSource/sonar-csharp/releases/tag/126.96.36.1993).
My guess is that the issue is not with the SDK or the StyleCop plugin. Are you seeing any issues generated by the SonarC# analyzer itself, or any code metrics? If not, then it's likely a more general problem with the build, rather than something specific to the SDK/StyleCop.
Some other suggestions:
* are you using the most recent version of the SonarQube extension for VSTS?

does it work correctly if you run the analysis from the line using the Scanner for MSBuild directly?
try increasing the log level of the SonarQube tasks by setting /d:sonar.verbose=true in the "Advanced/Additional Settings" section of the Begin task.
the output you are describing is consistent with the End step not being able to find the issues files and protobuf files that were created during the build. If you are using a non-hosted build agent, you could try looking at the sonar-project.properties file that is generated by the End task (it will be in the .sonarqube\out folder for the build). Check that the file contains references to *.RoslynCA.json files and that those files exist and contain the expected Roslyn issues. The file should also contain *.sonar.cs.analyzer.projectOutPath entries that point to folders on disc. The protobuf files should be in a subfolder called "output-cs".
finally, you could check for warnings in the MSBuild output as described in https://groups.google.com/forum/#!topic/sonarqube/11YITqrltIk.
Comment: Thanks for this - I haven't been able to completely resolve this, but this was very helpful for troubleshooting. I'm also not able to test v4 of the Scanner, since I'm only using the VSTS task, which seems to use v3. So far, StyleCop does work if used alongside the SonarC# rules (e.g. if you use Sonar Way as a parent profile) but if you disable ALL the SonarC# rules in the profile, it still finds issues, but just won't display the number of lines of code, or the number of duplications.


What's the best choice to get more gain out of a tube amplifier

Question: The story behind this question:
I got myself a nice little combo for my bedroom-workouts (related to playing guitar, of course :D ), a Blackstar HT1-R which is a lovely little 1 watt 2 channel amplifier.
My only problem is, that the overdrive-channel sometimes hasn't enough gain for my needs.
(Since I love to play metal.)
My goal is to use some sort of overdrive/distortion pedal in the already overdriven channel to keep it warm and tube'y (is that an word?), but with more gain.

Is this possible? (I tried something like this with an old transister amplifier quite a while ago and it sounded awful... maybe my choice of pedal wasn't good? It was some kind of a high-gain distortion pedal...)
If yes, what kind of pedal is recommended? (Vintage overdrive-pedal, like a tube-screamer or blues-overdrive?)

I hope someone can answer this, cause I couldn't find a clear answer in this matter on the net.
Comment: What do you mean by "juice" and "gain"? Do you want it louder, or more distorted? Once you've decided, I suggest editing the title and body accordingly.
Comment: @slim more distorted, to be frankly, 1W is more than enough for my bedroom; but the question is obsolete now. I sold the amp a while ago, wasn't flexible enough (Footswitchable; FX Loop etc) in the long run. I'm using an Engl now.
Comment: Mid emphasizing drive pedal (like ts clones) with low drive and high level into dirty amp should give this tight tone most metal bands have.
Answer: You have two options. You can try to simply push the preamp harder, or you can try to put a separate distortion unit in front.
The first option involves using a booster pedal. It does not have to be a dedicated one - I know folks who simply use an OD pedal, such as the Boss SD-1 or OD-3, roll down the drive and turn up the level. The idea is that the higher-level signal you send to the preamp tube, the more it will distort. Since you aren't introducing a lot of break-up from the pedal itself, you'll get more juice, while preserving the tone of your preamp.
The other option creates your distorted tone up-front and you mainly use the preamp as a further amplification stage. Since a lot more of your tone will come from the pedal, you'll have to look around for one that suits your needs. Great-sounding distortion units can get pricy, too, especially if they are tube units.
My suggestion would be to hit your local music shops and simply test-drive a number of pedals on the HT1 - if the store doesn't have one available, ask if you can bring in your own. Try both options: rolling the pedal drive down and turning up the preamp gain and turning up the pedal drive and rolling down gain on the preamp. Ultimately, getting a good distorted tone requires some experimentation and only you can ultimately say what works for you.
If, for whatever reason, this isn't practical, check out some pedal manufacturers' sites and listen to audio samples of their range. You already have a good sounding amp, so if you find a tone you like, odds are it will sound good on your amp too.
Answer: Faza covered the conventional stuff pretty well.  I have a few things to add that wouldn't fit in a comment.
This amp uses all preamp tubes so no matter what it may sound buzzy or fizzy.  The ones I sampled reminded me of Neal Schon with Journey in the 80s.  Very much a preamp distortion.
1) If your amp has a master volume, turn it all the way up and use the preamp controls for changing the volume.
2) Try to find an older MXR Dynacomp compression pedal.  Keep the sensitivity low and crank the output knob.
3) A tube screamer with the drive turned down low and the output cranked.
4) Try to figure out what the output tube is and change it to a higher gain tube.  The ECC82 (12AU7) is a lower gain tube than the ECC83 (12AX7).  Make sure you get competent advice from an amp tech if you decide to do this.  In some cases its ok to substitute tubes (fender all tube amps, typically) in others it is BAD (Crate tube amps, Older tube ampegs in some slots)
Otherwise, let your ears be your guide.
Comment: +1 Thanks for the great answer, but overall this could get a little pricy for me to get all of this. I forgot to mention, that I don't want to spend much more than ~100€. Anyway, very informative answer but I'll try Faza's "method" and try to test just use a single overdrive pedal. btw, the output tube is an ECC82.
Answer: The other guys have covered the bases, but I wanted to chime in with some additional points, one very important caution and some other clarifications.
The HT1R sports an ECC82 (12AU7) + ECC83 (12AX7), and for the life of me I cannot find anything that says which tube lives where in the circuit. If I find anything I'll update this answer to be more factual instead of conjectural. Typically in an amplifier of this configuration the 12AU7 would be a reverb tube and the 12AX7 the preamp tube--but even on the HT1 (which has no reverb) that tube combination is listed so I can't even make an educated guess. Most high gain tube based amplifiers have two to four preamp tubes--and all of them are usually the ubiquitous 12AX7--but the HT1R only lists two tubes with very different impedances and gain ratings, which could be responsible for a lack of gain in the circuit.
I absolutely do not recommend that you attempt to toss a 12AX7 into the spot where the 12AU7 lives on the HT1R. The impedance mismatch could cause some serious damage and at a minimum cause the circuit to run out of range and not operate optimally. Since we're dealing with electricity here, when something doesn't run in tolerance you have two options: release any excess energy in the form of heat and burn up or sag--both of which are not ideal. Push-pull amp circuits with preamplifier stages designed for 12AX7's can usually take a lower gain rating preamp tube (such as a 5751 or a 12AU7), but it hardly works right the other way around.
Another thing to consider is that the HT1R doesn't list any power amp tubes. One of the wonderful things about vacuum tubes is that they clip (or distort) when they get too much signal--and lots of amplifier designs have additional power amp stages that you can overdrive with enough signal. My Orange gets exponentially louder as I roll up the volume and boost the input signal with a pedal--a factor of the preamp stage being overloaded and the power amp stage being overloaded. So, the HT1R will not get as loud as a Mesa Boogie triple rectifier which sports a large array of preamp and power tubes for example but since the power stage in the HT1R is likely solid state it can still produce a good bit gain and definitely phenomenal cleans.
Another typical feature on a metal amplifier is the presence of more than one rectifier tube in the circuit (made famous by none other than Mesa Boogie). A rectifier simply converts AC current to DC current--but these magical little components are responsible for the tightened bass tones you hear on many metal records. A reliable rectifier circuit will keep the amplifier from sagging during playing--which results in a tighter, fuller response as the circuit gets the current it needs to run at full capacity quickly. The HT1R looks like it uses a single, solid state rectifier circuit (silicon diode based); that's perfectly fine for you applications--so no complaints there.
The final point I want to make is that the HT1R only has a single 8" speaker. That's hardly enough to push the blinding tight bass tones that lots of metal guru's are known for. Smaller speakers will have a harder time reproducing lower tones (just like in a car stereo) unless you happen to own an amplifier built by Bose ;D. Picking up a closed back 2x12 cabinet that your amplifier can run will definitely help it, but I don't know of many 1 watt amplifiers with the juice to push a pair of 12's at the correct impedance. If you get one, use it at your own risk.
So what can you do?
Well, you can most definitely help your amplifier out a little bit with pedals--but they'll need to be overdrive pedals. A boost will only do so much, and likely push the gain rating on the preamp stage about as far as it will go. Note that you will need to experiment as Faza suggests when mixing overdrives with stock amplifier distortion unless you are always playing the pedals clean. I don't really recommend doing that though because it will really kill the usefulness of the drive stage on the amp and whack most of the tube amp warmth that most of us are in love with. Most distortions sound very sterile unless they are tube based, germanium fuzzes, or hardcore algorithms on a complex modeling unit like AxeFX. There are definitely some awesome JFET based units out there that will fool you though--but the list is very short in my experience.
I have gone through countless overdrives (I recommend you buy used :D) and finally settled on a two overdrive + one boost configuration that I fell completely in love with, so it may take you some time. Be patient. Keep experimenting, and keep playing. My amp doesn't gain out that much either--it's a dual 12AX7 + 4xEL84 configuration, but after discovering a pattern of pedals that I liked and grabbing a closed-back* 2x12 speaker cabinet I can sound like most vintage and some modern metal gurus.
I'm not trying to be a downer here and say you bought a bad amp--but I definitely think that you should understand the shortcomings of the amplifier you purchased versus what you want it to do. Based on my one hour plus of research on the little guy it sounds like you have a really solid practice amp that will last you for a good while--but if you really want to sound like a metal god then you're going to have to pony up and get something louder, larger, and with a power amp stage :D.
*A closed back cabinet is also helpful in recovering some tightness and bass response
Comment: Useful and interesting answer, thank you. But, It doesn't needs to be THE ideal metal tone, it just needs more gain (it already has -> I just added a overdrive- and a lower-gain distortion-pedal in front (the overdrive pushes the distortion quite nice through the clean-channel imo)). I might add a 2x12 w/ V30's, because I heard that it really works well with 2x12 and 4x12 cabinets. In fact I tested it with a 2x12 box right after I bought the amp. So again, my goal was to get a usable amplifier for bedroom use (so 1W -> obvious choice). For "proper" equipment, I have my band session room.
Comment: If V30's means Vintage 30's then they will most certainly get you a little more breakup. I had them on my Orange and they broke up too much for my tastes. A closed back 2x12 cabinet will most definitely get you more towards what you are looking for. I was mainly trying to highlight that you will be making trade offs with an amplifier of that size and configuration :).
Comment: Also, unless Blackstar has invented a new way of building amplifiers I don't quite understand how a 1 Watt amplifier has enough load to optimally drive a pair of 12" speakers at 30 watts--whether in series or parallel. I guess it could theoretically do it, but I think it might be under powering the cabinet (which is safer than overpowering it, but I dunno how I feel about that exactly). If it works for you then go for it though!
Comment: thanks, your tips are highly appreciated. I will certainly test some cabinets first before I make a final decision. I'm just curious about a v30 equipped 212 box because I found this one: http://www.thomann.de/de/harley_benton_g212_vintage.htm - It's available in europe only (np for me, I live in germany), and I often heard of the box, that it has an insanely good value for the money. (pro box for pocket money) I'm not cheap, especially when it comes to guitar equipment, but I try to save money whenever it's possible. So I'll definitely try this one out. :D
Comment: How did the blackstar ht1 with the 2x12 v30,s work out? and what pedal do you use?
Answer: I just got a blackstar ht-5r combo tube amp and it sounds great with my distortion pedal, it's a Biyang Metal End King pedal. If you like playing heavy metal then this pedal will give you any type metal sound you want, very heavy, fat tones and not muddy like a lot of pedals! Cheap too, only $79. I play anything from black metal, thrash, or death metal and it's plenty heavy enough for any of it. That's my recommendation
Comment: well my question isn't that new, and in fact I don't own my HT-1 anymore. I'm now using an ENGL e530. But thanks for the answer.
Answer: My advice, and not to be terse about it, get an amp with some decent wattage.
These little watt amps do not have the depth and headroom because there is no wattage to push the sound.
Wattage is not about loudness, as so much marketing hype crap suggests.
Wattage is headroom, punch, power and low end body. The distortion is tinny and thin in these amps and that is that. There is no power amp "power".
Get at least a 40 watter going on a clean channel and use a good high end distortion pedal if you want metal gain that sounds good.
Most cannot afford a $3000 head and fry the tubes creating a gain level. Just just a decent size amp, metal is best served with a closed back cab and 12" speakers. More depth and punch.
Wattage is not loudness.
The basic rule of sound engineering is there is only a +3db potential of loudness for every doubling of watts. A 10watt is 3db louder than a 5watt, a 20watt only 3db louder than a 10watt. So a massive 100watt head is really only like 10db in potential loudness than a small 10watt amp. 10db is not much and all the removal of watts does is remove the heart and soul out of an amp. That is just the hard truth. Those who disagree are just wrong and their tone shows it. It's not about your ears or ones opinion it is simply a matter of sonic engineering law. There is no low end without wattage to push it.
Answer: The HT1-R is using a dual triode ECC82 pre-amp tube in push pull as a low wattage output tube. ECC83 has higher voltage gain but lower power output than an ECC82 so I don't recommend substituting the output tube!!! You could try nodding the amp by altering cathode resistor and bypass capacitor and anode resistance.
Comment: Hello Jonathan and thanks joining the Music.SE site! Could you elaborate for less informed users (like me) on what specific alterations would do the trick? Are you suggesting that bypassing the capacitor and anode resistance will do the trick because it will increase the voltage over the cathode resistor? Could you add some instructions on how someone would go about making these modifications? (Did you mean "modding" the amp rather than "nodding" the amp?) Thanks for sharing your expertise!


Harmonic Series and Its Divergence by Abel Sum and Cesaro Sum

Question: Already I know that harmonic series, $$\sum_{k=1}^n\frac1k $$ is divergent series.
And, it is also divergent by Abel Sum or Cesaro Sum.
However, I do not know how to prove it is divergent by concept of Abel or Cesaro.
Abel Sum or Cesaro Sum do not exist in this problem.
But, how can I prove it?
I have tried to prove for long time...
Can you please show your work?
Answer: A simple trick is to consider that:
$$\frac{1}{n}\geq\log\left(1+\frac{1}{n}\right)\tag{1}$$
holds regardless of which concept of convergence you are using, and the partial sums of the RHS of $(1)$ are fairly easy to compute by the telescopic property:
$$ \sum_{n=1}^{N}\log\left(1+\frac{1}{n}\right)=\log\prod_{n=1}^{N}\frac{n+1}{n} = \log(N+1).\tag{2}$$
As an alternative, given that $H_n=\sum_{k=1}^{n}\frac{1}{k}$, we have:
$$ \frac{H_1+\ldots+H_n}{n}=\frac{1}{n}\sum_{j=1}^{n}\sum_{k=1}^{j}\frac{1}{k}=\frac{1}{n}\sum_{j=1}^{n}\frac{n+1-j}{j}=\left(1+\frac{1}{n}\right)H_n-1.\tag{3}$$
Comment: I see your idea. lim [x^Nlog(N+1)] does not exist, right?

And, eqaution (3) is for Cesaro. But, If we get the fact that Abel sum is not possible, we naturally would know Cesaro Sum is also not possible. Right?
Comment: @qyong: yes, right.
Answer: Since the term of the Harmonic series are of O(1/n). Suppose, if the Harmonic series is Cesaro summable, then by Hardy's theorem, the series is convergent, but Harmonic series is divergent. So, Harmonic series is not Cesaro summable.
Answer: Because $H(n)>\ln(n+1)$, we want to show that $$\lim_{n\rightarrow\infty}\frac{\ln((n+1)!)}{n}$$diverges, which was derived by applying the addition to product rule. Because $\lim_{x\rightarrow\infty}\frac{x!}{e^x}=\infty$, this limit diverges.


Passing a node refernce id to a form via hook_form_alter()

Question: using Drupal 7
I'm trying to write a module that alters the node add form of a certain content type. This content type has a node reference field that normally displays as a drop-down. I'm also attaching this node add form to the bottom of the content it references, what I'm trying to do is populate the node reference field by using hook_form_alter() and passing in arg(1) of the page (passing it the nid of node I want it to reference).
I'm trying to figure out exactly how to do this. I've tried to do a drupal_set_message('
'. print_r($form, TRUE) .'
'); on the form to see how exactly to pass in the node reference, but it just prints out a huge nested array, and I'm unable to really figure out how I would format that as input I know it's going to look something like this;
function mymodule_form_alter(&$form, &$form_state, $form_id){
    if($form_id == "my_node_form")
    {
        $form['title']['required'] = FALSE;
        $form['node_reference_field_name']['nid'] = arg(1);
    }

}

but I'm still really confused on the actual syntax
Answer: ok, figured this out incase anyone is interested, may only work for my use case but perhaps it will help someone else.
function mymodule_form_alter(&$form, &$form_state, $form_id){
    if($form_id == "rating_node_form")
    {
        // dont require title
        $form['title']['required'] = FALSE;
        // validate
        $form['#validate'][] = 'mymodule_form_validate';
        // call after form is built
        $form['#after_build'][] = 'mymodule_after_build';
    }
}

function mymodule_after_build($form, &$form_state) {
       //set the nid to the argument of the nide you are viewing
       $form['field_bar_name']['und']['nid']['#value'][0] = arg(1);
       //set the default value
       $form['field_bar_name']['und'][0]['nid']['#default_value'] = arg(1);
       //set form value
       $form['field_bar_name']['und'][0]['nid']['#value'] = arg(1);
       return $form;
}

Comment: also, for some reason you need to have the node reference url widget enabled, I suppose this has something to do with arg(1) being available.


Has the Bach Compendium been completed?

Question: The Bach Compendium (BC) is a multi-volume attempt by renowned scholar Christoph Wolff and others to replace the traditional but obsolete BWV numbering system. It is my understanding that the first four volumes (work groups A through H, T, and V, encompassing all of Bach's vocal production) have been published in the 1980s, but after that the project appears to have been discontinued.
However, according to this page and other online sources (even Spotify), work groups J, K (organ music), L (keyboard and lute music), and even Q (didactic works) seem to have been compiled, at least partially. These letters roughly correspond to the contents of the fifth volume of the series, which is nowhere to be found.
Where has this part of the BC catalogue been published?
Comment: I can't find any evidence than the chapters J K L Q as you say in IMSLP. But they must be able to tell you so: (look up the tel. Nr.) https://www.neue-bachgesellschaft.de/bach-jahrbuch-2010-2/
Answer:
Where has this part of the BC catalogue been published?

After doing some research on this, I'd say it has not been "officially published." That is, I can't find any evidence that a completed version has been published by one of the recognized houses. I did however find numerous reputable sources stating the BC has not been completed past H.
Regarding that IMSLP listing, my guess is they assigned those numbers on their own. It is "manually maintained," not aggregated or copied from another source. The alphabetical categories had already been determined, and the works had already been ordered by BWV, so assigning corresponding BC numbers seems elementary.
All that being said, when IMSLP (or anyone else) posts online that is a form of "publishing" and IMSLP is highly regarded in classical music circles. Therefore one could argue that IMSLP is the answer to your question of "where."
I did find this link, which was described as being a "complete" listing of BC numbers - yet it seems to be a dead end (and actually has a few broken links, despite being managed by a Bach institute):
https://www.vifamusik.de/metaopac/search;jsessionid=122080EEB517D9E4CC9ECF75EC514D11.touch01?id=als86437vifamusik&View=mus

Has the Bach Compendium been completed?


Openstack Neutron on top of Amazon VPC

Question: Is it possible to run Openstack Neutron within an Amazon VPC and still be able to use all of Neutron's features (GRE mesh network, etc)?
I do know about Amazon's ENI and networking tools within a VPC but they have limits that are too small for our application and seem to be strictly enforced (and can't be expanded).
Comment: What limits are a problem for you? AWS lets you request nearly any limit be raised, you just have to explain why. I've never had trouble and it usually happens within 24h.
Comment: The limits I would be running into are the limits on number of ENI interfaces per instance type and the number of IPs that can be associated with those interfaces. It didn't mention a way of requesting an increase for those
Comment: Larter instances have more ENIs. You might want to contact AWS Support and ask if they can help you with your use case.
Comment: If I may ask, why the downvote? So far, the original question remains unanswered and I think it is still a valid question that I have not been able to find an answer to online.
Comment: The downvote is not mine.
Answer: Since nobody had any suggestions, I tested it out using Packstack on two CentOS instances within a VPC and was successfully able to have Neutron create GRE tunnels between compute nodes and was able to communicate over the tunnels between hosts.

Openstack Neutron on top of Amazon VPC


Homomorphism in Van Kampen Theorem

Question: I am having difficulty understanding the first homomorphism in the van Kampen Theorem (see Hatcher pp 42-43).  I will illustrate my confusion with the simple example of an annulus ($X$), which will be covered by two subsets ($A$ and $B$), each of which include the central opening and constructed so that their intersection is path connected, and so that each contains the base point.
The fundamental group of both $A$ and $B$ is $\mathbb{Z}$.  The free product of the fundamental groups of $A$ and $B$ is the argument of the homomorphism, and this free product is $\mathbb{Z}\star\mathbb{Z}$.
Now, my question:  How does the homomorphism map $\mathbb{Z}\star\mathbb{Z}$ to $\mathbb{Z}$, the fundamental group of the annulus?
If I understand the definitions correctly, a word $g_1g_2$ in the argument will be mapped to $g_1g_2$.  But the range of the homomorphism is $Z$, the fundamental group of $X$, and $g_1g_2$ is not in $Z$.
(P.S. I see, of course, that words like $g_1g_2$ can be reduced with the knowledge that there is only one hole in $X$, and so there never were two loops after all.  But this presupposes an understanding of the structure of $X$.  And, if one already understands $X$, there is no reason to resort to the van Kampen Theorem in the first place.)
Comment: Could you clarify?  What exactly are $A$ and $B$?  Cocentric annuli?  What do you mean by mapping the word $g_1 g_2$ to $g_1g_2$, if you then claim that $g_1 g_2$ is not in $\mathbb{Z}$?  Also, note that van-Kampen involves the intersection of $A$ and $B$, which seems to be absent from your question.
Comment: Think of A and B as being almost the same as the annulus, but missing a sliver on the left or the right.  The map of g1g2 that I refer to is the homomorphism that Hatcher refers to in the first sentence of his statement of the theorem.  I note that the intersections are path-connected, and both cover the base point and the opening in the center.  Otherwise, the intersections are not used in the first sentence.
Answer: Here is a step-by-step execution of your example:
Seifert-van Kampen (as in Hatcher, adapted). If $X$ is the union of path-connected open sets $A$ and $B$ each containing the basepoint $x_0 \in X$ and if $A\cap B$ is path-connected, then the homomorphism $\Phi : \pi_1(A)∗\pi_1(B)\to\pi_1(X)$ is surjective. The kernel of $\Phi$ is the normal subgroup $N$ generated by all elements of the form $i_{AB}(\omega)i_{BA}(\omega)^{−1}$ for $\omega\in \pi_1(A\cap B)$, and hence $\Phi$ induces an isomorphism $\pi_1(X) \cong  \pi_1(A)∗\pi_1(B)/N$.
Here, $i_{AB}$ is the homomorphism induced by the inclusion of $A\cap B$ into $A$, and similar for $i_{BA}$.
(Note that the second sentence originally has the condition that the triple intersections are path-connected, which is trivial here.)
In general, this is good news because for example, when your intersection is simply connected, your $i_{AB}$ and $i_{BA}$ must be trivial maps, so $N$ must be trivial. What you have to know is not $\pi_1(X)$ here, but rather $\pi_1(A\cap B)$. In your example, these are isomorphic, whence it is actually not too exciting for that special case.
Now to the particulars: What we have to know is $\pi_1(A)\cong \mathbb{Z}$, $\pi_1(B)\cong \mathbb{Z}$, $\pi_1(A\cap B)\cong \mathbb{Z}$. Let $a$ be a generator of $\pi_1(A)$, $b$ a generator of $\pi_1(B)$. These can be thought as fixed loops, as Michael described, say counterclockwise. Then the elements of $\pi_1(A)*\pi_1(B)$ look like, for instance, $\emptyset$, $aaabbba$, $abbaba$, $a^-b^-aab$, etc. Here with $a^-$ I denote $(-1)a$ (inverse), and if you feel more comfortable, you may write $a^3$ instead of $aaa$, etc.
Now the question is: What does $i_{AB}:\pi_1(A\cap B)\to\pi_1(A)$ do? All loops in $A$ are homotopic to some loop in $A\cap B$ relative endpoint (again, sort of boring because $A$ and $A\cap B$ look "too alike" for it to be interesting), so this map is an isomorphism. Take $\omega$ to be the preimage of $a$. We can easily see that $i_BA(\omega)=b$ (here we "use the knowledge" from $B$ and not from $X$!) So, the same applies for $i_{BA}$ and $\pi_1(B)$, except now we are taking the inverse of $i_{BA}(\omega)$ into account. when calculating $N$. $N$ is generated by $ab^-$, so we will "shorten" by $ab^-=0$, which you can basically think of as not differentiating between $a$ and $b$ (since $a=ab^-b\simeq b$). Then evidently, the free group modulo this identification is $\cong \mathbb{Z}$, given by word length after the identifications.
Summary: You indeed need to know the fundamental group structures of your two subsets that cover your space, and of their intersection. In this particular example, it is tempting to think that we are using the whole space's knowledge where we actually always use that of either one part, or the intersection. But all those groups/spaces were really similar here, so Seifert - van Kampen would be a real overkill. :)
Comment: Got it.  Thanks.
Answer: Let $g_1, g_2$ be the generators of $\pi_1(A)$, $\pi_1(B)$ given by counterclockwise loops around the central hole, and let $h$ be the generator of $\pi_1(X)$ given by a counterclockwise loop around the central hole. Then the map $\pi_1(A)\ast\pi_1(B) \to \pi_1(X)$ induced by the inclusions $A \hookrightarrow X$ and $B \hookrightarrow X$ is given by $g_1 \mapsto h$, $g_2 \mapsto h$. Therefore, the word $g_1g_2$ is mapped to $h^2$.
Comment: I see this, but it requires prior knowledge of the FG and nature of X.  So, what is van Kampen telling me that I didn't know before?
Comment: Well this example you've chosen isn't particularly insightful. The theorem is usually employed to calculate the fundamental group of $X$ (which you don't already know) in terms of $\pi_1(A)$ and $\pi_1(B)$.
Comment: Right, but even this simple example requires that I already know the FG of X.  In more complicated cases, it would be impossible. (I'm not trying to be argumentative, I'm just trying to really understand what the first sentence of the theorem is telling us.)
Comment: I disagree that you need to know what $\pi_1(X)$ is. The first sentence of Theorem 1.20 tells you that $\pi_1(A)\ast\pi_1(B)$ surjects onto $\pi_1(X)$. By the first isomorphism theorem, you only need to find the kernel to determine $\pi_1(X)$, and this is described by the second sentence of Theorem 1.20. In this case, it tells you that the kernel is the normal subgroup generated by $g_1g_2^{-1}$. Hence, $\pi_1(X)$ is isomorphic to the group $\langle g_1, g_2 \mid g_1 = g_2\rangle \cong \langle h\rangle \cong \mathbb{Z}$.
Comment: Maybe I am trying to get too much out of the first sentence, before moving onto the rest of the Theorem.

Homomorphism in Van Kampen Theorem


How can I identify a step in an increasing time series?

Question: The time series data, as shown in image 1, has 3 separate steps connected by increasing data. How do I detect these steps?
An easier set of data with one step could be created in R as follows:
x <- seq(1,220)
y <- c(seq(1,100),rep(100, times = 20),seq(101,200))
plot(x,y)

The output is shown in image 2. How would I detect this step? I think a solution to this simple problem would be similar to finding the steps in the earlier dataset.
Answer: There are many ways. For the sample with R code, it's as simple as differencing then analyzing the new series, e.g. see this code and the plot:
dy=diff(y)
plot(x[2:length(x)],dy)

This may not work for the real data though, if the data generating process is different from your sample code.
Comment: It does turn into a step problem if I smooth it first using moving averages. Would you happen to know how I find the optimum window size for moving averages? For example, how do I defend using a window size of 5% of the data vs 10% of the data?
Comment: I don't think that the window size can be wider than the step width.
Comment: @SPV: A rolling mean is essentially a low pass filter. That is, it attenuates the amplitude of frequencies above a certain point. A filter like this can therefore be used to smooth out high frequency noise, while retaining a lower frequency signal. In general terms, you want the window to be of a length so that it attenuates as much noise as possible, without distorting the signal you are interested in. As such Aksakal's tip is a reasonable one.


Effective ways of muting saxophone

Question: The question has been partially answered here before but for an inexperienced, self-taught musician who lacked the skills to control volume through embouchure adjustments.
I was wondering if there were any effective ways of reducing the played volume of saxophone (alto and "down", soprano and "up" doesn't really need it that much) to make it possible to play with other musicians or practise at home without disturbing the neighbours too much.
I have jammed a number of times with a friend who plays harmonica. He put his speaker up to a 100%, and we still had to play in different rooms. I had to ditch the low end and upper end/altissimo as well, as they require a bit more airflow to work properly. We plan on playing outside eventually, and neither one of us really knows how that will happen given the huge volume difference.
My opinion on saxophone mutes suggested as a solution for the other question is that they either don't work or mess up the resonance of the instrument where it's like you're playing a different horn. The same effect can be achieved by sticking a sock down the bell, as my teacher likes to joke.
The majority of my experience comes from playing with big orchestras, so I have not encountered that problem before.
One possible solution would be to play an EWI instead, but I would still like to know if anything could be done about a regular instrument. There's plenty of discussions on the net, but I personally couldn't find any solutions.
Comment: I pretty much have the same problem of being 'too loud', i live in a ground floor apartment and my upstairs neighbour goes ape even if i answer the phone and my ceiling becomes lke a bass drum. I could buy something to try and mute; but realistically i think my only other option is in a rehersal studio.
Comment: @MarkCook, your neighbour is not a very good person. I'm sorry you have to deal with someone like that. My neighbours have been nothing but thankful for my regular practice sessions (I'm not joking).
Answer: First of all, there are sax mutes which are basically small plush pillows to jam into the horn.
However,  the proper answer is, you guessed it:   practice, practice, practice.  It does take a lot of strength and training to play almost any wind instrument pp and maintain good tone.
One other mechanical thing to try: get a plastic or rubber mouthpiece with a relatively narrow face gap (opening between reed and mouthpiece).  That will make it easier to play quietly at the expense of top dynamic range.  (as opposed to a honkin' awesome Otto Link metal :-)  )
Comment: I know saxophone mutes exist, but they're useless, in my opinion. They tend to muffle the low end only or change the dynamics way too much and make playing extremely uncomfortable (the expensive full-body ones). My mouthpiece is definitely a problem. I play a hard rubber 8* 10MFAN Merlot which has a ridiculously huge chamber. Thank you very much for the advice, it's very helpful. I will invest more time into practising pianissimo playing.

Effective ways of muting saxophone


How do I only show products with a specific attribute based on a session var in Woocommerce?

Question: I'm building a zipcode/postcode based store in which you can only see products that are within a certain zipcode range. The zipcode range is an attribute of the product itself to make things easier. The costumers zipcode is stored in session (put in by a form) How would I prevent my costumers to see products that are not able to be bought in their zipcode? I'm building a child theme and would like to use code to solve this issue (no extra plugins).
Answer: Hook into woocommerce_product_query, here is an example:
<code>function so_42540393_product_query( $q ){
    if(isset($_SESSION['ZIPCODE-RANGE'])){
    $meta_query = $q->get( 'meta_query' );
    $meta_query[] = array(
        'key' => 'CUSTOM_META_KEY',
        'value' => $_SESSION['ZIPCODE-RANGE'],
        'compare' => 'BETWEEN'
        );

    $q->set( 'meta_query', $meta_query );
    }
}
add_action( 'woocommerce_product_query', 'so_42540393_product_query' );
</code>
This will remove them from all the stop archive pages. If they have a link for a product that is outside their range they would still be able to visit that page, but I assume that wouldn't be a problem? This is example code, so make sure you set the meta query and the session right. Does this help?


How to combine two filters in search api views?

Question: I have a search index which is indexing both content and files.
I have configured the search api views which displays the search results.
On this view, I have a filter on type which displays the content types as links using better exposed filter.
Is there any way to add another option to the same filter files but clicking on it will display files in the search results only.
Is it possible? Any custom solution hint?
I have tried the combined global filter but that only works with fields not with rendered entity.
Answer: I used an aggregated field of search api. And combined the Content: content type and Media: bundle fields.
And used that field in the search api views filter.
But the filter was a text field. So to display it as links, I have created a  custom views filter for that aggregated field.


Data and model blocks in JAGS

Question: I am trying to model a function of observed data in JAGS. For example
<code>   x[i,t]/phi ~ dpois(mu)
</code>
where x is observed and phi is a parameter in the model. I can't seem to get this working and am wondering if I may be missing some larger issue in JAGS.  I am pretty sure you can't have any transformation of the data in the model statement and need to use a data block but that doesn't seem to be dynamic, thus not allowing for me to fit phi.
Any ideas?
Comment: I believe you cannot fit the model that you have described: to my knowledge in JAGS you are not able to scale a distribution. What distribution $\phi$ is supposed to follow? If it is simple enough, you can derive the distribution of $x$ directly and sample from it.
Comment: Can you please add the formulas for your model?
Comment: it is a pretty convoluted model and I think it will complicate the question.  Is my question unclear?
Answer: So here the answer from the comment:
I believe there is no way to directly sample from a scaled distribution in JAGS, except the cases where the scaled distribution can be expressed as a standard distribution.
This being said there is trick, the so called "zero trick", but I have not used it so far. It might be useful in your case...

Data and model blocks in JAGS


repeated cross-validation shows similar results for different k-folds

Question: I need to see whether different k-fold cross-validations with repeats will improve my classification accuracy.
For that, I have tested with repeatedCV (k-fold=50, repeat=10, k-fold=100, repeat=10, and k-fold=1000, repeat=10)
But, accuracy values in the confusion matrix are same for the above different repeated cross-validations.
Please see my code below. I am using caret package for training the data.
I am thinking whether I am getting the same accuracy results because my my model is stable/robust or due to some other reason.
Any help is appreciated.
Thanks in advance.
standardized.X <- LossT[,-c(1:10)]
set.seed(50)
ind <- createDataPartition(LossT$LRPPcat, p=0.85, list = FALSE)
train.X <- standardized.X[ind,]
test.X <- standardized.X[-ind,]
train.Y=LossT$LRPPcat[ind]
test.Y=LossT$LRPPcat[-ind]

set.seed(123)
seeds <- vector(mode = "list", length = 501)
for(i in 1:501) seeds[[i]] <- sample.int(1000, 20)
## For the last model:
seeds[[501]] <- sample.int(1000, 1)

# Train the model
ctrl <- trainControl(method = "repeatedcv",number = 50, repeats = 10, preProcOptions = list(thresh = 0.85, k = 5),seeds = seeds, selectionFunction = "best", savePredictions = TRUE, allowParallel = TRUE)
modelknn <- train(train.X,train.Y, method = "knn",preProcess = c("pca"), trControl = ctrl,metric = "Accuracy",tuneLength = 20)

fitted.results <- predict(modelknn,test.X)
cm <- confusionMatrix(data=fitted.results, reference=test.Y)

Comment: I think you have a misunderstanding about what cross validation is for. K is not a tuning parameter in a model, it does not change the complexity of the model being fit to the data. Different k's give you different estimates of the same quantity, for the same model, for the same population.
Comment: Thanks for your reply and appreciate it. But, when used "knn" the numbers of nearest neighbors (k) is the tuning parameter. That's why I have put the tuning length there. But, I cannot understand the reason why for different repeatedCV gives me the same classification output results. Any idea? Thanks again.
Comment: As Matthew Drury pointed out, the $k$ in CV is not a tuning parameter. Conversely, the $k$ in KNN is. There is no relationship between the two. Perhaps you might want to review a textbook on these matters.
Comment: @Ami, Thanks for your comment. I totally understand it. Do you think my code needs to be revised? If yes, where should I change? Please let me know. Thanks a lot again.
Answer:
But, accuracy values in the confusion matrix are same for the above different repeated cross-validations.

This is how it should ideally be. The actual choice of the number of folds does not influence the results much, see also Choice of K in K-fold cross-validation

I am thinking whether I am getting the same accuracy results because my my model is stable/robust or due to some other reason.

The aggregated confusion matrix (summing up the results of all surrogate models) is not the best place to characterize stability [of the predictions wrt. small changes in the training set]. You can get that more directly by comparing predictions across the repetitions.
The "other reason" may be that you have many cases (as you do k = 1000), so the random (relative) differences you observe in the final result are typically smaller than those observed with small data sets.


How to retrieve the first added node when a LinkedList is created?

Question: I have created a linkedlist in Python using below classes:
class: Node
class Node:
    def __init__(self, data):
        self.data = data
        self.next = None

class LinkedList:
class LinkedList:
    def __init__(self):
        self.head = None
    def insert_at_beginning(self, data):
        temp = Node(data)
        if self.head is None:
            self.head = temp
        else:
            temp.next = self.head
            self.head = temp

    def insert_at_end(self, data):
        temp = Node(data)
        if self.head is None:
            self.head = temp
        else:
            curr = self.head
            while curr.next is not None:
                curr = curr.next
            curr.next = temp

    def insert_at_position(self, pos, data):
        temp = Node(data)
        if self.head is None:
            self.head = temp
        else:
            curr = self.head
            i = 1
            while i < pos:
                curr = curr.next
                i += 1
            temp.next = curr.next
            curr.next = temp

    def traverse_list(self):
        if self.head is None:
            print('List is empty')
        else:
            curr = self.head
            while curr.next is not None:
                print(curr.data)
                curr = curr.next

Below is the order I am calling the insertion methods.
ll = LinkedList()
ll.head = Node(1)
ll.insert_at_beginning(data=10)
ll.insert_at_beginning(data=11)
ll.traverse()

The elements printed in the output are: 11 & 10 which are the ones I inserted but I don't see the value 1 which is a node I added in the beginning.
Is there any logic I missed here that is making the first value skip or not being considered ?
Comment: Tried any debugging yet? What were your findings?
Answer: Your problem comes from while curr.next is not None: in traverse_list(). Assuming you're pointing your last node, its next node is obviously None. Therefore, it will stop the loop before printing the last node's data.
You can try this:
def traverse_list(self):
    if self.head is None:
        print('List is empty')
    else:
        curr = self.head
        while curr is not None:
            print(curr.data)
            curr = curr.next


mysql-apt-config takes a verry long time to install on Ubuntu 14.04.1

Question: i'm trying to upgrade mysql from 5.6 to 5.7 using mysql-apt-config.
i've downloaded the deb and trying to install it.
but it 'hangs' on
Setting up mysql-apt-config (0.3.2-1ubuntu14.04) ...

and it stays there for hours. nothing happens.
could this be the issue?: https://unix.stackexchange.com/questions/158052/how-to-configure-the-mysql-apt-repo-on-ubuntu-on-a-non-interactive-shell
that i'm not getting the selection menu, because i'm installing through SSH
Answer: I suffered with the same problem and I found the solution at the link that you mentioned:
export DEBIAN_FRONTEND=noninteractive
echo mysql-apt-config mysql-apt-config/enable-repo select mysql-5.6 | sudo debconf-set-selections
dpkg --configure -a

Don't forget to check your MySQL Version!
Thanks to you!


Como publicar em uma pagina do Facebook utilizando a SDK PHP (5.0)?

Question: Estou desenvolvendo uma aplicação para agendar publicações de hora em hora para minha pagina do Facebook, já consigo efetuar o loguin do usuário obter ID Token de acesso do usuário e  tudo mais, tenho as permissões necessárias no aplicativo do Facebook, e já consegui ate postar mensagens com imagem no perfil do usuário, porém não consegui nada relacionado a paginas, todos os codigos que encontrei não funcionaram.
Preciso obter a lista de paginas que o usuário gerencia, obter o token de acesso(prolongado) de uma determinada pagina e publicar fotos no Facebook com o perfil da pagina (Não quero publicar links, quero publicar apenas imagens com legenda).
Alguém poderia me ajudar?
Answer: Você tem que gerar um Token específico para página. Tem no Facebook pra fazer isso.
Depois disso são os mesmos processos para fazer um POST num perfil, com a diferença na URL.
<code>$fb = new Facebook\Facebook([
   'app_id'                => [APP_ID],
   'app_secret'            => [APP_SECRET],
   'default_graph_version' => 'v2.7',
]);

$linkData = [
    'link'          => '',
    'caption'       => '',
    'description'   => '',
    'name'          => '',
    'message'       => '',
    'picture'       => ''
];

$pageAccessToken ='';

try {
    $response = $fb->post('/NOME_DA_PAGE/feed', $linkData, $pageAccessToken);
} catch(Facebook\Exceptions\FacebookResponseException $e) {
    exit;
} catch(Facebook\Exceptions\FacebookSDKException $e) {
    exit;
}
$graphNode = $response->getGraphNode();
</code>
Comment: Muito obrigado! Consegui postar utilizando um token obtido na https://developers.facebook.com/tools/explorer/ porém este token dura apenas uma hora.

Você sabe como obter um token dentro da própria aplicação?
Comment: Tem um lugar aí que você pega o Token eterno, dá uma lida na página que pega o Token... https://developers.facebook.com/tools/debug/accesstoken
Comment: Essa parada do token eterno pelo que pesquisei foi descontinuada, os tokens atuais duram por aproximadamente dois meses, o token de 2 meses já me ajuda, mas o ideal seria conseguir um token pela própria aplicação,vou continuar pesquisando aqui, muito obrigado!
Comment: Cara, eu uso uma aqui... nunca deu problema, desde junho.
Comment: Tem que clicar no botão **Extends Access Token**. Quando clicar vai aparecer uma frase e um Token. **This new long-lived access token will never expire:**
Comment: Consegui o token de acesso eterno para a pagina, valeu mesmo @Gumball
Você tem resposta para está pergunta > http://pt.stackoverflow.com/questions/172745/como-mencionar-uma-pagina-ou-perfil-do-facebook-com-sdk-php


Is there any way in drupal to get all primary links through views?

Question: My objective is to get the primary links or any kind of menus items, to be shown as JSON, so that an app could consume it and show it.
I tried it using views and I have services module and views_datasource as well, but no success.
If there is no easy way then I am open to build a custom module as well.
Any suggestions?
Answer: Can you create a hook_menu() and then in the callback do a https://api.drupal.org/api/drupal/includes!menu.inc/function/menu_tree_all_data/7 to get the menu tree and then expose that to services or return  json.
Comment: I was thinking about it, but could you suggest me a good tutorials to expose my module to services, or how does integration with services module work?
Comment: you can create a custom services resource here https://www.drupal.org/node/783460 or the cheap way is to return json from your hook_menu()
Comment: Well..I would have returned json as a last resort, but I will play with this services stuff first. Thanks :)


Shopware 6 Custom plugin that changes the schedule task interval value

Question: I want to create a plugin that can set the scheduled task interval. Shopware has database field called run_interval in Schedule_task table. The admin will enter a value, this value will change the run_interval in the scheduled_task table. Is this possible? How can I do that?
Answer: I'm assuming you want to use a plugin configuration for the input field. You could then set up a subscriber to listen for changes of the config field and update the scheduled task entity.
Service definition:
<service id="Foo\MyPlugin\Subscriber\SystemConfigChangedSubscriber">
    <argument type="service" id="scheduled_task.repository"/>
    <tag name="kernel.event_subscriber"/>
</service>

Subscriber:
class SystemConfigChangedSubscriber implements EventSubscriberInterface
{
    private EntityRepositoryInterface $repository;

    public function __construct(EntityRepositoryInterface $repository)
    {
        $this->repository = $repository;
    }

    public static function getSubscribedEvents(): array
    {
        return [
            SystemConfigChangedEvent::class => 'onSystemConfigChanged',
        ];
    }

    public function onSystemConfigChanged(SystemConfigChangedEvent $event): void
    {
        if ($event->getKey() !== 'MyPluginName.config.nameOfMyField') {
            return;
        }

        $context = Context::createDefaultContext();
        $criteria = new Criteria();
        $criteria->addFilter(new EqualsFilter('name', 'name_of_the.task'));
        $taskId = $this->repository->searchIds($criteria, $context)->firstId();

        if (!$taskId) {
            return;
        }

        $this->repository->update([
            [
                'id' => $taskId,
                'runInterval' => $event->getValue(),
            ],
        ], $context);
    }
}

Comment: I followed all the steps, it worked. Thanks a lot!


Connection closed by 127.0.0.1

Question: i am working here with hadoop framework . i installed hadoop before from this sites : http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html. it works before for me. but now i have to install it in another pc. i tried same steps but stucked here
at command :  ssh localhost
  it shows me error:  connection closed by 127.0.0.1
i tried to googled many times but unable to get it . any body pls help:
i working here with cygwin in windows 7:
error prompt is here :
<code>$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is 37:80:a5:b5:29:29:35:ee:f3:7c:67:28:d0:27:9b:d4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Connection closed by 127.0.0.1
</code>
after using ssh -v localhost :
<code> ssh -v localhost
OpenSSH_6.4, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/VINOTHINI/.ssh/id_rsa type 1
debug1: identity file /home/VINOTHINI/.ssh/id_rsa-cert type -1
debug1: identity file /home/VINOTHINI/.ssh/id_dsa type -1
debug1: identity file /home/VINOTHINI/.ssh/id_dsa-cert type -1
debug1: identity file /home/VINOTHINI/.ssh/id_ecdsa type -1
debug1: identity file /home/VINOTHINI/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.4
debug1: match: OpenSSH_6.4 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr firstname.lastname@example.com none
debug1: kex: client->server aes128-ctr email@example.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 37:80:a5:b5:29:29:35:ee:f3:7c:67:28:d0:27:9b:d4
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /home/VINOTHINI/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/VINOTHINI/.ssh/id_rsa
Connection closed by 127.0.0.1
</code>
Comment: Try running in verbose mode: `ssh -v localhost` and post output here.
Comment: Edit your question and put it there, it will be much clearer
Comment: @Stefano i tried it not working . it shows many more debugging and at last connection closed by 127.0.0.1
Comment: Yeah exactly. **We need _more debugging info_ to be able to help you**
Comment: @Stefano i have edited my question pls see it there
Comment: Is your public key in the `authorized_keys` file?
Comment: @Stefano Yeah there is ssh-rsa key which is encrypted.
Answer: at last i got it , it was because of my Avast antivirus. it was blocking the local server to open. I uninstalled it and got running. May it will help someone.
Comment: Yep, Add rule in antivirus program so in future it wont block any port which you are using

Connection closed by 127.0.0.1


How to use random input in Selenium IDE using HTML?

Question: I want create a script in selenium IDE using html. I am not able to set the random/auto generated input in any field(Ex: email address, text, numbers, etc.).
Or I want to increment the input value entered in any field.
Can someone provide me the solution to implement this logic/script?
Thanks!
Answer: You can use javascript to generate random values and increment numbers as well.
Links on how to use javascript in ide :
http://seleniumhq.org/docs/02_selenium_ide.jsp#store-commands-and-selenium-variables
http://seleniumhq.org/docs/02_selenium_ide.jsp#javascript-and-selenese-parameters
Answer: I just posted this extension to type a Lorem Ipsum text. The first param is the locator and the second one (optional) tells how many words to type


How to use labeller with facet wrap to generate unique facet labels?

Question: Consider this example if I wanted each facet to be labelled with the first 11 letters of the alphabet? rather than the value of gear then carb.
data(mtcars)
ggplot(mtcars)+
  geom_point(aes(x = hp, y = disp))+
  facet_wrap(gear~carb, labeller = c(LETTERS[1:11]))

I can't use facet_grid
Answer: This can be achieved like so:
labels is a data.frame with two cols gear and carb where each row corresponds to one combination of the two vars. The labeller should return a list or dataframe, e.g. when using a list the first element if the first line of the label, the second element the second line ...
Hence to label the facets with letters we just have to return a list with one element, i.e. the first nrow(labels) elements of LETTERS.
library(ggplot2)

ggplot(mtcars)+
  geom_point(aes(x = hp, y = disp))+
  facet_wrap(gear~carb, labeller = function(labels) list(LETTERS[1:nrow(labels)]))

Comment: Thanks very much, out of interest. Is it possible to expand the answer to how you would match the labels to specific combinations of the variables. i.e. if  1st facet was to be labelled B, the second one E, without editing the positions in the list?
Comment: I just added some explanatory comments. In principle you can do whatever transformations you like in the function, e.g. reverse the vector via rev or draw a sample or ...