securing images - php base64 vs readfile

Question: Right now I am using base64 to control who has access to images on my website. I store these images outside of the webroot directory so no one has direct access to them. I then determine who has access to these images by what records they have access to in a postgresql database. Basically if they can query record "a" then they can see picture "a".
if (file_exists($pic)) {
    $imgbinary = fread(fopen($pic, "r"), filesize($pic));
    $picbase64 = "data:image/gif;base64," . base64_encode($imgbinary);
    echo('');

How can I use readfile instead of base64 while still maintaining my security? For example when using the readfile method to deliver the images to the clients browser this requires me to use the $_GET method to send a variable to a php script that has the readfile and header type that then displays the images on the clients browser. The problem is anyone can look at the html source and see the  and then call the script in their browser with any argument they like http://www.hostname/phpscript.php?imagename=bar.jpg . With the base64 method I am currently using this is not possible since I do not need to have a separate script to process my images that takes a $_GET argument, instead the images are embedded directly in the html page.
Thanks for any help!
-Edit-
Sorry guys I failed to mention that there is also anonymous access allowed and certain information is public. Meaning if you are not authenticated and you can read record "a" you can also see image "a".
Comment: Wouldn't the separate image requests come in with the user's cookie? You could check if a user was logged in and had access in `phpscript.php`.
Answer: The substantive difference between what you're doing now and what you need to do is to send the files over multiple requests. This requires authenticating or otherwise remembering the user between requests.
What you have right now is:
if (file_exists($pic)) {
    // do stuff
}

And what you need is:
if (file_exists($pic) && user_has_image_permission($pic)) {
    // do stuff
}

So the details you need to settle on are the logic behind the user_has_image_permission() function.
If you need this to be secure, I'd be inclined to approach this with sessions, but you could also use cookies.
If you don't mind trading some security for a bit of simplicity, you could use some sort of "url pass phrase", like those commonly used to share documents and files in web services like "anyone with the link" in Google Docs, AWS, private RSS feeds, etc.
Finally, you could use a solution completely outside of PHP, based on http authentication. In Apache, this takes the form of Access Control.
Comment: -Edit-
Sessions works instead of passing the argument to the php script. Thanks!
Comment: -Edit-
Sessions work instead of passing the argument to the php script. However if you trying to display multiple pictures by changing the session variable and passing it to the image script it will not work. For now I see no way around this but to use base64. Thanks!
Comment: @spaghettiwestern The comments are a bit too small to go into design details, but the strategy is not to save individual files in the session, rather, check the user's session to see what his/her permissions are and then check those permissions against the database to see whether you can display the image that's being requested *right now*.I.e.: User requests image id 50. In the image.php script, check user's permission in session. User's permission is "B". Does B have permission to see image 50? If yes, display the image, if not, send 403 headers.
Comment: @spaghettiwestern For the anonymous case you simply add something to `user_has_image_permission()` along the lines of: "If image is public, return true." Or you could use the following logic: "If file exists AND (user has permission OR image is public)." If you want to chat more in-depth you can contact me via my profile page.
Answer: I agree; checking access with a cookie would be the most straightforward way.  If you're looking to speed things up, search for X-Sendfile.  It can offload file transfers to your webserver once they've been authenticated.  It's built in to lighttpd, and there is an apache mod, https://tn123.org/mod_xsendfile/ .


Meaning of the applicance wattage range 2000-2400W that is written on a tally

Question: Checking power consumption of a kettle with a wattmeter it shows constantly about 1970W (close to the lowest value of the wattage range on the ketler that is 2000W).
What does mean upper up to 2400W wattage range provided on the kettle's tally?
In which circumstances 2400W consumption can happen when boiling water?
Update to answers:
In dockerhab, when kettle is in use, I can see constantly ~2000W power consumption until the end when kettle powers off / water is boiled (values arrives via mqtt broker)
<code>2020-05-28 07:44:30.618 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 1970
2020-05-28 07:44:30.621 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1970 to 1973
2020-05-28 07:44:31.617 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 1987
2020-05-28 07:44:33.625 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1987 to 1973
2020-05-28 07:44:34.622 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 1983
2020-05-28 07:44:35.624 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1983 to 1973
2020-05-28 07:44:38.629 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 1984
2020-05-28 07:44:39.630 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1984 to 1973
2020-05-28 07:44:41.635 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 1987
2020-05-28 07:44:42.635 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1987 to 1973
2020-05-28 07:44:42.937 [vent.ItemStateChangedEvent] - RaspberryOpenhabTemperature changed from 48.3 to 47.8
2020-05-28 07:44:44.640 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 1970
2020-05-28 07:44:45.640 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1970 to 1982
2020-05-28 07:44:46.640 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1982 to 1970
2020-05-28 07:44:47.643 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1970 to 1973
2020-05-28 07:44:49.943 [vent.ItemStateChangedEvent] - RaspberryOpenhabTemperature changed from 47.8 to 48.3
2020-05-28 07:44:50.646 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 1970
2020-05-28 07:44:51.647 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1970 to 1973
2020-05-28 07:44:53.658 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 1970
2020-05-28 07:44:53.667 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1970 to 1973
2020-05-28 07:44:54.653 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 1973 to 39
2020-05-28 07:45:02.661 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 39 to 3
2020-05-28 07:45:04.663 [vent.ItemStateChangedEvent] - GosundSP111-7 changed from 3 to 0
</code>
Answer: Does the nameplate also state a voltage tolerance range, such as 230-240V? Because that would make a difference in the wattage. In a load that is a fixed resistance, Power (watts) varies at the square of the voltage change. So if the kettle is rated for 2000W at 230V, but you apply 240V, the new voltage is 240/220 = 109%, so it will consume 1.09sq. the amount of power, so roughly 120% power, i.e. 2400W.
Comment: correct, there is 220-240V tolerance range on nameplate
Comment: Ah yes. This is the correct answer, not mine. I'd not give a thought to voltage tolerance.
Answer: Probably that's the initial surge when it's turned on.
When a heating element is cold, it's resistance is lower than when it's hot. This means that before it comes up to it's normal operating temperature (which may take 10s or so), it will draw more current (and hence dissipate more power).
Answer: Completing SiHa's answer: Besides the varying resistor (cold=lower=more power), the power net can not be clamped to the exact same voltage all the time, it depends f.e. on how much power is needed and has to be provided by the power transformers (high, middle, low voltage systems) every second. Normally there is a specification of +- 5 or 10% tolerance. Also it depends on how much power is needed in the building at the same time. Even if the resistance R of the kettle was absolutely constant, a +-10% voltage variation would result in a power variation from 81 to 121% according U²/R.
Answer: Edit: Apparently Gosund has achieved an ETL listing.
Unfortunately the mail-order world is almost entirely made up of very cheaply made and dangerous  Alibaba junk skillfully marketed and with lots of fake 5-star reviews.  And typically with a faked CE mark (which is so common we just call it "China Export" lol).
There is a list of well-trusted testing labs (called NRTLs), and one of them is ETL and it's popular with those few Chinese builders who bother to get properly certified.  In my experience a great many lie or mislead or say they're in the middle of application, but they usually lie about UL listing (it's more famous).   I just saw a cheap Aliababa switch that was placed in a UL-listed box - not really the same thing :)  The surest way to know is look at the UL or ETL file number, and call up UL or ETL and ask if that file number is legit.
The entire Amazon, Banggood, DealExtreme, Ebay, and AliExpress supply chain will always be full of dangerous junk.  I recommend acquiring things locally through bricks-and-mortar sellers, so your nation's consumer protection system is not being bypassed.
Other reasons, though
There are legit reasons why a (presumably simple, cheap) kettle (that in theory should only have one "speed") would perform below spec.  First is, obviously, manufacturing variations or quality issues with the kettle itself, especially if it is not certified by a competent testing lab.
But it could also be a lower-voltage problem: Some countries run 220V in contrast with the UK's 240V.  This is a plain resistor, so current will be proportional to voltage, and power will be proportional to the square of voltage.  It will not behave like a switching power supply and pull more current at lower voltage.
Further, 2400W is a full 10 amps at 240V, and that is a lot for the relatively (to NEC) thin wires Europe authorizes for circuits.  That, with a long distance run, could account for the additional drop.
Lastly it's possible that some of your voltage is being consumed in a weak connection - somewhere a wire connection is heating up.  A walk-around with a FLIR or simply feeling receptacles can expose this.
Comment: Gosund is quite popular in Germany and the other EU countries (available not only in Amazon shop). I can't really complaint about product quality, many of my friends use it without any issues, on amazon 4.3 ot of 5 customers are satisfied of total 15k buyers. It has ome certificates like ETL and CE. The number it shows are the same as professional Wattmeter shows (plus-minus several watts as I could check)
Comment: @Chris CE is meaningless, but ETL is a good one.  These NRTL listings aren't about popularity or usefulness, they're about the thing not electrocuting you or turning into an incendiary bomb if it has an electronic failure. ETL is a proper NRTL though, so I suspect it's one of the other reasons - supply voltage, vdrop or a wiring fault.


Java 8 Update 71 is trying to install a new helper tool. (On Mac)

Question: I just was prompted to update Java.  It tells me:

Does anybody know what that "helper" tool does?  If it is new I probably don't need it.  Is there a way to opt out?  Or at least know what it is before installing it?
Note that I already checked the option to suppress sponsor offers in the Java Control Panel under System Preferences.

Note also that if I don't enter the admin password, the update doesn't get installed.  It is not an offer that is presented besides the update and that I can accept or reject.
If this is a bad question or the wrong place, please comment.

Here is a followup.
I got the message again and I searched for more information about it.  I learned that the update process has been wrapped into an application.  The applications wraps the actual package and proposes additional software.  If you opt out I guess it just installs the java update.
The "new helper tool" is probably this installation wrapper, or possibly the additional software that is proposed if you don't opt out.
If you prefer to install a clean package, you can extract it form the installer.  After the update downloaded, click cancel.  Then go to ~/Library/Application Support/Java (~/Library is hidden, so use Go > Go to folder...).  You can see there all the past java update downloads.  Choose the latest version.  Inside the folder you see an app and a dmg.  Right-click on the app and select Show Package Contents.  Inside the app go to Contents/Resources.  There you find a package JavaAppletPlugin.pkg.  That is the actual package.  Double-click it to install.  It will still ask for your name and password, but this time it is asked by OSX/macOS and used for installing that package and nothing else.

Latest update: Today the update fails.  When that happens, the download is immediately removed.  In that case the method above doesn't work.

This starts to look like a blog.  The failing download was due to my antivirus.  When I disable it it works again.  Anyway I now trust the message to be harmless.  It is OS X asking if I agree to run the Java installation program.  It doesn't try to install anything else.  I just run it.
Comment: I  understand why this was voted as off-topic.  It's not a programming problem.  But it is an annoying problem with a software tool that's essential to programming (if you're programming in Java).  Moreover, it's about the installation of a version that's supposed to fix several security problems in the programming tool.  So ... maybe that does belong here?
Comment: Is there a better place to ask this question?
Comment: As it happens, I have java 8 because I installed the JDK 8.  Ordinary people (non-programmers) should have Java 6.  Anyway, I downloaded the last JDK 8 and installed it.  It didn't warn about a new tool, but it is possible it installed it anyway.
Comment: The "helper tool" is not new. It is the the installer application that actually installs Java. See my SU answer [What exactly is Java's new helper tool? Updating from Java 8 update 60 to Java update 71 on a Mac](http://superuser.com/a/1035833)
Comment: Thanks very much for the update; the specified method (`~/Library -> ...`) worked perfectly for me.
Answer: From: https://docs.oracle.com/javase/8/docs/technotes/guides/install/mac_jre.html
Oracle has partnered with companies that offer various products. The installer may present you with the option to install these programs when you install Java.
You can disable the presentation of offers during installation by specifying the --args SPONSORS=0 option in the terminal command line. For example:
open /Volumes/Java\ 8\ Update\ 40/Java\ 8\ Update\ 40.app --args SPONSORS=0

Comment: Thanks.  Does it mean that I have to manually download it first?
Comment: This is confusing.  Am I supposed to do this before I start the Java download and install, or will this disable the pernicious effects of the "helper" after it's installed.  "during installation" suggests before, but it doesn't work before it's installed (the directory doesn't exist).  Also note that it appears that you can do the same thing by checking the last option in the Advanced section of the Java admin app in System Preferences in OSX, but that doesn't stop the requirement to install the ... "helper".  (Help me avoid the helper!)
Comment: Somehow this doesn't seem to work for me with Java 8 update 71. Anyone that had success with the specific update? (I had to manually downloaded the update.)
Comment: @Mars This answer doesn't address what the helper tool actually does. The "helper tool" is not new. It is the the installer application that actually installs Java. See my SU answer [What exactly is Java's new helper tool? Updating from Java 8 update 60 to Java update 71 on a Mac](http://superuser.com/a/1035833)
Comment: I accepted the answer.  From what I have learned, the "new helper tool" is the installation program.  It installs the update while also offering sponsor offers.  The offers can be opted out.


Full width background slider Wordpress

Question: Hi i'm trying make a website that has a header at the top and and div that loops through some images in the background. at the moment i'm trying just to get it to display one image that is responsive to the screen size so no matter what screen your on the background image will scale to fit the height and width. The problem i'm having is that i can't seem to get display anything without a hardcoded width or height.
heres what i have so far
banner.php

global.css
/* BANNER */
#banner { style="width:100%; height:100%; position:fixed; left:0;top:50;overflow:hidden; background:#F00; }
.bgslider { style="width:100%; height:750px; position:fixed; left:0;top:50;overflow:hidden; background-image:url(../images/banners/trawberry_Mockup.jpg)  }

Answer: take out style=" from within the brackets
try this
#banner { width:100%; height:100%; position:fixed; left:0;top:50;overflow:hidden; background:#F00; }
.bgslider { width:100%; height:100%; position:fixed; left:0;top:50;overflow:hidden; background-image:url(../images/banners/trawberry_Mockup.jpg);background-size:cover;  }

http://jsfiddle.net/EE2Jc/

Full width background slider Wordpress


Where do I get or find the PublicKeyToken for the latest version of Entity Framework?

Question: I am upgrading from EF 6.0 to EF 6.1. The actual EF dll version is 6.1.30225.0. When I upgrade via NuGet, this is the dll version I see. After the upgrade, I am still seeing references in project files to EntityFrameworkSection Version=22.214.171.124 in my config files (app.config, web.config, etc).
Do I need to update these references as well? If so, what version for 6.1.30225.0? 188.8.131.52 or just 184.108.40.206? Do I need a new PublicKeyToken?
Thanks for the help!
Answer: They are still 184.108.40.206 So, you dont have to update them. :)
Good luck!


How to update Roblox without the app store

Question: I want to update Roblox without any app stores, is there a way to do that?
Comment: Why? Do you not have the app store?
Answer: Technically, this is not allowed legally, due to the guidelines that both Apple and Android try to follow by:

Apps should be self-contained in their bundles, and may not read or
write data outside the designated container area, nor may they
download, install, or execute code which introduces or changes
features or functionality of the app, including other apps.

I found this quote from Apple's App Development Guidelines, and Googles has nearly the same guidelines as well.
Comment: There is nothing legal about that
Comment: @RayWu What do you mean? Apple and Android both have this set guideline in place, meaning breaking it is technically a violation of their terms of service and thus is not allowed (or illegal). I didn't mean illegal by government means.
Comment: Alright but that sounded like it was about the law lol sorry
Comment: Thank you guys!
Comment: @Essie You're welcome! If it worked, remember to click the check mark so people know it does!
Answer: I don't believe so - however there are some videos online claiming you can play without updating. I won't link any here because I don't quite trust them. I suppose you could download it off of google play and technically wouldn't have to update it via the App Store, but I don't know if this would help in your scenario.
Comment: Google Play does not work on iOS


Why online shopping websites are removing comparsion feature?

Question: Recently I'm watching online shopping website and found out most of them are removing a useful feature which allows users to compare products.
Some leading brands like Amazon and Newegg don't let user to compare. Even I realized that some websites like Target, DickSmith, RadioShack and etc which formerly had this feature, now have removed it.
Are these brands following some new studies which proves comparsion should be removed? If no, how all of them are doing the same?
Comment: I do that too, paying attention to the big sites to see what they're doing. It's like I'm taking advantage of their research. (Assuming their changes are based on research and not the whims of someone over in Marketing.)
Comment: @KenMohnkern: As I outlined in [my other comment](http://ux.stackexchange.com/questions/102682/why-online-shopping-websites-are-removing-comparsion-feature/102686#comment165557_102686), I suspect their changes *are* based on research, but not necessarily on research how to improve usability or usefulness for users, but possibly also/instead on research how to improve revenue.
Answer: We have to first define "useful feature"…..
To the owner of a shopping website a feature is only useful if it results in them making more profits.   This requires a feature to result in more customers choosing to buy a high margin product/service.
Sites like Amazon have a continue program of improvement, they are testing many versions of each page, tracking the customers actions.   Therefore they must have found that "comparison feature" results in less profit.
It has been shown many time that by giving someone more information you make it harder for them to decide, often the person will walk out of the shop that has 101 different vegetables on sale and buy from the stop that only has 10 different veg for sale so as to reduce cognitive load.
Comment: I was about to upvote until I hit the last paragraph. Seriously, customers are not going to get stuck with a mental overload because they can see two products at a time instead of seeing them one by one. A much more likely reason for the removal of a comparison feature is, exactly as you say, that it "results in less profit" - because more informed customers will make a more informed choice, which may well *not* be the "high margin product/service".
Comment: @O.R.Mapper,  See https://ideas.repec.org/a/eee/eecrev/v78y2015icp97-119.html  (Moreover, within-subject analysis indicates that cognitive load leads to more risk-averse behavior, more impatience over money, and (nominally) more likelihood to anchor. )
Comment: Subjects in that study received "1- and 8-digit memorization treatments". In other words, they had to concentrate on something entirely unrelated while making a choice, and *that* was how the mental load was increased. But irrespective of that, seeing how you get flooded with link lists of the "Products who bought X also bought ..." type on almost every Amazon page lets me confidently say that **Amazon is *not* aiming at reducing users' mental load**.
Comment: @O.R.Mapper, Amazon does not think about anything, they "just" A/B test every option any member of their staff can come up with....
Comment: And then? Do you think they'll go for the one for which A/B testing yielded the highest user satisfaction in term of usability or usefulness, or rather for the one for which A/B testing promised "making more profits" (quote from your answer)?
Answer: Good reads:
How to stop choice paralysis damaging your sales by Paul Boag
The Paradox of Choice: freedom overload equals the Analysis Paralysis by Arthur Rabier

The effort of finding a product and the fear of making a wrong decision cause visitors to abandon their cart and hinder online sales. From this perspective, a website's goal is to alleviate this fear by building a clear, accessible and logical path to conversion, without any unnecessary information.

TL;DR Analysis paralysis/Noise leads to users giving up and walking away without making a purchase.
You can think about the dating scene and moving to a large city...with so many potential suitors it can be an overwhelming experience and leave one feeling exhausted.
I'm sure studies do exist to prove this, most likely sites are A/B testing different product pages - I'd suggest doing the same as some products might benefit from having detailed comparisons while for others it can harm sales.
Answer: Comparison gets harder when two products are having different set of fields in them.
So in nowadays case most of the data that are in shopping sites doesn't match the other product of same site or different site.
Do when data is not matching and there is compare option user would go gaga.
So to avoid Miss Conception Among users this is being removed in most of the sites.
But mostly this are used in comparing vehicles and electronic products as they have same or related fields in them.


Preserving newlines entered into a textarea

Question:
Possible Duplicate:
Why entered key word is not store in database?

I'm writing a simple page in which users can enter a testimonial about the website into a textarea box. This gets stored in my database inside a table called testimonials under "testimonial" which is of type text(1000)
When I try to retrieve the testimonials and present them and an unordered list, newlines get ignored.
For example:
I hate this website.
It has tons of bugs.
Will print as

I hate this website. It has tons of bugs

How would one preserve the newline the user entered?
EDIT: I should mention that the testimonial is stored correctly with newlines in the database, when I look at it in phyMyAdmin
Comment: You need to convert the newline into a  character - it's still there, but not displayed. I think `nl2br()` is what you want: http://php.net/manual/en/function.nl2br.php
Answer: As a browser will not render \n you need to use a function called nl2br() to replace new lines with

Answer: Call nl2br() right after you had called htmlspecialchars() with your content.
E.g. it should be like this:
echo nl2br(htmlspecialchars($content));

Another solution would be to format the content with CSS that it preserves whitespace, like inside
 tags.


Can I move the contents of the boot partition to the root drive?

Question: So the updater told me my /boot partition is out of space.
I came here from this answer, which recommended (if you can't expand it directly) to just move the contents of the boot partition to the primary partition and installing grub on it.
This was for 10.10, and it sounds rather risky to me. So is that still safe to do on 14.04 LTS?
This is not the same as this question, whose hard drive is about to die. The answer does not mention moving contents of /boot.
Comment: Everything is risky if you don't have backups, if you do, nothing is.
Comment: Of course I have backups, what I mean with "risky" is that it has a chance of destroying (necessary) data, even if you have some sort of insurance.
Answer: If your boot partition is out of space it is most likely because there are too many old kernel images.
You can safely uninstall old kernels that you don't need/use anymore.
You can find the kernel version you are using with:
uname -a

To find all the installed kernels, instead:
dpkg -l 'linux-image*'|grep 'ii'

Comment: Easier yet, do `sudo apt-get autoremove`. That will usually remove all but two or three kernels.
Comment: Did that, cleaned up at least 50MB. Thanks! Also, can you do the same for the kernel config files, the "abi" files (of which I don't know what they are), and the initrd images?
Comment: Uninstalling the linux-image-* packages with `apt-get remove` should take care of removing all the files you mentined. I would follow @RodSmith's advice and just run `sudo apt-get autoremove`
Comment: I did that, but it didn't remove any old kernels. I'll use the apt-get method.
Edit: That actually cleaned up almost 1 GB, which is indeed a lot better than 50 MB. Thanks!


What is the rate of recharge for the Divine Blessing?

Question: I've hit the limit of my divine blessing this week.
I was next to the statue and it was empty. Didn't think I used it that much, and didn't hit the limit again after. Was weird.
But it sparks the question for me: How does the statue's blessing regen? Is it an amount per minute? And at what rate?
Comment: You can also configure how much the statue heals when you're near it. The default is 70% of characters HP. Besides having a low treshold here, you can still heal your characters fully by entering the "Statues Blessing"-menu when interacting with it and then clicking the desired characters.
Comment: Yeah thats one of the firsts thing i did. Put it to 100%. Wonder why its even a choice.
Answer: Based on multiple sources it takes 25 minutes for the statue Healing Power to fully recharge. Since this number is referenced by players that have reached different amounts of maximum charge (the total charge is based on the number of unlocked Statues), it is also quite believable that the recharge rate is percentage based instead of a fixed value.
The wiki claims that

Restorative Power is used to heal characters and regenerates itself at a rate of 1% of the maximum Restorative Power per 15 seconds in real time, even while the game is paused (always fully restored in 25 minutes). The maximum Restorative Power is 5,000 × the number of Statues of the Seven that the player has unlocked.

Judging on empirical data, this seems to be accurate.


UnificationEngine: Unable to send Requests via Python using TLS or SSL

Question: I am attempting to use a Raspberry Pi 3 (Model B) to send Requests by Python to the Unification Engine. With SSL/TLS verification disabled, the request happens normally, but I need to get SSL/TLS working with it.
The below code is meant to force a session to use TLSv1 for the purpose of sending Python Requests to UnificationEngine:
class ForceTLSV1Adapter(HTTPAdapter):
    def init_poolmanager(self, connection, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connection,maxsize=maxsize,block=block,ssl_version=ssl.PROTOCOL_TLSv1)

def proxy_manager_for(self, proxy, **proxy_kwargs):
    proxy_kwargs['ssl_version'] = ssl.PROTOCOL_TLSv1
    return super(ForceTLVS1Adapter, self).proxy_manager_for(proxy, **proxy_kwargs)

----Some Code here----

s = requests.Session()
s.mount('https://apiv2.unificationengine.com', ForceTLSV1Adapter())

----Some Code Here----

However, this error pops up after I send the request.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 724, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 237, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 123, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.4/ssl.py", line 364, in wrap_socket
    _context=self)
  File "/usr/lib/python3.4/ssl.py", line 577, in __init__
    self.do_handshake()
  File "/usr/lib/python3.4/ssl.py", line 804, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 543, in urlopen
    raise SSLError(e)
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/Documents/HumidityRequest.py", line 134, in <module>
    sendEmail(round(temp,3))
  File "/home/pi/Documents/HumidityRequest.py", line 92, in sendEmail
    r = requests.post('https://apiv2.unificationengine.com/v2/message/send', auth=(key,secret),data=userMessage)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 94, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

I'm not sure what is the issue right now.
Comment: Does the server support tlsv1?
Answer: Looks like the certificate verification isn't done correctly.
Have a look at:
https://the.randomengineer.com/2014/01/29/using-ssl-wrap_socket-for-secure-sockets-in-python/
https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl
you could also try the httplib.


Where can I find the configure options used to build a package?

Question: I need to install asterisk 188.8.131.52 on Ubuntu 10.10
I would like to know what the configure options were that were used to generate the ubuntu package of asterisk 126.96.36.199.
My favorite server distro is Slackware and on that I just reference the Slackbuild file to find out how a package was made and possibly customised in order to remake it with a newer source, but I cannot figure this out on Ubuntu. I am under time pressure now, so I would appreciate some pointers.
Thank you!
Answer: The easiest thing to do may be to look at the build logs for the package in question. You can find them by starting at, e.g., https://launchpad.net/ubuntu/+source/asterisk, then follow the link for the version you care about (in this case, https://launchpad.net/ubuntu/maverick/+source/asterisk/1:22.214.171.124-1ubuntu1.1)
Under the "Builds" header, you'll see a link for each architecture the package was built on. Don't worry about this too much, and just pick one - I'll look at amd64.
On the build record page, there's a link to the buildlog - https://launchpadlibrarian.net/62593317/buildlog_ubuntu-maverick-amd64.asterisk_1%3A220.127.116.11-1ubuntu1.1_BUILDING.txt.gz
And if you search the log for "./configure", you'll find this:
PATH=$PATH:/build/buildd/asterisk-126.96.36.199/debian/dummyprogs ./configure \
        --host=x86_64-linux-gnu --build=x86_64-linux-gnu \
        --prefix=/usr \
        --mandir=\${prefix}/share/man \
        --infodir=\${prefix}/share/info \
        --with-cap \
        --with-gsm \
        --with-imap=system \
        --with-pwlib=/usr/share/pwlib/include/ \
        --with-h323=/usr/share/openh323/

Answer: I think that you won't be able to see the configure options from the binary (.deb) package.
You can just find some information regarding the dependencies involved by using:
apt-cache showpkg asterisk
If you need to check the configure options, i think you should download the source files by typing:
apt-get source asterisk
and then check for the information you need in the debian/rules file.
Comment: As @Pavlos G. said, see in the file `asterisk-184.108.40.206/debian/rules`, go to line 61.
Comment: And `dpkg -l | grep something` if you forgot the package name.
Comment: Thanks for that. I see all the patches now that were applied, but I don't see the actual configure command used anywhere in the files retrieved by apt-get source asterisk. That's not necesarily important for this package, but in general it is something I would like to know.


Accessing Fill Color of rectangle inside DataTemplate

Question: Please go through my problem before marking it duplicate.
I am making a Windows Phone app in xaml c# VS2013.
I am consuming Web API in a longlistselector (because I want to allow user to select item like in listbox).
XAML
                    <phone:LongListSelector Margin="24,0" x:Name="Longlist" d:IsHidden="True" ItemTemplate="{StaticResource GoalTemplate}"/>

<phone:PhoneApplicationPage.Resources>
    <DataTemplate x:Key="GoalTemplate">
        <Grid Width="432" Height="90" Margin="0,33,0,0">
            <StackPanel Orientation="Horizontal">
                <Rectangle HorizontalAlignment="Left" Height="90" StrokeThickness="0" VerticalAlignment="Top" Width="4">
                    <Rectangle.Fill>
                        <SolidColorBrush Color="{Binding rcolor, Mode=OneWay}"/>
                    </Rectangle.Fill>
                </Rectangle>
                <StackPanel HorizontalAlignment="Left" Height="86" Margin="27,4,0,0" VerticalAlignment="Top" Width="400">
                    <TextBlock Text="{Binding Name}" HorizontalAlignment="Left" Height="37" TextWrapping="Wrap" VerticalAlignment="Top" Width="405" Foreground="{StaticResource FlatUI-Blue1}" FontSize="26.667"/>
                    <TextBlock Text="{Binding Description}" TextWrapping="Wrap"  FontSize="14.667" Height="49" Foreground="{StaticResource FlatUI-Grey}"/>
                </StackPanel>
            </StackPanel>
        </Grid>
    </DataTemplate>

</phone:PhoneApplicationPage.Resources>

XAML.CS Code
void Maths_Loaded(object sender, RoutedEventArgs e)
    {
        string uri = "http://localhost:1361/api/chore";
        WebClient client = new WebClient();
        client.Headers["Accept"] = "application/json";
        client.DownloadStringAsync(new Uri(uri));
        client.DownloadStringCompleted += (s1, e1) =>
        {
            //var data = JsonConvert.DeserializeObject<HomeWork[]>(e1.Result.ToString());
            //MessageBox.Show(data.ToString());
            var hw = JsonConvert.DeserializeObject<HomeWork[]>(e1.Result.ToString());
            foreach (HomeWork c in hw)
            {
                if (c.Chore_Type == "Maths")
                {
                    result.Add(c);
                    // result is a List<HomeWork> to store only maths
                    // homework in longlistselector
                }

            }

            Longlist.ItemsSource = result;

        };
    }

Now my problem is
I have a rectangle which indicates priority of homework.
In the database i have priority column containing "Normal","medium", and "High" Priority.
And I want Rectangle's fill color to be blue, orange, and green respectively.
But I am not able to change the color of the rectangle with respect to its priority(that is consumed from Web API.
Kindly guide me how to access the rectangle in data template an apply if-else Clause or something to change the color based on priority.
Thank you,
Please Guide me.
If you donot understand, please inform so i can explain.
Answer: I understand in the HomeWork class you have a property called Priority of type Enum or string,
Do the following:
1: Bind the Rectangle Fill Color to that property with a converter:
   <Rectangle HorizontalAlignment="Left" Height="90" StrokeThickness="0" VerticalAlignment="Top" Width="4">
        <Rectangle.Fill>
              <SolidColorBrush Color="{Binding Priority, Converter={StaticResource PriorityColorConverter}"/>
         </Rectangle.Fill>
   </Rectangle>

2: And now add a converter in the following way:
<Page...>
...
<Page.Resources>
 <Converters:PriorityColorConverter x:Key="PriorityColorConverter"/>
</Page.Resources>

3: And Create that converter:
public class PriorityColorConverter : IValueConverter
  {
    public object Convert(object value, Type targetType, object parameter, string language)
    {
       var priority = (Priorities)value;

        if (priority == Priorities.High)
            return Colors.Blue;
        else if (priority == Priorities.Medium)
            return Colors.Orange;
        else
            return Colors.Green;
    }

    public object ConvertBack(object value, Type targetType, object parameter, string language)
    {
        throw new NotImplementedException();
    }
}

Of course add in Page the xmlns namespace where you have created the converter.


When does list multiplication create multiple copies of the object and when does it reference to the same object?

Question: My confusion arises from the question - Difference between [[0]*m]*n and [[0 for i in range(0,m)] for j in range(0,n)]
There it says that integers are not referenced because they are cheap but since lists are expensive to copy, but how is it decided what to copy and what not to, should I be worried about similar behaviour for strings in any case or for any other objects in case I decide to do something like
<code>[<object>]*m
</code>
Comment: I highly recommend this read: https://nedbatchelder.com/text/names.html
Comment: This "explanation" at the end of the answer is completely wrong and makes no sense. The only reason why you don't get into problems in this case is that you can't mutate an int, so the only thing you can do to an int that is an item of a list is replace it by another object. Problems occur when the list items are lists and you mutate (not replace) some of these lists.
Comment: That answer is incorrect. Python is not C, and has no "copy-by-reference"/"copy-by-value" distinction. The behavior asked about in that question is due to lists being mutable and integers being immutable.
Comment: Probably dupe of https://stackoverflow.com/questions/240178/list-of-lists-changes-reflected-across-sublists-unexpectedly
Answer: In fact, creating <code>[0] * m</code> also refers to the same integer:
<code>>>> lst = [0] * 3
>>> [id(i) for i in lst]
[2751327895760, 2751327895760, 2751327895760]
</code>
However, for immutable types, the reason why this is not harmful is that they do not operate in place. If you apply an action that seems to be an operation in place to them, you will only get a new object, which makes it impossible for you to modify the referenced integer in place in the list, so other integers will not change because an element is modified:
<code>>>> x = 0
>>> id(x)
2751327895760
>>> x += 1
>>> id(x)
2751327895792
</code>
Therefore, for objects such as strings, integers and so on (there is no problem with tuples in most cases, but be careful if there are mutable objects inside tuples! Thanks for @Kelly Bundy's reminder), there is no harm in using list multiplication. When you want to create a list containing the same objects by multiplication, if you can ensure that you will not modify any of them in place, then this operation can be said to be safe.
Comment: Better remove tuples from the the list of harmless cases, since they can contain mutable objects.


Hotjar regex for matching prestashop product pages

Question: I have installed HotJar script on my Prestashop web site.
I am tracking now main page.
I have activated now Friendly URL and mod_rewrite on site.
I would like to track as well all pages that matches Product page : (without Friendly URLwould be /myprestashop.com/index.php?controller=product&id_product)
I have taken from Presta code following regexp for product pattern pages :
"#^/(([_a-zA-Z0-9-\pL])/)?(?P[0-9]+)-([_a-zA-Z0-9\pL\pS-]).html$#u"
However if I paste that in hotjar regexp field it says that it's invalid.
Could you please help me to reformulate it so could HotJar accept it ?
Thank you
Daniel
Answer: I have used https://regex101.com/ validator and that it's really helpfull.
It seems that has to be like a javascript flavor :
/(([_a-zA-Z0-9-\pL])/)?([0-9]+)-([_a-zA-Z0-9\pL\pS-]).html$
Hotjar site has accepted it.
Comment: I'm glad to hear you managed to solve your issue yourself. Sorry I didn't get here in time to help out :).You're spot on btw! we require a JS flavored regex for our targeting.

It's vaguely mentioned in our [regex docs](https://help.hotjar.com/hc/en-us/articles/115012727628-Using-Regular-Expressions-to-Target-Multiple-Pages), but that doesn't really help much when you're in the product.

I'll make sure that we at least insert a link to the docs in the targeting UI. Thanks for finding this UX flaw for us!
Comment: HEllo eKIK ! thank you for answering ! what improvement suggestion I have for Hotjar: to propose automatically for shops those kind of regexp. for example if is default presta to propose a set for product page and listing page. an so on for other ecomm solution.
Comment: That's a really good idea Daniel - some kind of "common regex library", which could also include platform specific ones (like Prestashop etc).


automatic send email based on cell value & adjust with previous Script

Question: How to auto send email based Column D "Today"
to Email on Column A with Subject of COlumn B and Body of Columnn C

I found a script quite similar to my condition, but it only send to a static email
Script Source
function sendEmails() {
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Email");  // To only handle the trigger sheet
  var startRow = 2;  // First row of data to process
  var numRows = 2;   // Number of rows to process
  // Fetch the range of cells A2:B3
  var dataRange = sheet.getRange(startRow, 1, numRows, 2)
  // Fetch values for each row in the Range.
  var data = dataRange.getValues();
  for (i in data) {
    var row = data[i];
    if (row[2] === "Today") {       // Trigger only if Column C is "Yes"
      var emailAddress = row[0];  // First column
      var message = row[1];       // Second column
      var subject = "Bday ==" + row[2]; // Add "Yes" although by your trigger logic it will always say yes in the email
      MailApp.sendEmail(emailAddress, subject, message);
    }
  }
}

And is it possile to make it compatible with my previous script
Link Source
this script about Dynamic Dependent Drop Down Lists
function onEdit(event)
{
  var maxRows = false;

  // Change Settings:
  //--------------------------------------------------------------------------------------
  var TargetSheet = 'Main'; // name of sheet with data validation
  var LogSheet = 'Data1'; // name of sheet with data
  var NumOfLevels = 4; // number of levels of data validation
  var lcol = 2; // number of column where validation starts; A = 1, B = 2, etc.
  var lrow = 2; // number of row where validation starts
  var offsets = [1,1,1,2]; // offsets for levels
  //                   ^ means offset column #4 on one position right.
  // var maxRows = 500; // to set the last row of validation; delete this row if not needed

  // =====================================================================================

  SmartDataValidation(event, TargetSheet, LogSheet, NumOfLevels, lcol, lrow, offsets, maxRows);

  // Change Settings:
  //--------------------------------------------------------------------------------------
  var TargetSheet = 'Main'; // name of sheet with data validation
  var LogSheet = 'Data2'; // name of sheet with data
  var NumOfLevels = 7; // number of levels of data validation
  var lcol = 9; // number of column where validation starts; A = 1, B = 2, etc.
  var lrow = 2; // number of row where validation starts
  var offsets = [1,1,1,1,1,1,1]; // offsets for levels
  // var maxRows = 500; // to set the last row of validation, delete this row if not needed
  // =====================================================================================
  SmartDataValidation(event, TargetSheet, LogSheet, NumOfLevels, lcol, lrow, offsets, maxRows);

}

function SmartDataValidation(event, TargetSheet, LogSheet, NumOfLevels, lcol, lrow, offsets, maxRows)

..... Etc etc

Sorry the script is so long, i got warning "your post mostly code"
i just post some of it here and you can check full script onLink Source
Answer: You can change the script as such to check the rows and send email based on value of column D:
function sendEmails() {
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Email");  // To only handle the trigger sheet
  var startRow = 2;  // First row of data to process
  var numRows = sheet.getLastRow()-1;   // Number of rows to process
  // Fetch the range of cells A2:D
  var dataRange = sheet.getRange(startRow, 1, numRows, 4)
  // Fetch values for each row in the Range.
  var data = dataRange.getValues();
  for (i in data) {
    var row = data[i];
    if (row[3] === "Today") {       // Trigger only if Column D is "Today"
      var emailAddress = row[0];
      var subject = row[1];
      var message = row[2];
      MailApp.sendEmail(emailAddress, subject, message);
    }
  }
}

To use this in a trigger similar to your second script you need to create an Installable Trigger and specify the function name. For example, if you want to trigger the sending every 24 hours:
function createTimeDrivenTriggers() {
  // Trigger every 24 hours.
  ScriptApp.newTrigger('sendEmails')
      .timeBased()
      .everyHours(24)
      .create();
}

Comment: Thankyou, i put this on my gsheet and 2-3 days to evaluate. i'll report here with result
Comment: Sir the email doesnt send, i copy paste your script on top of Dynamic Dependent Drop Down Lists script and allowed permission it asked
Comment: This is the Gsheet https://docs.google.com/spreadsheets/d/1A9zec2A1cp9KSPWrhFsLRjgzwd6LW8yovMlnNMaTfE0/edit?usp=sharing, it only Works when i Press "run" on script. how to make automaticly triggered?
Comment: It automatically triggers every 24 hours / 1 day. If you want to test the trigger you can change the duration to like every 5 minutes as such ````everyMinutes(5)````
Comment: thankyou so much sir, i adjust the trigger as my need


Using std::is_base_of with decltype on class hierarchy

Question: For the purposes of brevity, this is the simplified hierarchy:
class IBase
{
public:
    virtual ~IBase() = 0 { };
};  // eo IBase

class IDerived : public virtual IBase
{
public:
    virtual ~IDerived() = 0 { };
};  // eo IDerived

class Base : public virtual IBase
{
public:
    Base() { };
    virtual ~Base() { };
};  // eo Base

class Derived : public IDerived
              , public Base
{
};  // eo Derived

And a function to determine if a particular pointer to a class implements the passed "interface":
template<typename T>
bool same(IBase* base)
{
    if(std::is_base_of<T, decltype(*base)>::value)
        return true;
    return false;
};

And the sample:
IDerived* i(new Derived());
bool isSame = same<IDerived>(i);

I am aware I may me mis-using decltype here.  It appears whatever I try, std::is_base_of::value is always false.  What I want this function to do is, answer the question:
Does the object pointed to, derive from the type (T) passed as the template parameter?
Answer: decltype, like sizeof, is a compile-time construct. That means, decltype(*base) will give the static type of the expression *base which is IBase. So what you intend to achieve cannot be done this way.
I would suggest this solution:
template<typename T>
bool same(IBase* base)
{
     return dynamic_cast<T*>(base) != nullptr;
};

Comment: Thanks for this.  I was hoping to avoid using dynamic_cast.  But looks like it's the way to go for now.
Answer: template<typename T>
bool same(IBase* base)
{
    if(std::is_base_of<T, decltype(*base)>::value)

This can't work. decltype(*base) would be IBase (always), so it would never reflect the runtime type of base.
Probably the best you can do is dynamic_cast<T*>(base)!=0.


Can I use a 50 amp range cord to connect a new dryer to a 50 amp receptacle?

Question: I have read the original posts here, but just want to make sure I am correct.  Just bought a new dryer (been using mother nature for the past year)
I have 50 amp (double pole) breakers in my service panel, labeled "Range Basement" (must have been a range in the basement at one point).  It feeds roughly 25 feet of 6/3 wire, at the end is a 50 AMP 3 prong (all diagonal) range outlet.
I should probably change the breakers to 30 AMP (in case something goes wrong with the new drier), but here is where I am still a little confused.  Do I replace (if I keep the 50 AMP breakers) the outlet with a new 50 AMP, or can I just go down and buy a new 50 AMP range power cord for the drier?
Its obvious that the dryer will never pull enough amps to trip the 50 AMP beakers, and there will never be a Range in my basement again.
Answer: It sounds like you have a 50A circuit in good order.  For 30A and 50A circuits, code absolutely requires that the breaker match the receptacle amperage (which is to say, the pattern of its pins).
I see 30A to 50A adapter cables are not sold, except for RVs and boats, which tells me code prohibits them for non-vehicles. Code is written in ash, so I would be reluctant to change that cord. Instead

leave the wire as it is (it will be oversize, but that is both allowed and good)
change to a 30A receptacle to match the dryer
change the breaker to 30A
If there's no ground, add one with same wire size (so you're good for 50A).

geeknote, for the gory details on making outlets match circuit sizes, see table 210.21(B)(3) here  30 amp dryer plug is hooked to a 40 amp double breaker, is this ok?
Answer: The dryer could pull 50 amps if it malfunctions or sustains damage. Use a breaker that's appropriate for the equipment.
Why would you change the cord and not the receptacle? It probably violates code to use a cord with the 50-amp configuration on a 30 amp circuit. Someone down the road could plug a range back in and have an overload situation.
Change the breaker and the receptacle.
Comment: Plugging a 50A appliance into a 50A outlet protected by a 30A breaker shouldn't cause an unsafe condition (assuming the wiring is rated for 50A) - the breaker may trip, but it won't risk a fire. The risk of just swapping out the breakers (or not) and putting a 50A cord on a (presumably) 30A dryer is that someone may plug that dryer into a 50A outlet, which could allow an overload and possible fire in the wiring inside the dryer that's not rated for 50A. If the dryer is rated for 30A, use 30A breakers and replace the outlet.
Comment: I think we're in agreement. Perhaps your lengthy comment should be an alternative answer. :)


Execute a desktop shortcut to a jar file on Ubuntu 18.04

Question: I need to run a .desktop file as an admin but I have issues achieving this. I am working with this .desktop file
#!/usr/bin/env xdg-open
[Desktop Entry]
Type=Application
Name=Some Application
Exec="/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java" -Xmx768M -jar /opt/some/application/boot.jar
Icon=/opt/some/application/logo.png
Categories=Application;

I am running Ubuntu 18.04. I have found this question here but it does not work for Ubuntu 18.04.
I tried putting an admin:// in front of it as well but that did not work either:
Neither
Exec="admin:///usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java" -Xmx768M -jar /opt/some/application/boot.jar

nor
Exec=admin:///usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx768M -jar /opt/some/application/boot.jar

work. If I want to execute that file from the command line it only opens the content. Any help is highly appreciated. I was unable to find anything on Google in regard to Ubuntu 18.04.
Comment: What command from that question did you try?
Comment: I have tried all of them
Answer: Defining Path might help. The following works for me:
#!/usr/bin/env xdg-open
[Desktop Entry]
Version=1.0
Type=Application
Terminal=false
Exec=java -jar '/home/cenap/someprogram.jar'
Path=/home/cenap/
Name=SomeProgram
Comment=SomeProgram
Icon=/home/cenap/SomeProgram-icon.png


sed to remove text if it exists between first space and first number

Question: I have a file that looks like this:
firstname.lastname@example.com email@example.com 184.108.40.206
email@example.com email@example.com 184.108.40.206
firstname.lastname@example.com email@example.com 18.104.22.168
email@example.com firstname.lastname@example.com email@example.com 22.214.171.124
email@example.com 126.96.36.199
firstname.lastname@example.com email@example.com email@example.com 22.214.171.124
email@example.com 22.214.171.124
firstname.lastname@example.com 220.127.116.11

I'd like to use sed (or awk) to manipulate the file so that everything after the first space and up to (but not including the space before) the first occurrence of a number is removed. So the above would end up looking like this:
email@example.com 184.108.40.206
firstname.lastname@example.com 184.108.40.206
email@example.com 18.104.22.168
email@example.com 126.96.36.199
firstname.lastname@example.com 188.8.131.52
email@example.com 188.8.131.52
email@example.com 184.108.40.206
firstname.lastname@example.com 184.108.40.206

I've tried:
sed  '/\s.*$\([0-9]\)/\1/g' <filename>
sed  '/\s.*$/,/0-9/p' <filename>
sed  '/\s.*$/,/\(\[0-9\]\)/p' <filename>

and many other combinations of examples I've found on the Web. I think the problem may be that sometimes the line is already as I want it; other times there are two or more extra email addresses between the first email address and the ip address.
Edit: I will always want the first email address occurring on a line (sometimes there is only one; other times there are more than one), and everything else on the line removed up until the space before the IP address at the end. I need to keep a space between the email address and the IP address. If the line contains only one email address and the IP address, I'd like to leave it alone.
Can anyone (please) help?
Comment: Please [edit] your question and clarify. You say you don't want to include numbers but your desired output shows numbers as well. Also, will your files always be like that? Will you always want the 2nd and last fields?
Comment: (Thank you for fixing my post Byte Commander...this is the first time I've used this.)
Comment: Also, can we just say that you want the last occurence of an email address followed by an IP on each line? Anything wrong with this wording?
Comment: kos, that would work as well.
Comment: terdon, I misworded my original question. Thank you for catching that. I've fixed it and hopefully clarified what I'd like to do. The files will always be in this format, where there is one or more email addresses on a line, followed by a single IP address.
Comment: OK, so is Kos right? Do you simply always want the last 2 fields? In the cases where there are multiple emails do you always want the 1nd email or always want the last email?
Comment: Yep, sorry but it's still not clear, I asked you if the criterium could be reworded for simplicity to "print only the *last* occurence of an email address followed by an IP address on each line" and you said that would work as well, but in the question you now edited "I will always want the *first* email address occurring on a line", so what do you need exactly?
Comment: Hi kos, either way would've worked - as long as I end up with a single email address followed by a space and then a single IP address. The email addresses are always the same on each line - it's just that sometimes there is just one (in which case the line is fine the way it is), other times there are two or three as in the example. It doesn't matter which address it uses as they are always the same on any given line, but there can't be more than one once the awk is run against the "misformatted" original file. Hope that helps to clarify.
Answer: To get the output you show (and not the one you describe), you can simply print the 1st and the last fields using awk:
awk '{print $1,$NF}' file

Or perl:
perl -lane 'print "$F[0] $F[$#F]"' file

Comment: Thank you terdon. Will this also work if there are three or four occurrences of the email address before the IP address? And will it retain the ones that are already properly formatted?

Thanks, Sam
Comment: Actually I answered my own question... This seems to work. Thank you so much for your help!
Comment: @samgoober Yes, it will work for an arbitrary number of email addresses as long as i) the 2nd field is the email you want and ii) the last field is the IP you want.
Comment: I was wondering how to accept the solution. I worked on this for about three hours, and you answered it in about three minutes. I'm very grateful to you for your help. (I did change it to where it pulls the first email address since there will always be at least one but not always two, but wouldn't have gotten that far without your help.) Thank you again!
Comment: @kos eek! You're right, I missed that and so did the OP, probably. Thanks, fixed.
Comment: @samgoober I had a bug in the first version, please see the update.
Comment: Hi terdon, yeah, I did catch that and changed it to $1. Appreciate you following up though. Looking forward to the day I know enough about this stuff to be able to help people.... :)


How do I tell which faction a pandaren player belongs to?

Question: In Mists of Pandaria, both factions can choose the Pandaren race.
How can I tell if a Pandaren player is horde or alliance?
Answer: All players, as of Mists of Pandaria, now have their faction included in the tooltip:

If you're suitably knowledgeable about armour rewards and looks, you will sometimes be able to tell by the style of armour they are wearing as well.
Comment: Because, to quote myself, it "didn't include the easiest way".
Comment: I was fighting a boss, couldn't write, but you updated the answer to the correct answer before I could end the fight and post
Comment: My previous answer was 100% correct.  It just didn't include the easiest way (the beta finally finished downloading its update and I got in).
Comment: ... if it was 100% correct whey did you need to edit it?
Answer: In PVP, the most intuitive way of knowing wether a Pandaren is part of your faction is by checking their nameplate. If theirs is colored Blue, they are part of your faction.
Additionally, in MoP the faction information will be included in the tooltip whenever you mouseover on any player.
Comment: Isn't everyone blue in neutral zones (on PvE servers)?
Comment: As long as they aren't flagged for PVP and you're not flagged either.


Flume ng / Avro source, memory channel and HDFS Sink - Too many small files

Question: I'm facing a strange issue.
I'm looking to aggregate a lot of information from flume to an HDFS.
I applied recommanded configuration to avoid a too many small files, but it didn't works.
Here is my configuration file.
# single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 5458
a1.sources.r1.threads = 20

# Describe the HDFS sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://myhost:myport/user/myuser/flume/events/%{senderType}/%{senderName}/%{senderEnv}/%y-%m-%d/%H%M
a1.sinks.k1.hdfs.filePrefix = logs-
a1.sinks.k1.hdfs.fileSuffix = .jsonlog
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.batchSize = 100
a1.sinks.k1.hdfs.useLocalTimeStamp = true
#never roll-based on time
a1.sinks.k1.hdfs.rollInterval=0
##10MB=10485760, 128MB=134217728, 256MB=268435456
a1.sinks.kl.hdfs.rollSize=10485760
##never roll base on number of events
a1.sinks.kl.hdfs.rollCount=0
a1.sinks.kl.hdfs.round=false

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 5000
a1.channels.c1.transactionCapacity = 1000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

This configuration works, and I see my files. But the weight average of the file is 1.5kb.
Flume console output provide this kind of information.
16/08/03 09:48:31 INFO hdfs.BucketWriter: Creating  hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484507.jsonlog.tmp
16/08/03 09:48:31 INFO hdfs.BucketWriter: Closing hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484507.jsonlog.tmp
16/08/03 09:48:31 INFO hdfs.BucketWriter: Renaming hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484507.jsonlog.tmp to hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484507.jsonlog
16/08/03 09:48:31 INFO hdfs.BucketWriter: Creating hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484508.jsonlog.tmp
16/08/03 09:48:31 INFO hdfs.BucketWriter: Closing hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484508.jsonlog.tmp
16/08/03 09:48:31 INFO hdfs.BucketWriter: Renaming hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484508.jsonlog.tmp to hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484508.jsonlog
16/08/03 09:48:31 INFO hdfs.BucketWriter: Creating hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484509.jsonlog.tmp
16/08/03 09:48:31 INFO hdfs.BucketWriter: Closing hdfs://myhost:myport/user/myuser/flume/events/a/b/c/16-08-03/0948/logs-.1470210484509.jsonlog.tmp

Someone has an idea about the issue ?

Here is some info about flume behavior.
The command is
     flume-ng agent -n a1 -c /path/to/flume/conf  --conf-file sample-flume.conf  -Dflume.root.logger=TRACE,console -Xms8192m -Xmx16384m
Note : the logger directive doesn't works. I don't understand why but I'm ...
Flume starting output is :
16/08/03 15:32:55 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
16/08/03 15:32:55 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:sample-flume.conf
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:kl
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: a1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:kl
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:kl
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [a1]
16/08/03 15:32:55 INFO node.AbstractConfigurationProvider: Creating channels
16/08/03 15:32:55 INFO channel.DefaultChannelFactory: Creating instance of channel c1 type memory
16/08/03 15:32:55 INFO node.AbstractConfigurationProvider: Created channel c1
16/08/03 15:32:55 INFO source.DefaultSourceFactory: Creating instance of source r1, type avro
16/08/03 15:32:55 INFO sink.DefaultSinkFactory: Creating instance of sink: k1, type: hdfs
16/08/03 15:32:56 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
16/08/03 15:32:56 INFO node.AbstractConfigurationProvider: Channel c1 connected to [r1, k1]
16/08/03 15:32:56 INFO node.Application: Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: { bindAddress: 0.0.0.0, port: 5458 } }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@466ab18a counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
16/08/03 15:32:56 INFO node.Application: Starting Channel c1
16/08/03 15:32:56 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
16/08/03 15:32:56 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
16/08/03 15:32:56 INFO node.Application: Starting Sink k1
16/08/03 15:32:56 INFO node.Application: Starting Source r1
16/08/03 15:32:56 INFO source.AvroSource: Starting Avro source r1: { bindAddress: 0.0.0.0, port: 5458 }...
16/08/03 15:32:56 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: k1: Successfully registered new MBean.
16/08/03 15:32:56 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started
16/08/03 15:32:56 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
16/08/03 15:32:56 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r1 started
16/08/03 15:32:56 INFO source.AvroSource: Avro source r1 started.

Since I cannot have more verbose output, I have to suppose that information like
[...]
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: a1
16/08/03 15:32:55 INFO conf.FlumeConfiguration: Processing:k1
[...]

indicates that the sinks is correctly configured.

PS : I saw followings answers but none of those works (I should miss something ...).
flume-hdfs-sink-generates-lots-of-tiny-files-on-hdfs
too-many-small-files-hdfs-sink-flume
flume-tiering-data-flows-using-the-avro-source-and-sink
flume-hdfs-sink-keeps-rolling-small-files
Comment: Have you tried if other options for rolling are working fine? Try aswell checking the log right after starting Flume, generally there might be a hint. I have checked your sink configuration against the configuration I was using awhile ago and it seems very similar (we have a bit bigger channel capacities configured, but it should not affect the sink).
Comment: @Serhiy Thanks for your input. I edited my post with the flume's console output. I'm trying to change some rolling information. It seems to be all rolling configuration information which is failing. I actually tried to set the  rolling counr (a1.sinks.kl.hdfs.rollCount=1000) but didn't works. I'm running the HDP 2.3.2, may be a tweak from here ?
Comment: Can you try to setup logging with debug level? In my installation there is `log4j.properties` in `flume/conf` directory. Try setting `flume.root.logger`, `log4j.logger.org.apache.flume.lifecycle` and `log4j.logger.org.apache.hadoop` to `DEBUG`.
Comment: Sadly, no way of doing that this works. I suspect a specific configuration setup by a teammates with admin rights. Everything looks correct both for logging (DEBUG nor TRACE) and for rolling. Temporary solution : setup a fresh install, doing the agent working as expected and apply a diff to look for a missconfiguration. Thanks for your input !
Answer: Increase batch size as per your requirement

a1.sinks.k1.hdfs.batchSize =


Where to locate png for nether portal in assets

Question: Where do I locate the png for nether portal blocks? I have already changed end portals to be entirely green but I cant locate nether portals. what file would it be in?
Answer: Assuming you are using a texturepack (not the default Minecraft textures, ex Conquest of the Sun texture pack), and on a version 1.9 or higher.
In the texturepack .zip file, go to

assets/minecraft/textures/blocks

And you'll find a portal.png file. Edit it to change the texture.
This file usually consists in several images stacked vertically, to be able to add an animation effect to the portal.

If you are using default Minecraft textures, you can find the textures_0.png (and following textures_1.png, 2, 3...) file in the following folder :

C:\Users\yourUserName\AppData\Roaming\.minecraft

The portal texture is in around the middle, in the 17th column and 7th line.


What's the most common aspect ratio in terms of web design?

Question: I'd like to know what is the most common aspect ratio to aim for in web design? I'm having problems with a site that works fine in 1920x1080 (16:9), but it doesn't work out so great in 1680x1050 (16:10). I just want to decide which would be the best choice to aim for. I realise there are mobile resolutions to bear in mind, but I'm thinking specifically of desktop users.
Comment: Possible duplicate of [Common screen resolution?](https://ux.stackexchange.com/questions/6929/common-screen-resolution)
Comment: @locationunknown it is related and the answer for my question could be derived from the answer to that one... but my question was specifically about aspect ratio. The answer provided by Tohster is what I was looking for.
Answer: This is a question that can be answered easily and quickly by Google.
But since it is a popular question, here's a reusable approach:

W3Schools keeps statistics on web browser screen widths.
The last set is from Jan 2014, but it's enough to make the point since the movement has been towards 16:9 since then:

Based on this analysis I would certainly aim for 16:9 for desktops.  Remember that browsers usually have even higher aspect ratios because the nav/tab bar and status bar will reduce the height more than the scrollbar reduces width.
But the popularity of 16:9 isn't an excuse for poor design.  It's usually important to ensure websites render well on a range of window sizes, because users don't always browse with the browser window maximized.  Even if they do, desktop users often have taskbars (e.g. windows, mac) or sidebars (e.g. chat, social) which will change the browser aspect ratio.
Comment: Thanks very much! I will always keep in mind different aspect ratios of course!
Comment: Happy it helped... Good luck and welcome to the site :-)
Answer: The correct answer is, your web design should not be dependent on the aspect ratio of the user's device. Using proper modern day techniques, a.k.a Responsive design, means that your website/webapp should be readily fluid between a phone's portrait view to a 16:9, 4K desktop screen.
Comment: One example where it is dependent on the user's device is that the background image has some content that should be visible. The background image itself can be dynamic and exist on every screen size, but different parts of the background image are getting cropped away. Designing background images with content that should be visible, can be good to know the aspect ratio to have the parts wanted to be shown to be visible.


Django-OAuth-Twitter error in Google App Engine 1.4.0 -- Unexpected keyword arguments: next_count, prefetch_count, offset

Question: Google recently updated their appengine sdk to 1.4.0, and at the same time I've started seeing exceptions both on my devbox and in production.  Here's the trace:
<code>Exception in request:
Traceback (most recent call last):
  File "/base/data/home/apps/ezcrd2/10.346707310382875890/common-apps/django/core/handlers/base.py", line 101, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/ezcrd2/10.346707310382875890/django_oauth_twitter/views.py", line 159, in callback
    user = self._authenticate(userinfo=userinfo)
  File "/base/data/home/apps/ezcrd2/10.346707310382875890/django_oauth_twitter/views.py", line 239, in _authenticate
    user = TwitterUser.objects.get(twitter_id=userinfo.id).user
  File "/base/data/home/apps/ezcrd2/10.346707310382875890/common-apps/django/db/models/manager.py", line 132, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/base/data/home/apps/ezcrd2/10.346707310382875890/common-apps/django/db/models/query.py", line 334, in get
    num = len(clone)
  File "/base/data/home/apps/ezcrd2/10.346707310382875890/common-apps/django/db/models/query.py", line 79, in __len__
    self._result_cache = list(self.iterator())
  File "/base/data/home/apps/ezcrd2/10.346707310382875890/common-apps/django/db/models/query.py", line 267, in iterator
    for row in compiler.results_iter():
  File "/base/data/home/apps/ezcrd2/10.346707310382875890/common-apps/djangoappengine/db/compiler.py", line 96, in results_iter
    next_count=75)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1304, in Run
    config = _Rpc2Config(_GetConfigFromKwargs(kwargs))
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 169, in _GetConfigFromKwargs
    'Unexpected keyword arguments: %s' % ', '.join(args_diff))
DatabaseError: Unexpected keyword arguments: next_count, prefetch_count, offset
</code>
I'm guessing the breakage is likely related to this appengine change, but not sure how to fix it: https://gist.github.com/717904
Any ideas?
Answer: Ah, the problem was evidently that I was using an old version of django-nonrel.  It had nothing to do with django-oauth-twitter


How can you make ASDF stop trying to load a nonexistent file?

Question: On Debian, I had a bunch of cruft installed in /usr/lib/sbcl/site-systems that wouldn't load because the FASLs didn't match the version of SBCL that is actually installed.
For some reason, none of these files were associated with any Debian package (this is an old computer that has been running the same Debian install for over a decade– it's on Debian Sid).
I deleted the bad systems one at a time, and for most of them, Quicklisp did the right thing and downloaded the Quicklisp version. Sometimes, ASDF would insist that the system should exist at its previous path, but restarting SBCL got past that problem.
But for one system, ASDF has persistently cached the location of its .asd file as being in the /usr/lib/sbcl/site-systems/ directory. Loading this system is impossible because ASDF will not look anywhere else, even after restarting SBCL.
I tried looking in all the paths specified in various config files under /etc/common-lisp. None of those files contain a reference to the now-missing library.
I've resorted to doing a grep -rli across all the files under /usr. I don't expect that to complete in less than a day, and it might not find anything, in which case I'll be forced to grep the whole hard drive, which might take a whole week. Hopefully, the cache isn't compressed, because then I'll never find it.
Does anyone happen to know how ASDF persists the paths of files?
Answer: After a lot of excruciating debugging, I discovered that the files in /usr/lib/sbcl/site-systems/ actually do exist. They're broken symlinks.
The files I deleted were in the similar-looking path /usr/lib/sbcl/site/, to which the symlinks pointed.
Removing the symlinks fixed all the loading errors.
Answer: A couple of ideas about troubleshooting Quicklisp, particularly if your getting bizarre behavior.:

If you use Quicklisp for any length of time you'll probably eventually use local packages, found here by default, ~/quicklisp/local-projects It's valid to symlink you're projects into that directory. If you ever rename one of your projects, of course, don't forget to create a new symlink and delete the old one
Likewise, if you rename a local project, also delete the system index which Quicklisp will then recreate the next time it runs: ~/quicklisp/local-projects/system-index.txt Doesn't hurt to delete it from time to time just to keep your system fresh.
your *.fasl files can become stale too, deleting the system cache forces quicklisp to recompile everything. On an Ubuntu system running SBCL that would mean deleting the contents of:

rm -rf ~/.cache/common-lisp

Try updating the Quicklisp client

(ql:update-client)

Potentially deleting and reinstalling Quicklisp itself at ~/quicklisp can be necessary. (It's possible to inadvertently edit source files when your debugging and using Swanks lookup definition feature, breaking installed packages that used to work. Not that I would ever have done something as careless as that.)
Also, don't forget that ASDF decends into directories looking for *.asd files. If you have a stray one that's improperly structured that can cause havoc on you build system. (See my comment above about registering local projects to Quicklisp)
Finally, don't forget to check your lisp init file, e.g. .sbclrc for any debugging or quick and dirty hacks you might have left there and forgot about.

These are all things that have worked for me at one time or another, hopefully I'm not perpetuating legend and cant on things have have long since been fixed!


Is there any reason to use an old account?

Question: Is there any reason to use an old Blizzard account for WoW classic? I've not used the Authenticator in ages. And if I try now, it simply doesn't notice that it needs to generate a code. Nor does the "manual" option work. I can go the long route of getting it removed, uploading pictures of my ID, but it seems far simpler to just create a new account. The whole point of WoW classic is to restart at level 1 anyway, so what benefit is there to lose?
The chief problems I see are the need to use a new email (no big deal) and a problem with the Authenticator only supporting a single account (but that Authenticator is still optional? So I can just skip it - it's always been a hassle)
A side problem of this authentication issue is that I can't access the Blizzard fora either, as you need the Authenticator even for that.
Answer: The main reason why you would want to use an old account is if you had the original Collector's Edition redeemed on your account, because having the original CE from 15 years ago grants your classic characters access to those pets.
Regardless of whether you choose to create a new account or jump through the hoops to recover your old account, I can recommend getting the App Authenticator if you have a phone that supports it. It is far easier to use and just as secure.
Answer:
Ease of use
Having multiple accounts may become a hassle in the future if you ever return to the 'retail' version of the game. You've already had issues with recovering an old account; imagine having to do this for multiple accounts in the future.

Financial
Having your characters on the same account means you can pay a single subscription for both your 'retail' characters, and your classic characters. Although you may not want to now - you may very well want to do so in the future.

Social
Creating a new blizzard account means you need to migrate your friends list to your new account. Keep in mind that this list applies to all blizzard games. So, even if you don't want to return to retail, you may end up playing another blizzard game, and will need (or want) to keep your friends list in sync. Not to mention the annoyance of others having to add both your accounts to their friends list.

I did the same when hearthstone came out; I now have an account just for hearthstone, and one for everything else. Sure, it's not that big of a deal - but in the end, I needed to recover my old account in the end anyway, and now I'm stuck with two accounts which can't be merged.


Use of language on wikipedia - what kind of distribution?

Question: I have an interesting problem and was wondering whether anyone would be able to point me in the right direction.
I am wondering whether the use of a word in the english language on Wikipedia is normally distributed and/or whether normal distribution could be used as an approximation to calculate a p-value.
The problem:

Take a search term lets say 'news'
Crawl 10000 entries on Wikipedia and record the number of occurrences
We know that the occurrence of the word 'attic' in the entire english language is 2722 (based on http://www.wordfrequency.info/intro.asp)

Could we use this information to develop a Hypothesis test to ascertain whether the results from our 10000-page crawls are statistically significant?
If not - how could one go about determining statistical significance?
I assume the English language is not normally distributed. Could normal distribution still be used as an approximation?
Thank you very much
M
Comment: A normal distribution is not suitable for word frequencies.  Try http://imonad.com/seo/wikipedia-word-frequency-list/ for an analysis of frequencies on Wikipedia.  If its log-log graph was linear (in fact it bends) then the frequencies would be following Zipf's law
Answer: You've got an implicit model here: that Wikipedia pages are generated from some distribution, perhaps by random draws of a random number of words. (Alternatively, you're saying that wiki pages are your measure space, and there's a 0/1 random variable saying whether or not the word "news" appears in the page; in that case, it's not clear what the underlying probability mass function on the set of pages would be.)
You can test whether, with this assumption of how pages are generated, some page or other is anomalous, or some word is used more often than it should be, but any results you find may well end up being an indictment of your model rather than a point of serious statistical significance.
For instance, the word "Wikipedia" appears disproportionately often on wikipedia pages. So does the word 'external'. Neither of these is the slightest bit surprising, despite their high variation from the usual word-frequencies for English.


Skyrim letter bug

Question: So I recently recieved a letter from a courier in Winterhold, but I don't know who sent it. The letter is labeled "Letter from [...]". This is what it says, "Shadth Gra-Dulamakh,
You may or may not realize this, but by taking care of [...], you have indirectly helped me. Don't worry about how I know it was you, I won't be informing the authorities. But I would like to reward you for your efforts.
Please visit me at your earliest convenience.

[...]" Unfortunately I couldn't provide a picture. Has anyone else run into this problem, and if so, is there a way to fix it?
Comment: What exactly is this picture? Can you give a description about it?
Answer: It's not a big issue.
This letter is a generic dynamically generated quest based on two NPCs.  The [...] is suppose to be another NPC's name, but something weird happened with the script and it couldn't pick up the NPC's name.
This isn't going to break any real quests for you, you don't need to worry about it.
The second place where this can show up are inheritance letters when NPCs are killed.
Comment: Specifically, this can happen if you have a mod installed which added a character who has a relationship which would trigger the letter, but doesn't have a complete record for their name in the CK.  Mods adding large amounts of NPCs can cause this.

Skyrim letter bug


Defined sort order with Isotope

Question: I am using isotope to sort and filter a number of tiles relating to features for different products. Some of the tiles are unique to each product, while others are shared. When you switch between products, it filters the tiles to show only those tiles tagged with the appropriate class names, however, I want to control the order of these tiles as the features have a different priority for each product. If I don't implement any way of controlling the order, they appear in the order that they are within the markup, which means that the first product's features are going to be the first tiles for subsequent products in the markup.
I am trying to figure out the most efficient way to do this. Right now I have added attribute's as such:
Tiles

Feature 1

Feature 2

jQuery/Isotope code
$container.isotope({
    getSortData: {
        productA: function ($elem) {
            return parseInt($elem.hasClass('productA').attr('productAOrder').text(), 10);
        },
        productB: function ($elem) {
            return parseInt($elem.hasClass('productB').attr('ProductBOrder').text(), 10);
        }
    }
});

$container.isotope({ filter: '.productA'});
$container.isotope({ sortBy: 'productA', sortAscending : true});

For whatever reason, I can't get the tiles to follow the order I have defined. Is there a better way to do this?
Answer: jQuery's hasClass() returns a boolean so
$elem.hasClass('productB').attr('ProductBOrder').text()

throws an error.
Replacing hasClass('productB') with filter('.productB') should have the effect you're looking for, assuming that call was in there so that it would return 0 if it didn't exist.

Defined sort order with Isotope


how to read data from mongodb block by block and write to posrgres

Question: I have a mongo db running in production. I want to move the data from MongoDB to Postgres for some migration requirement.
Now comes to data part, I am planning to write one utility which will read data from MongoDB and write to Postgres.
Here I want to read all the data from mongo db (contains 240335 rows) to Postgres.
I can not read entire data into memory in the application. I want to read in batch of 10000 then write do some modification and write those to Postgres and then again read next 10000 again repeat this.
How can I do this?
Answer: I never did it, but I think, you can use cursors to upload records severally.
Problem - the solution will be inefficient.
Example
<code>var myCursor = db.bios.find( );
var myDocument = myCursor.hasNext() ? myCursor.next() : null;

if (myDocument) {
    var myName = myDocument.name;
    print (tojson(myName));
    //put record to db or add to batch, and upload if 1000 in collection
}
</code>
Maybe you can use the streams? I do not know if streams can be used in MongoDB.


Nvim with prose: how to set up proper `autoformat` line-wrapping

Question: I'm working a Neovim config for writing prose. I want it to autoformat the current paragraph, continuously line-wrapping. Suppose I use :set formatoptions=a to accomplish this. This leads to the following annoying behavior, where █ represents the cursor:
here is some text
█

Now, if I enter insert mode and type and here is more, it puts it on the previous line:
here is some text and here is more

Instead, I want it to leave the newline like this:
here is some text
and here is more

Looking at :help fo-table, I see an interesting option:
w       Trailing white space indicates a paragraph continues in the next line.
        A line that ends in a non-white character ends a paragraph.

If I enable it with :set fo+=w, now the above example works as desired. However, now if I try manually formatting a long line with gq, it will add (actual) trailing spaces to the end of every line:
Here is a very very very very very very very very very very very very very very long line.

becomes
Here is a very very very very very very very very very very very very very_
very long line.

where _ represents a space.
I want it to not add spaces when manually formatting (as when the w option isn't used), but I also want it leave the newline intact when inserting a second line (as when w is used). How can I achieve both of these?
Comment: See `:hel 'formatexpr'`.
Comment: While I'm still curious how to do this, I did remembered that Markdown renderers will remove single newlines when rendering, so it's actually fine to not use `w` and simply let it move the cursor to the end of the preceding line when inserting. This "annoyance" just reflects how it will look when rendered, which is important.
Answer: Maybe you can just keep using gq, and have the following autocmd to get rid of the trailing spaces
au BufWritePre * :let pos=winsaveview()| silent! %s/\([^[:space:]]\)\s\+$/\1/e | call histdel('search', -1) | call winrestview(pos)