How long does it take for a blob to update in Azure Blob Storage?

Question: I am using Azure Blob storage to store images for my new website.
When I have an image called background.jpg and replace it with a different image, but with the same name (background.jpg) it can take up to a week to actually change to the new image on the website.
The only way to test it is to either visit the website through a VPN or use someone else's machine (who's never accessed the website before).
I have cleared my local storage and cache multiple times.
It's almost as if blob storage has it's own caching mechanism on the server side.
I've had a look at the Microsoft Docs, but they are not so straight-forward to understand in this area.
Thanks in advance!
Comment: What is the access tier of your container which stored the specific image?
Comment: @DorisLv the access tier is "Hot"
Comment: Hi, @ Joachim Velzel,  if the solution did some help, do you mind accepting it as the answer for others to refer?
Comment: Hello @DorisLv, unfortunately, I have not come further with this. I am using azure storage explorer also, but I might have access to more settings in the Azure Portal.

This problem is not really a show-stopper, it's just annoying that the only way to test is through a VPN.
Answer: Actually the update in blobs should not be so long. The reason may not only related to blob Storage. You can check it first if the picture in your storage account changed so long either or just your website.
There are three reasons I can imagine:
1. Using CDN
Considering you said "The only way to test it is to either visit the website through a VPN or use someone else's machine", you should check if you use CDN on your website.

In theory, this situation is not very scientific, but the reason is
very simple and easy to understand. If you don't have record keeping,
then the manufacturer can only provide you with overseas nodes for
relay, which will lead to slower access to the content of your
website, including the image.

2. Didn't delete browser cache
Another sentence you said "I have cleared my local storage and cache multiple times." I am not sure where the cache you clear, if it is not the browser's cache, you can try clear it too.
3. Use Cache-Control in Azure Storage
You should check if you set Cache-Control in Blob Properties. If you set it, you can do this for not caching: Cache-Control "private, no-cache"


Move and clone VirtualBox machines with filesystem commands

Question: I know of 2 ways to clone a VirtualBox machine on a linux host, one is by using the VirtualBox gui and exporting and re-importing as appliance (in the file menu of VirtualBox).
The other is by cloning only the virtual disk container files:
VBoxManage clonevdi /mnt/md1/original/root-fs.vdi /mnt/md1/cloned/root-fs.vdi

(Taken from http://forums.virtualbox.org/viewtopic.php?p=853#p858 )

Edit: this does not seem to work for VDI files that are already part of ~/VirtualBox/VirtualBox.xml, I get an error message:
ERROR: Cannot register the hard disk '/mnt/md1/original/root-fs.vdi' with UUID {96a1462b-c30a-4911-a954-d9093122f083} because a hard disk '/mnt/md1/original/root-fs.vdi' with UUID {96a1462b-c30a-4911-a954-d9093122f083} already exists in the media registry ('/home/mit/.VirtualBox/VirtualBox.xml')

I would have to create a new VM afterwards and use the cloned virtual disk containers.
Is it also possible to clone a VM by copying the virtual disk contianer files and the config files using filesystem commands and do the rest by hand?
I'd have to manually edit the   ~/VirtualBox/VirtualBox.xml and insert a new disk and a new machine:

Can I just make up UUIDs or how would this work?
I would very much prefer this hardcore method of doing things as it allows me to freely and rapdily backup, restore, move or clone machines. Or ist there a better way to do this?
Answer: You can change the UUID of a vdi using the command
VBoxManage internalcommands setvdiuuid disk.vdi

or if you are using VBox V4.x at least to 4.0.10
 VBoxManage internalcommands sethduuid disk.vdi

then use the GUI  import the new 'unique' disk.
If you want to clone a particular VirtualBox machine then you are probably better off making an appliance of it and then using it. To make an appliance use the GUI Export Appliance feature or the vboxmanage  export command. The resultant appliance can then be imported using the GUI Import Appliance feature or the vboxmanage import command.
Comment: +1 million if I could. Note, in my version (4.x) this is "sethduuid", not "setvdiuuid".
Comment: @Mike: Cheers - updated to reflect changes
Answer: Or you can edit the "Virtualbox.xml" setupfile with the right uuids for the machineid. Worked great for me moving a virtual machine from one host to another (both hosts being 64-bits linux, no idea if it had worked moving to e.g. windows or mac) see http://sickel.net/blogg/?p=1581 for details.
This works even if the vboxmanager on the original host for some reason can not be run.


CSS: two side-by-side p tags, same height, left one fixed-width & known height

Question: I'm trying to get a vertical list of
's of varying height, where each has two
 tags next to each other and has a height of the tallest
 tag; the
 on the left has a fixed width (128px) and the
 on the right should take up the rest of the page but not wrap underneath the first
.
Here's sample HTML:

Something (128px wide)

Something long...
        Shouldn't wrap underneath .category

Another thing

Another long description...

What should the CSS look like?
Comment: Could you do this in a table instead? It's exactly what tables are designed for. You could probably hack it together with CSS but unless there's a specific reason for using a UL you should use a table
Comment: Yes, I hadn't thought of that, thanks! I hope it's alright - I've never used tables for layouts but this is a "categories"-in-the-left-column, "descriptions"-in-the-right-column thing, so I guess it's a good use-case for ``?
Comment: Actually, that's entirely the opposite of what tables were designed for. They were created for tabular data, not layout. Using them for layout is the hack. Using CSS for layout is not a hack.
Comment: Weird, I didn't see your comment prior to my last one. Like I said elsewhere in comments, if it is tabular data, then absolutely. But, use a TR that contains TH tags at the top so that it is a proper table. If the data doesn't compel the use of the headers, then I would assert that you were on the right track with the list items.
Answer: Flexbox can do the trick for you.

ul {
  width: 300px;
}
li {
  width: 100%;
  clear: both;
  display: flex;
  flex-direction: row;
}
li p {
  display: inline;
  float: left;
  background-color: #eeeeee;
}
p.category {
  width: 128px;
}

Something (known width)

Something long... Shouldn't wrap underneath .category

Another thing

Another long description...

More Info:
https://css-tricks.com/snippets/css/a-guide-to-flexbox/
Answer: There are several ways to skin this cat. If I'm understanding what you're after correctly, I think that using table-row and table-cell display properties may be the easiest way to go.
Here's a jsfiddle:
    https://jsfiddle.net/s2jzh31z/2/
That includes the following,
HTML:

Something (known width)

Something long and multiple lines.
        Shouldn't wrap underneath category.
      Lorem ipsum dolor sit amet consecteur adipiscing elit no numi.
      Lorem ipsum dolor sit amet consecteur adipiscing elit no numi.
      Lorem ipsum dolor sit amet consecteur adipiscing elit no numi.
      Lorem ipsum dolor sit amet consecteur adipiscing elit no numi.
      Lorem ipsum dolor sit amet consecteur adipiscing elit no numi.
      Lorem ipsum dolor sit amet consecteur adipiscing elit no numi.

Another thing(known width)

Something to the right,
      determines the height of this "<li>"

CSS:
ul {
    margin: 0;
}

li {
    clear: both;
    display: table-row;
    width: 100%;
}

p {
    display: table-cell;
    padding: .5em;
    vertical-align: top;
}

p.category {
    width: 128px;
}

I took the liberty of vertical aligning and adjusting the padding a bit to make it look neater. The vertical align was one of the benefits I was looking for in using the table-row / table-cell approach.
Comment: I didn't know about `display: table-row` or `-cell`. Thanks for that! I think I'll make the whole thing a `` though :)
Comment: See my other comments. I highly recommend against using a table for layout. CSS was created to eliminate the need to hack layout with tags that have a different semantic meaning. You're welcome! Cheers
Answer: To be semantically correct and 'new age' you could do this using CSS3's Flexbox layout but if this is table data (looks to be, using category and description) then you could restructure it as a table (which also helps if you need to support older browsers).
HTML would look like this:

        Something (known width)
        Something long and multiple lines.sdsdsdsdsdsdsdsdsdsdsdsdddddddddddddddddddddddddddddddsdsdsdsdsdsdsdsdsdsdsdsdsdsdsdsdsdsdsssssssssssssssssssssssssssssdddddddddddddddddddddddsdssdsssssssssssssssssssssssssss
            Shouldn't wrap underneath .category

        Another (known width)
        Another long and multiple lines.
            Shouldn't wrap underneath .category

and your CSS:
.category {
    width: 128px;
}

Comment: Tables should only be used for actual tabular data, not layout. HTML should be sematically correct. This is how it would have been done in the 90s.
Comment: I couldn't agree more. To me this looks like categorical data which I wouldn't call layout at all so this is correct.
Comment: If it is indeed tabular data, then yep. ADA requirements would be better met with a table when it comes to tabular data, however a reader would assist the visually impaired user better if the table had a table row (tr) at the top that contained table headings (th). Cheers
Comment: There should be a badge for having an accepted answer which is in negative votes lol


Dvorak vs QWERTY scancodes

Question: Do Dvorak keyboards use the same scan codes as QWERTY keyboards? I mean, does the letter U have the same scan code on both types of keyboards?
Or, is it more like taking a QWERTY keyboard and moving the plastic key tops around to mod it into a Dvorak keyboard?
Thanks.
Comment: For macOS, key codes reference the physical QWERTY position: https://stackoverflow.com/a/16125341/1265393
Answer:
Do Dvorak keyboards use the same scan codes as QWERTY keyboards?

This has no single answer, but the typical and recommended approach is that scancodes are attached to positions on keyboard, but usually named according to the "default" QWERTY layout. See ISO 9995 for the main scheme and scancode set descriptions. For example, position D01 has code 0x10 in PC set 1 despite it is Q on QWERTY, A on AZERTY or single quote on Dvorak.
The USB "HID usage tables" document explicitly says:

Note A general note on Usages and languages: Due to the variation of keyboards from language to language, it is not feasible to specify exact key mappings for every language. Where this list is not specific for a key function in a language, the closest equivalent key position should be used, so that a keyboard may be modified for a different language by simply printing different keycaps. One example is the Y key on a North American keyboard. In Germany this is typically Z. Rather than changing the keyboard firmware to put the Z Usage into that place in the descriptor list, the vendor should use the Y Usage on both the North American and German keyboards. This continues to be the existing practice in the industry, in order to minimize the number of changes to the electronics to accommodate other languages.

After scancodes are translated to hardware-independent codes, layout translation is applied and this is where difference between takes place.
Answer: Yes and No
Key-codes specify the position
Usually scan code applies to the key position (not letter), and is the same for all keyboards.
For example my key-caps are printed for querty, but the OS (well actually the windowing system) maps them to dvorak (so long as I don't look down it is ok).
Sometimes key-codes, change
Some keyboards, move the position of the key-codes. This is done so that they work on systems that can not have the keyboard layout changed. Some have a switch, to change between dvorak and querty.

Dvorak vs QWERTY scancodes


Representing a multiline value in Augeas using '\'

Question: I recently assembled a lens in Augeas and now it's not functioning quite as desired.
My file contains a few references with the following syntax:
Value1 = KEY 32 OR\
   KEY 33 OR\
   ....

How do I represent the backslash newline combination and treat it as part of the same value?
EDIT - I'm running version .10.0
...
  let comment  = IniFile.comment IniFile.comment_re IniFile.comment_default
  let sep      = IniFile.sep IniFile.sep_re IniFile.sep_default
  let empty    = IniFile.empty

  let setting   = IniFile.entry_re
  let title   =  IniFile.indented_title_label "target" IniFile.record_label_re
  let entry    = [ key IniFile.entry_re . sep . IniFile.sto_to_comment . (comment|IniFile.eol) ] |
                 [ key IniFile.entry_re . store // . (comment|IniFile.eol) ] |
                 [ key /\![A-Za-z][A-Za-z0-9\._-]+/ . del / / " " . store /\/[A-Za-z0-9\.\/_-]+/ . (comment|IniFile.eol) ] |
                 comment

  let record    = IniFile.record title entry
  let record_anon = [ label ".anon" . ( entry | empty )+ ]

  let lns       = record_anon | record*

...
Comment: Can you post your lens code, too?
Comment: Are you sure your file is really an IniFile? IniFiles usually don't use a `\` for multiline values, just a newline with an indented line.
Comment: @ℝaphink - It isn't perfect but it was easier than creating any additional primitives.
Comment: I mean, do you really have sections in your file, or is it more of a [`Simplevars`](http://git.fedorahosted.org/cgit/augeas.git/tree/lenses/tests/test_simplevars.aug) kind of file with continuated lines?
Comment: @ℝaphink  -There are sections included. Most of my files have sections, a few use the anon entries.
Comment: In that case, you will need to create a new kind of entry, similar to what is used in `Sudoers` or `Shellvars`, to support these continuated lines.
Answer: Your lens is not written for multiline entries. If you want to support multiline entries, you need to use IniFile.entry_multiline:
let entry_multiline (kw:regexp) (sep:lens) (comment:lens)
                       = [ key kw . sep . sto_multiline? . (comment|eol) ] | comment

However, this definition doesn't recognize \ as a line continuation symbol because I've never seen an IniFile use it to this day. While \ is commonly used as a line continuation symbol for various key/value formats, IniFiles usually simply use indentation to determine if a line is a continuation of the previous one.
As a note, I don't think your definition of lns is really what you intend it to be:
let lns       = record_anon | record*

means, "one record_anon, followed by zero, one or more records", while I guess you probably mean "zero, one or more record_anon, followed by zero, one or more records", which is:
let lns       = record_anon* . record*


class not found during run test with uiautomator

Question: There is my class:
package org.akovtunenko.test;

import com.android.uiautomator.core.UiObject;
import com.android.uiautomator.core.UiObjectNotFoundException;
import com.android.uiautomator.core.UiSelector;
import com.android.uiautomator.testrunner.UiAutomatorTestCase;

public class LoginGmailTest extends UiAutomatorTestCase {

    public void testClick() throws UiObjectNotFoundException
    {
        UiObject seven = new UiObject(new UiSelector().resourceId("com.slice.qa:id/btnLogin"));
        seven.click();
        getUiDevice().pressBack();
    }
}

I exported jar file  using Eclipse and push it to my Android device.
Where I run this class in this way
adb shell  uiautomator runtest /mnt/sdcard/accept.jar -c org.akovtunenko.test.LoginGmailTest

I got error:

INSTRUMENTATION_RESULT: shortMsg=java.lang.RuntimeException
  INSTRUMENTATION_RESULT: longMsg=Didn't find class "org.akovtunenko.test.LoginGmailTest" on path: DexPathList[[zip file "/system/framework/android.test.runner.jar", zip file "/system/framework/uiautomator.jar", zip file "/mnt/sdcard/accept.jar"],nativeLibraryDirectories=[/vendor/lib, /system/lib]]
  INSTRUMENTATION_CODE: 0

How can I run test class on my  android device?
Comment: I build with ant and uibuild.xml and this solve problem
Answer: I am build with uibuild.xml and ant  using command android create project


kafka startup failure (version 0.8.0 beta1 )

Question: I'm trying to start a kafka service with zookeeper version (3.3.6) on standalone mode (on ec2).
So I run 1) sbt update 2) sbt package 3)sbt assembly-package-dependency    and then start the zookeeper service, and then the kafka server. However, I received the following error message:
for kafka server log:
ERROR Error while electing or becoming leader on broker 0 (kafka.server.ZookeeperLeaderElector)
java.net.ConnectException: Connection timed out
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:465)
    at sun.nio.ch.Net.connect(Net.java:457)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
    at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
    at kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:84)
    at kafka.controller.ControllerChannelManager$$anonfun$1.apply(ControllerChannelManager.scala:35)
    at kafka.controller.ControllerChannelManager$$anonfun$1.apply(ControllerChannelManager.scala:35)
    at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
    at kafka.controller.ControllerChannelManager.<init>(ControllerChannelManager.scala:35)
    at kafka.controller.KafkaController.startChannelManager(KafkaController.scala:503)
    at kafka.controller.KafkaController.initializeControllerContext(KafkaController.scala:467)
    at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:215)
    at kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:89)
    at kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:53)
    at kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:106)
    at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:549)
    at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)

for zookeeper log:
    2014-07-15 15:49:22,996 - INFO  [ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x1473b82f52e0004 type:create cxid:0x57 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics/edwintest Error:KeeperErrorCode = NodeExists for /brokers/topics/edwintest
2014-07-15 15:49:23,102 - INFO  [ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x1473b82f52e0004 type:create cxid:0x59 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics/edwintest Error:KeeperErrorCode = NodeExists for /brokers/topics/edwintest
2014-07-15 15:49:23,109 - INFO  [ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x1473b82f52e0004 type:create cxid:0x5b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics/edwintest Error:KeeperErrorCode = NodeExists for /brokers/topics/edwintest
2014-07-15 15:49:23,215 - INFO  [ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x1473b82f52e0004 type:create cxid:0x5d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics/edwintest Error:KeeperErrorCode = NodeExists for /brokers/topics/edwintest

for kafka producer log:
[2014-07-15 15:49:23,107] INFO Fetching metadata from broker id:0,host:localhost,port:9092 with correlation id 23 for 1 topic(s) Set(edwintest) (kafka.client.ClientUtils$)
[2014-07-15 15:49:23,107] INFO Connected to localhost:9092 for producing (kafka.producer.SyncProducer)
[2014-07-15 15:49:23,111] INFO Disconnecting from localhost:9092 (kafka.producer.SyncProducer)
[2014-07-15 15:49:23,111] WARN Error while fetching metadata [{TopicMetadata for topic edwintest ->
No partition metadata for topic edwintest due to kafka.common.LeaderNotAvailableException}] for topic [edwintest]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2014-07-15 15:49:23,112] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: edwintest (kafka.producer.async.DefaultEventHandler)
[2014-07-15 15:49:23,112] INFO Back off for 100 ms before retrying send. Remaining retries = 0 (kafka.producer.async.DefaultEventHandler)
[2014-07-15 15:49:23,213] INFO Fetching metadata from broker id:0,host:localhost,port:9092 with correlation id 24 for 1 topic(s) Set(edwintest) (kafka.client.ClientUtils$)
[2014-07-15 15:49:23,213] INFO Connected to localhost:9092 for producing (kafka.producer.SyncProducer)
[2014-07-15 15:49:23,217] INFO Disconnecting from localhost:9092 (kafka.producer.SyncProducer)
[2014-07-15 15:49:23,218] WARN Error while fetching metadata [{TopicMetadata for topic edwintest ->
No partition metadata for topic edwintest due to kafka.common.LeaderNotAvailableException}] for topic [edwintest]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2014-07-15 15:49:23,219] ERROR Failed to send requests for topics edwintest with correlation ids in [17,24] (kafka.producer.async.DefaultEventHandler)
[2014-07-15 15:49:23,219] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
    at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
    at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
    at scala.collection.immutable.Stream.foreach(Stream.scala:254)
    at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
    at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)

my /etc/hosts config
127.0.0.1       ip-172-32-1-95 localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6

my server.properties file
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to and advertise to producers and consumers.
# If not set, the server will bind to all interfaces and advertise the value returned from
# from java.net.InetAddress.getCanonicalHostName().
#host.name=localhost

# The number of threads handling network requests
num.network.threads=2

# The number of threads doing disk I/O
num.io.threads=2

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=1048576

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=1048576

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600

############################# Log Basics #############################

# The directory under which to store log files
log.dir=/tmp/kafka-logs

# The number of logical partitions per topic per server. More partitions allow greater parallelism
# for consumption, but also mean more files.
num.partitions=1

############################# Log Flush Policy #############################

# The following configurations control the flush of data to disk. This is the most
# important performance knob in kafka.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data is at greater risk of loss in the event of a crash.
#    2. Latency: Data is not made available to consumers until it is flushed (which adds latency).
#    3. Throughput: The flush is generally the most expensive operation.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
log.flush.interval.ms=1000

# Per-topic overrides for log.flush.interval.ms
#log.flush.intervals.ms.per.topic=topic1:1000, topic2:3000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=536870912

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.cleanup.interval.mins=1

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=1000000

# metrics reporter properties
kafka.metrics.polling.interval.secs=5
kafka.metrics.reporters=kafka.metrics.KafkaCSVMetricsReporter
kafka.csv.metrics.dir=/tmp/kafka_metrics
# Disable csv reporting by default.
kafka.csv.metrics.reporter.enabled=false

my zookeeper config  zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181

I tried to delete all the information for kafka and zookeeper under /tmp/zookeeper or /tmp/kafka-logs, and restart everything, but still receive the same error.
Comment: I am certainly not an operations guy, but your hosts file looks odd. Have you altered it for some purpose? If you're running single broker the problem is very likely is that you have misconfigured loopback interface.
Answer: Cool! I guess you are running kafka-console-producer to publish a message to a topic "edwintest". Before running your producer, create the topic using this command
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 5 --topic edwintest
And the start your console producer. Hope this should fix your problem.
[EDIT]
Obviously you have to ensure your ec2 security groups are updated properly to open zk and kafka broker ports for producers.


Calculation of integral $\int\exp \left(-\alpha \sin^2 \left(\frac{x}{2} \right) \right) dx$

Question: Given $\alpha$ is a constant. How to calculate the following integral?
\begin{equation}
\int \exp \bigg(-\alpha \sin^2 \bigg(\frac{x}{2} \bigg) \bigg) dx
\end{equation}
Thanks for your answer.
Comment: I don't think this is going to be elementary.
Comment: @OP: Are you sure you are asked to calculate the indefinite integral not a definite one?
Answer: The indefinite integral cannot be expressed in terms of elementary functions. See Liouville's theorem and the Risch algorithm for more details. However, for $n\in$ Z we have the following identity in terms involving Bessel functions:
$$\int_0^{n\pi}e^{-\alpha\sin^2\Big(\tfrac x2\Big)}dx=\frac{n\cdot\pi}{\sqrt{e^\alpha}}\cdot\text{Bessel I}_0\bigg(\frac\alpha2\bigg)$$
Comment: @mvw: See *Graham Hesketh's* answer. Also, one could simply expand the integrand using the Taylor series for the exponential function, then switch the order of summation and integration and use [Wallis' integrals](http://en.wikipedia.org/wiki/Wallis'_integrals) to re-write the integral in terms of an infinite series, which is then identified as belonging to the modified Bessel function *I*, using the definition provided in the links.
Comment: Can you explain how you derived it?
Answer: A series solution is given via the Jacobi-Anger expansion:
$$-\alpha\, \left( \sin \left( \frac{1}{2}\,x \right)  \right) ^{2}=\frac{1}{2}\,\alpha
\, \left( \cos \left( x \right) -1 \right) $$
$${{\rm e}^{\frac{1}{2}\,\alpha\, \left( \cos \left( x \right) -1 \right) }}={
{\rm e}^{-\frac{1}{2}\,\alpha}}\,{{\rm I}_0\left(\frac{1}{2}\,\alpha\right)}+2\,{
{\rm e}^{-\frac{1}{2}\,\alpha}}\sum _{n=1}^{\infty }
{{\rm I}_n\left(\frac{1}{2}\,\alpha\right)}\cos \left( nx \right)$$
where:
$${i}^{n}{{\rm J}_n\left(-\frac{1}{2}\,i\alpha\right)}=
{{\rm I}_n\left(\frac{1}{2}\,\alpha\right)}$$
was used to relate the Bessel function of the first kind $J$ to the modified Bessel function $I$. Integration term by term gives:
$$\int \!{{\rm e}^{\frac{1}{2}\,\alpha\, \left( \cos \left( x \right) -1
 \right) }}{dx}={{\rm e}^{-\frac{1}{2}\,\alpha}}
{{\rm I}_0\left(\frac{1}{2}\,\alpha\right)}x+2\,{{\rm e}^{-\frac{1}{2}\,\alpha}}
\sum _{n=1}^{\infty }{\frac {{{\rm I}_n\left(\frac{1}{2}\alpha\right)}
\sin \left( nx \right) }{n}}$$
For integration between zero and integer multiples of $\pi$ the sum clearly vanishes and you recover @Lucian's solution.
Answer: Let $u=\dfrac{x}{2}$ ,
Then $x=2u$
$dx=2~du$
$\therefore\int e^{-\alpha\sin^2\frac{x}{2}}~dx$
$=2\int e^{-\alpha\sin^2u}~du$
$=2\int\sum\limits_{n=0}^\infty\dfrac{(-1)^n\alpha^n\sin^{2n}u}{n!}du$
$=2\int\left(1+\sum\limits_{n=1}^\infty\dfrac{(-1)^n\alpha^n\sin^{2n}u}{n!}\right)du$
For $n$ is any natural number,
$\int\sin^{2n}u~du=\dfrac{(2n)!u}{4^n(n!)^2}-\sum\limits_{k=1}^n\dfrac{(2n)!((k-1)!)^2\sin^{2k-1}u\cos u}{4^{n-k+1}(n!)^2(2k-1)!}+C$
This result can be done by successive integration by parts.
$\therefore2\int\left(1+\sum\limits_{n=1}^\infty\dfrac{(-1)^n\alpha^n\sin^{2n}u}{n!}\right)du$
$=2u+2\sum\limits_{n=1}^\infty\dfrac{(-1)^n\alpha^n(2n)!u}{4^n(n!)^3}-2\sum\limits_{n=1}^\infty\sum\limits_{k=1}^n\dfrac{(-1)^n\alpha^n(2n)!((k-1)!)^2\sin^{2k-1}u\cos u}{4^{n-k+1}(n!)^3(2k-1)!}+C$
$=\sum\limits_{n=0}^\infty\dfrac{(-1)^n\alpha^n(2n)!2u}{4^n(n!)^3}-\sum\limits_{n=1}^\infty\sum\limits_{k=1}^n\dfrac{(-1)^n\alpha^n(2n)!((k-1)!)^2\sin^{2k-1}u\cos u}{2^{2n-2k+1}(n!)^3(2k-1)!}+C$
$=\sum\limits_{n=0}^\infty\dfrac{(-1)^n\alpha^n(2n)!x}{4^n(n!)^3}-\sum\limits_{n=1}^\infty\sum\limits_{k=1}^n\dfrac{(-1)^n\alpha^n(2n)!((k-1)!)^2\sin^{2k-1}\dfrac{x}{2}\cos\dfrac{x}{2}}{2^{2n-2k+1}(n!)^3(2k-1)!}+C$
Or you can express in terms of the incomplete bessel K function:
$\int e^{-\alpha\sin^2\frac{x}{2}}~dx$
$=\int e^\frac{\alpha(\cos x-1)}{2}~dx$
$=e^{-\frac{\alpha}{2}}\int e^\frac{\alpha\cos x}{2}~dx$
$=e^{-\frac{\alpha}{2}}\int e^\frac{\alpha(e^{ix}+e^{-ix})}{4}~dx$
$=e^{-\frac{\alpha}{2}}\int e^{\frac{\alpha e^{-ix}}{4}+\frac{\alpha}{4e^{-ix}}}~dx$
Let $u=e^{-ix}$ ,
Then $x=i\ln u$
$dx=\dfrac{i}{u}du$
$\therefore e^{-\frac{\alpha}{2}}\int e^{\frac{\alpha e^{-ix}}{4}+\frac{\alpha}{4e^{-ix}}}~dx$
$=ie^{-\frac{\alpha}{2}}\int\dfrac{e^{\frac{\alpha u}{4}+\frac{\alpha}{4u}}}{u}du$
$=ie^{-\frac{\alpha}{2}}\int_0^u\dfrac{e^{\frac{\alpha t}{4}+\frac{\alpha}{4t}}}{t}dt+C$
$=ie^{-\frac{\alpha}{2}}\int_0^1\dfrac{e^{\frac{\alpha ut}{4}+\frac{\alpha}{4ut}}}{ut}d(ut)+C$
$=ie^{-\frac{\alpha}{2}}\int_0^1\dfrac{e^{\frac{\alpha ut}{4}+\frac{\alpha}{4ut}}}{t}dt+C$
$=ie^{-\frac{\alpha}{2}}\int_\infty^1te^{\frac{\alpha u}{4t}+\frac{\alpha t}{4u}}~d\left(\dfrac{1}{t}\right)+C$
$=ie^{-\frac{\alpha}{2}}\int_1^\infty\dfrac{e^{\frac{\alpha t}{4u}+\frac{\alpha u}{4t}}}{t}dt+C$
$=ie^{-\frac{\alpha}{2}}K_0\left(-\dfrac{\alpha}{4u},-\dfrac{\alpha u}{4}\right)+C$ (according to http://artax.karlin.mff.cuni.cz/r-help/library/DistributionUtils/html/incompleteBesselK.html)
$=ie^{-\frac{\alpha}{2}}K_0\left(-\dfrac{\alpha e^{ix}}{4},-\dfrac{\alpha e^{-ix}}{4}\right)+C$


Dynamically instantiating an ActiveRecord Observer for multiple models

Question: I'm trying to develop a plugin/gem at the moment which observes multiple models. Ideally, the observer should be instantiated automatically with just one singleton method...
class MyModel < ActiveRecord::Base

  # a class method like this will tell the observer to observe this model
    observe_me

end

My initial approach was to define class methods included into AR base:
module ClassMethods

  def observe_me
    @observe_me = true
  end

  def should_observe_me?
    @observe_me
  end
end
ActiveRecord::Base.extend(ClassMethods)

And then use this to detect which models to observe within the Observer:
class MyObserver < ActiveRecord::Observer

  # this should observe all models where should_observe_me? #=> true
  observe ActiveRecord::Base.descendants.select { |m| m.try(:should_observe_me?) }.map(&:model_name)

end

The problem that I'm running into is that the observer is being loaded before the models are defined, so ActiveRecord has no descendants and MyObserver doesn't know which models to observe.
My next attempt was to hack around with ActiveRecord::Base.observers and ActiveRecord::Base.instantiate_observers but with no luck.
So, as it is at the moment:
Observer is defined but doesn't know which models to observe.
Models are defined and flag themselves to be observed but the observer has already been observed.
Is there a way I can delay the loading of the observer until after the models are defined or can someone think of a better approach to this problem?
Answer: @gavin:  The structure of the application initialization has changed in Rails3 -- this might be your problem.
When / how are you including the ClassMethods module?  IF you are in Rails3, and IF you added "require 'observe_me'" to $ROOT/config/environment.rb, then you'd see the (mis)behavior you describe.
If so, instead, create $ROOT/config/initializers/my_extensions.rb and stick the "require ..." in there.


Quadratic program over box constraints

Question: Consider $f:\mathbb{R}^n \rightarrow \mathbb{R}$ defined as
$$ f(x) := x^\top x + c^\top x $$
for some $c \in \mathbb{R}^n$. Define the (compact) "box"
$$X := \left\{ x \in \mathbb{R}^n \mid x_i \in [ x_i^{\min}, x_i^{\max} ] \ \forall i \in \{1, \dots , n\} \right\},$$
for some $\{ x_i^{\min}, x_i^{\max} \}_{i=1}^{n}$. Since $x^\star := \arg\min_{x \in \mathbb{R}^n} f(x) = -c/2$ (unconstrained minimizer) I am wondering if the constrained minimizer reads as
$$ \arg\min_{x \in X} f(x) = \text{Proj}_{X}\left( -c/2 \right) $$
where $\text{Proj}_X$ denotes the (Euclidean) projection onto $X$.
Answer: Yes, it is. Note that your problem is completely decoupled. That is, you're really solving $n$ separate scalar optimization problems:$$\begin{array}{ll}\text{minimize} & x_i^2+c_ix_i \\ \text{subject to} & x^{\min}_i \leq x \leq x^{\max}_i\end{array}$$
Note also that $x_i^2+c_ix_i=(x_i+c_i/2)^2-c_i^2/4$. Dropping the constant and then taking the square root of the square doesn't change the solution, you're effectively solving
$$\begin{array}{ll}\text{minimize} & |x_i+c_i/2| \\ \text{subject to} & x^{\min}_i \leq x \leq x^{\max}_i\end{array}$$
It should be even clearer now that your projection approach is correct.

Quadratic program over box constraints


Android Image permissions

Question: my application saves user details as a qr code png image in the android mobile's SD card,
is there a possibility to block the image from being transfered to other smartphones, (ie) blocking the user from transferring the qr image through bluetooth or mms services
can i save the image with permission allowing only to read.
Comment: i think saving the image to your applications internal memory will be the "safest" method available. I put safest in quotes because I dont believe their is a true way to make your file fully read/write proof especially when it comes to users with rooted phones that can access virtually everything in the file system.
Comment: @dymmeh saving a qrcode in other memory is not possible i hope so, because of the size of the image
Answer: Take a look at this:
setWritable
this should do the trick ...
<code>file.setWritable(false, true);
</code>
You should be able to set permissions this way.  Good Luck!!!
Answer: No, you can't reasonably stop the user from transferring a file on their SD card to another device/user. What you could do is attempt to render that file useless. You could store an encrypted file instead of just the image file and have the encryption key generated on the device in the internal directory on first use. This would mean that the encrypted image file is not usable across devices. This is circumventable (by either copying the decrypted file when it's in use or by rooting the device and finding the key), but adds a measure of prevention.
Comment: is there any good example tutorial for image encryption and decryption in android friend
Comment: Highly doubtful, and not that I know of, but it's just a file. Just look for a basic encryption tutorial.

Android Image permissions


Showing $\int_C \ln |x - y| dy = 0$ where $C$ is the unit circle?

Question: I saw it mentioned in a proof that
$$\int_C \ln |x - y| dy = 0,$$
where $x, y \in \mathbb{R}^2$ and they are both on the unit circle $C$. I'm trying to figure out the calculation but I'm not sure how to show it. First of all I switch to polars and get
$$\int_0^{2\pi} \ln \sqrt{(\cos \theta_x - \cos\theta_y)^2 - (\sin \theta_x - \sin\theta_y)^2} d\theta_y \\ = \frac{1}{2}\int_0^{2\pi} \ln (\cos^2 \theta_x - 2 \cos \theta_x \cos \theta_y + \cos^2 \theta_y + \sin^2 \theta_x - 2 \sin \theta_x \sin \theta_y + \sin^2 \theta_y) d\theta_y \\ = \frac{1}{2}\int_0^{2\pi} \ln (2 - 2 \cos \theta_x \cos \theta_y - 2 \sin \theta_x \sin \theta_y) d\theta_y \\ = \frac{1}{2}\int_0^{2\pi} \ln 2 d\theta_y  + \frac{1}{2}\int_0^{2\pi} \ln (1 - (\cos \theta_x \cos \theta_y + \sin \theta_x \sin \theta_y) d\theta_y$$
What steps are needed to get this expression equal to zero?
Comment: Note that $$\cos \theta_x \cos \theta_y + \sin \theta_x \sin \theta_y = \cos(\theta_y - \theta_x)$$ If you are going to work with trig functions, you have to learn to recognize that one and the similar one for sine.
Comment: Your integral is a 'line integral' but $y \in \mathbb{R}^{2}$. I don't understand your integral !!!.
Comment: @FelixMarin $y = (y_1, y_2)$ is a vector $\in \mathbb{R}^2$ and its components are given by the region of integration which is the unit circle. So in polar coordinates, $y$ becomes $(\cos \theta_y, \sin \theta_y)$. $x = (x_1, x_2)$ is a particular fixed point on the unit circle so in polar coordinates it is $(\cos \theta_x, \sin \theta_x)$.
Answer: $\newcommand{\bbx}[1]{\,\bbox[8px,border:1px groove navy]{\displaystyle{#1}}\,}
 \newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace}
 \newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack}
 \newcommand{\dd}{\mathrm{d}}
 \newcommand{\ds}[1]{\displaystyle{#1}}
 \newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,}
 \newcommand{\ic}{\mathrm{i}}
 \newcommand{\mc}[1]{\mathcal{#1}}
 \newcommand{\mrm}[1]{\mathrm{#1}}
 \newcommand{\pars}[1]{\left(\,{#1}\,\right)}
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,}
 \newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}}
 \newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$

Your last integral is, indeed, $\ds{\theta_{x}}$-independent $\ds{\pars{~Why\ ?~}}$. So,

\begin{align}
&{1 \over 2}\int_{0}^{2\pi}\ln\pars{1 - \cos\pars{\theta_{y}}}\,\dd\theta_{y} =
{1 \over 2}\int_{0}^{2\pi}\ln\pars{2\sin^{2}\pars{\theta_{y} \over 2}}
\,\dd\theta_{y}
\\[5mm] = &
{1 \over 2}\int_{0}^{\pi}\bracks{\ln\pars{2\sin^{2}\pars{\theta_{y} \over 2}} +
\ln\pars{2\cos^{2}\pars{\theta_{y} \over 2}}}\,\dd\theta_{y}
\\[5mm] = &\
\int_{0}^{\pi}\ln\pars{\sin\pars{\theta_{y}}}\,\dd\theta_{y} = \bbx{\ds{-\pi\ln\pars{2}}}
\end{align}

It cancels the first integral $\ds{{1 \over 2}\int_{0}^{2\pi}\ln\pars{2}\,\dd\theta_{x}}$ !!!.
Comment: Ok great thanks!
Comment: @eurocoder Thanks. I'm glad it was useful.


How to obtain the results of the Nemenyi post-hoc test in a table showing grouped pairs?

Question: I performed a Friedman's test, which showed that there are differences among my treatments. Thus, I choose to perform the Nemenyi's post-hoc test in order to infer which treatments statistically differ among them. Thus, I used the posthoc.friedman.nemenyi.test from the PMCMR package to perform pairwise comparisons between treatments.
Now, I want to divide my treatments into groups, like the example below.
Treatment 1    a
Treatment 2    ab
Treatment 3     b
Treatment 4      c

My question is: Given the p-values of a multiple-comparison post-hoc test, how can I split my treatments into groups as the above example?

Note: This question is very similar to How to obtain the results of a Tukey HSD post-hoc test in a table showing grouped pairs?. However, I want to get this information for the Nemenyi's test instead of the Tukey's test.
Comment: What do you mean by "How can I do that?"? Are you asking for code? (Note that questions that are only about how to use software are off topic here.)
Comment: This is virtually the same question as pointed in my reference (that was answered here in Cross Validated). I want an algorithm (or the idea of an algorithm) to perform such a task, not the code itself. I edited the question for a clearer objective.
Answer: After a little more of research, I found a R code that solved this question. However, I still don't know how it works.

It is possible to use the multcompLetters function of the multcompView package along with the fullPTable function of the rcompanion package.
Thus, the following code outputs the grouped treatments:
if(!require(multcompView)){install.packages("multcompView")}
if(!require(PMCMR)){install.packages("PMCMR")}
if(!require(rcompanion)){install.packages("rcompanion")}

library(PMCMR)
library(rcompanion)
library(multcompView)

pTable  = posthoc.friedman.nemenyi.test(input)$p.value
pTableComplete = fullPTable(pTable)
multcompLetters(pTableComplete)


Carregar uma lista em json no JS

Question: Gostaria de carregar uma lista do meu arquivo JSON para usar no meu JavaScript ,
para usar o console.table() com as informações fornecidas ,
Esse é o código meu do JavaScript :

fetch("infos.json") // especifique o caminho
.then(response => response.json())
.then(json => console.table(json))

E esse é o do JSON :

{"Jogadores":[{"Jogador":"Player1","Pontuação":"15 pnts"},{"Jogador":"Player2","Pontuação":"12,3 pnts"},{"Jogador":"Player3","Pontuação":"11,6 pnts"},{"Jogador":"Player4","Pontuação":"11,2 pnts"}]}

Comment: Qual o erro que está gerando?
Comment: Desculpe agor fui ver tem um erro de caracteres no JSON !
Comment: Sou novo em progamação
Comment: Erro no JSON : SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 10 of the JSON data .
Comment: Adicione o erro diretamente na pergunta.
Answer: Basta executar um fetch
fetch("dados.json") // especifique o caminho
.then(response => response.json())
.then(json => console.table(json))

@EDIT agora que vi o erro de sintaxe. De qualquer forma, fica a sugestão de se usar o fetch :)

Carregar uma lista em json no JS


How to make worker threads quit after work is finished in a multithreaded producer-consumer pattern?

Question: I am trying to implement a multithreaded producer-consumer pattern using
Queue.Queue in Python 2.7. I am trying to figure out how to make the
consumers, i.e. the worker threads, stop once all required work is done.
See the second comment by Martin James to this answer: https://stackoverflow.com/a/19369877/1175080

Send an 'I am finished' task, instructing the pool threads to terminate. Any thread that gets such a task requeues it and then commits suicide.

But this does not work for me. See the following code for example.
import Queue
import threading
import time

def worker(n, q):
    # n - Worker ID
    # q - Queue from which to receive data
    while True:
        data = q.get()
        print 'worker', n, 'got', data
        time.sleep(1)  # Simulate noticeable data processing time
        q.task_done()
        if data == -1: # -1 is used to indicate that the worker should stop
            # Requeue the exit indicator.
            q.put(-1)
            # Commit suicide.
            print 'worker', n, 'is exiting'
            break

def master():
    # master() sends data to worker() via q.
    q = Queue.Queue()

    # Create 3 workers.
    for i in range(3):
        t = threading.Thread(target=worker, args=(i, q))
        t.start()

    # Send 10 items to work on.
    for i in range(10):
        q.put(i)
        time.sleep(0.5)

    # Send an exit indicator for all threads to consume.
    q.put(-1)

    print 'waiting for workers to finish ...'
    q.join()
    print 'done'

master()

This program hangs after all three workers have read the exit indicator,
i.e. -1 from the queue, because each worker requeues -1 before
exiting, so the queue never becomes empty and q.join() never returns.
I came up with the following but ugly solution where I send a -1 exit
indicator for each worker via the queue, so that each worker can see it
and commit suicide. But the fact that I have to send an exit indicator
for each worker feels a little ugly.
import Queue
import threading
import time

def worker(n, q):
    # n - Worker ID
    # q - Queue from which to receive data
    while True:
        data = q.get()
        print 'worker', n, 'got', data
        time.sleep(1)  # Simulate noticeable data processing time
        q.task_done()
        if data == -1: # -1 is used to indicate that the worker should stop
            print 'worker', n, 'is exiting'
            break

def master():
    # master() sends data to worker() via q.
    q = Queue.Queue()

    # Create 3 workers.
    for i in range(3):
        t = threading.Thread(target=worker, args=(i, q))
        t.start()

    # Send 10 items to work on.
    for i in range(10):
        q.put(i)
        time.sleep(0.5)

    # Send one stop indicator for each worker.
    for i in range(3):
        q.put(-1)

    print 'waiting for workers to finish ...'
    q.join()
    print 'done'

master()

I have two questions.

Can the method of sending a single exit indicator for all threads (as explained in the second comment of https://stackoverflow.com/a/19369877/1175080 by Martin James) even work?
If the answer to the previous question is "No", is there a way to solve the problem in a way that I don't have to send a separate exit indicator for each worker thread?
Comment: Note that there is a [`ThreadPool`](https://stackoverflow.com/a/3386632/3767239) class available which takes the load of "manually" distributing tasks between multiple threads. You can `join` such a pool (instead of the queue) and then sending the *"stop"* signal will eventually terminate all threads. Actually I don't see why you want to `join` the queue instead of the threads here. With Python 3 you have even more functionality (and better documented) through the [concurrent](https://docs.python.org/dev/library/concurrent.futures.html#threadpoolexecutor) module.
Comment: sending a killing signal for each worked looks like a good solution for me, I wouldnt say is so ugly. You can also just join the threads instead of join the queue
Comment: Some additional remarks. From your example code it is not clear why you would use such a *"stop"* command in the first place (you could just leave that part out and wait for the queue to `join`). Then - if you use such a command - it is not guaranteed that each thread will be shut down "properly": `q.join()` might resume before all threads received the `-1` because you call `q.task_done()` before re-putting the `-1` to the queue (which means the task count may reach zero before `-1` is re-put (which increases the count) and thus `q.join()` may resume).
Comment: @a_guest Have you actually taken my code, edited it as you suggest, and seen the outcome? If I leave that part ("stop" command) out and only wait for the queue to join, then the program hangs forever after the queue joins, i.e. after `q.join()` returns because the worker threads do not exit. That's why the "stop" command is there to ensure that the worker threads commit suicide and the main program can terminate.
Comment: @a_guest In this specific example, there is no possibility of `q.join()` resuming prematurely because at least one `q.task_done()` occurs after a `q.put(-1)` due to the artificial 1-second delay in the worker threads. However, what you mention is true for real-world program like this where there would be no such artificial delay.
Comment: @LoneLearner If that is your concern, just add `os._exit(0)` at the end - not a "clean" exit but your example code doesn't seem to require one anyway (no resources to be released, ...). About the `q.join`: A worker needs 1 second to complete a task, there are 3 workers and 0.5 seconds between two subsequently queued tasks. This means each task will be immediately consumed by a worker (also the `-1`). Because you `sleep(1)` also when `-1` is received, by the moment this call returns all other workers have finished their tasks already 0.5 seconds ago and `q.task_done()` brings the count to zero.
Comment: @a_guest Did you actually run my code and observe the behavior you mention, or are you just theorizing? What you mention about `q.join()` does not happen if I run this code on my system. Here's why: After the last `q.put()` call in the `for`-loop in the master, i.e. the `q.put(9)` call in the master, some worker receives gets this `9` from the queue by calling `q.get()`. After this, that worker sleeps for 1 second but the master sleeps only for 0.5 second and then immediately calls `q.put(-1)` three times. So the queue is never really empty until all the three workers get these `-1`s.
Comment: @a_guest In fact, in my second example code it is guaranteed that the following sequence of operations occur: a worker gets 0, another gets 1, the worker that got 0 calls `q.task_done()`, ..., some worker gets 8, some worker that got 7 calls `q.task_done()`, another worker gets 9, the worker that got 8 calls `q.task_done()`, the three `-1`s are sent, two free workers (that had 7 and 8 earlier) get the `-1`s, the worker that got 9 now calls `q.task_done()` and then it gets the last `-1`. So the `q` is never empty until the worker that once had 9 gets the last `-1` and calls `q.task_done()`.
Comment: @LoneLearner I didn't refer to your second example but to your first (where you put only a single `-1`). My point is that the queue's task count drops to zero *before* all threads received (and processed) the `-1`. Admittedly this was "theoretical" in a sense that I didn't observe that behavior (which is probably due to `-1` being re-put "immediately" after `q.task_done()`). However if you add (for example) `time.sleep(0.001)` in the `if`-branch before `q.put(-1)` then you will observe `q.join()` to resume *before* all threads have received the `-1`.
Comment: @a_guest I agree that you are right about what you mention about the first code example.
Comment: Why structure the multithreading this way in the first place? I think this approach attempts to use the `Queue()` in an unintended way, specifically overloading it as a means of thread communication (sending `-1` "kill" signals). Not queueing these signals makes the application behave as expected (when the queue is empty, the parent thread unblocks and exits killing the child threads).
Comment: @tdube I have already explained that in my question. See second paragraph where I explain that I got this idea from the second comment by Martin James to this answer: https://stackoverflow.com/a/19369877/1175080 . Do you have a suggestion on how to better structure multithreading here? Also, Travis' solution ( https://stackoverflow.com/a/45471622/1175080 ) below seems to solve the problem without the need for any overloading of `Queue.Queue` with sending kill signals.
Comment: @LoneLearner My recommendation would change based on the actual problem you're trying to solve. Yes, Travis' solution looks good to me.
Answer: Don't call it a special case for a task.
Use an Event instead, with non-blocking implementation for your workers.
stopping = threading.Event()

def worker(n, q, timeout=1):
    # run until the master thread indicates we're done
    while not stopping.is_set():
        try:
            # don't block indefinitely so we can return to the top
            # of the loop and check the stopping event
            data = q.get(True, timeout)
        # raised by q.get if we reach the timeout on an empty queue
        except queue.Empty:
            continue
        q.task_done()

def master():
    ...

    print 'waiting for workers to finish'
    q.join()
    stopping.set()
    print 'done'

Comment: I believe you meant `q.get(True, timeout)`, i.e. you meant the `block` parameter of `q.get()` to be `True` instead of `False`. If you set it as `False`, then the `q.get()` does not block at all and returns immediately which causes the `while` loop to spin very fast. If you set it to `True`, then it blocks for `timeout` seconds before unblocking.
Answer: Can the method of sending a single exit indicator for all threads (as explained in the second comment of https://stackoverflow.com/a/19369877/1175080 by Martin James) even work?
As you have notice it can't work, spreading the message will make the last thread to update the queue with one more item and since you are waiting for a queue that will never be empty, not with the code you have.
If the answer to the previous question is "No", is there a way to solve the problem in a way that I don't have to send a separate exit indicator for each worker thread?
You can join the threads instead of the queue:
def worker(n, q):
    # n - Worker ID
    # q - Queue from which to receive data
    while True:
        data = q.get()
        print 'worker', n, 'got', data
        time.sleep(1)  # Simulate noticeable data processing time
        q.task_done()
        if data == -1: # -1 is used to indicate that the worker should stop
            # Requeue the exit indicator.
            q.put(-1)
            # Commit suicide.
            print 'worker', n, 'is exiting'
            break

def master():
    # master() sends data to worker() via q.
    q = Queue.Queue()

    # Create 3 workers.
    threads = [threading.Thread(target=worker, args=(i, q)) for i in range(3)]
    for t in threads:
        threads.start()
    # Send 10 items to work on.
    for i in range(10):
        q.put(i)
        time.sleep(0.5)

    # Send an exit indicator for all threads to consume.
    q.put(-1)

    print 'waiting for workers to finish ...'
    for t in threads:
        t.join()
    print 'done'

master()

As the Queue documentation explain get method will rise an execption once its empty so if you know already the data to process you can fill the queue and then spam the threads:
import Queue
import threading
import time

def worker(n, q):
    # n - Worker ID
    # q - Queue from which to receive data
    while True:
        try:
            data = q.get(block=False, timeout=1)
            print 'worker', n, 'got', data
            time.sleep(1)  # Simulate noticeable data processing time
            q.task_done()
        except Queue.Empty:
            break

def master():
    # master() sends data to worker() via q.
    q = Queue.Queue()

    # Send 10 items to work on.
    for i in range(10):
        q.put(i)

    # Create 3 workers.
    for i in range(3):
        t = threading.Thread(target=worker, args=(i, q))
        t.start()

    print 'waiting for workers to finish ...'
    q.join()
    print 'done'

master()

Here you have a live example
Comment: Could you please answer my first question too?
Comment: @LoneLearner, yes, give me a sec :)
Comment: @DanielSanchez Your answer is good, but your second example changed the question a bit. The OP instantiated his worker threads first and then populated the queue, while you swap those tasks. If you didn't swap those tasks, you might have some worker threads exiting early (due to timeout) if they see the empty queue before work loads are added. Your approach of removing the additional thread communication signals is appropriate. Personally, I think having threads block for work and the main thread controlling exit is the cleanest approach for most applications.
Comment: @tdube, yes, that is just about changing the timeout, I specified that if he knows the exact number or tasks (if he dont want to stream them) its better to fill the queue first so I changed the example for a reason I explained.  If it is not clear enough Im open for an expand. any sugestion? Thanks
Comment: Sorry, I guess I missed your reference to that. :)
Answer: Just for completeness sake:
You could also enqueue a stop signal which is -(thread count).
Each thread then can increment it by one and re queue it only if the stop signal is != 0.
    if data < 0: # negative numbers are used to indicate that the worker should stop
        if data < -1:
            q.put(data + 1)
        # Commit suicide.
        print 'worker', n, 'is exiting'
        break

But i'd personally go with Travis Mehlinger or
Daniel Sanchez answer.
Answer:
In addition to @DanielSanchez excellent answer, I propose to actually
  rely on a similar mechanism as a Java CountDownLatch.

The gist being,

you create a latch that will open only after a certain counter went down,
when the latch is opened, the thread(s) waiting on it will be allowed to proceed with their execution.
I made an overly simple example, check here for a class like example of such a latch:
import threading
import Queue
import time

WORKER_COUNT = 3
latch = threading.Condition()
count = 3

def wait():
    latch.acquire()
    while count > 0:
        latch.wait()
    latch.release()

def count_down():
    global count
    latch.acquire()
    count -= 1
    if count <= 0:
        latch.notify_all()
    latch.release()

def worker(n, q):
    # n - Worker ID
    # q - Queue from which to receive data
    while True:
        data = q.get()
        print 'worker', n, 'got', data
        time.sleep(1)  # Simulate noticeable data processing time
        q.task_done()
        if data == -1: # -1 is used to indicate that the worker should stop
            # Requeue the exit indicator.
            q.put(-1)
            # Commit suicide.
            count_down()
            print 'worker', n, 'is exiting'
            break

# master() sends data to worker() via q.

def master():
    q = Queue.Queue()

    # Create 3 workers.
    for i in range(WORKER_COUNT):
        t = threading.Thread(target=worker, args=(i, q))
        t.start()

    # Send 10 items to work on.
    for i in range(10):
        q.put(i)
        time.sleep(0.5)

    # Send an exit indicator for all threads to consume.
    q.put(-1)
    wait()
    print 'done'

master()


Why does my model output show levels within the fixed effects and interactions?

Question: Apologies in advance for a possible duplicate or off-topic question, but I haven't been able to solve this via Googling and I'm not even sure of the search terms, because I don't know if this is a problem where I need to improve my understanding of generalised linear mixed models, or if it is more a problem relating to using R.
Anyways, I'm analysing correct/incorrect responses from an experiment with categorical factors A (levels = 2, ordered), B (levels = 3, ordered), and covariate C with a logit glmer.
Model ~ A * B * C + (1|randomEffect1) + (A|randomEffect2)

I use Anova from the Car package to calculate chi-sq statistics for reporting the fixed effects. The problem is that this is the output for summary(Model):
Fixed effects:
                      Estimate Std. Error z value Pr(>|z|)
(Intercept)           -0.52272    0.17052  -3.066 0.002173 **
C                     -0.38642    0.16332  -2.366 0.017979 *
B2                    -0.64491    0.22017  -2.929 0.003399 **
C3                    -0.05960    0.18619  -0.320 0.748893
C2                    -0.08181    0.13666  -0.599 0.549427
C:A2                  -0.37842    0.21410  -1.767 0.077146 .
C:B3                  -0.29552    0.15930  -1.855 0.063581 .
C:B2                  -0.30852    0.12110  -2.548 0.010846 *
A2:B3                 -1.11872    0.29313  -3.816 0.000135 ***
A2:B2                 -0.52202    0.20609  -2.533 0.011311 *
C:A2:B3                0.57988    0.28050   2.067 0.038703 *
C:A2:B2                0.28810    0.18973   1.518 0.128890

However, the output for car::Anova(Model, test.statistic = "Chisq", type=2):
                  Chisq Df Pr(>Chisq)
C                 12.6060  1  0.0003845 ***
A                 19.2982  1  1.118e-05 ***
B                 15.9511  2  0.0003438 ***
C:A               1.3361  1  0.2477263
C:B               4.2334  2  0.1204260
A:B               22.1787  2  1.527e-05 ***
A:B:C             5.4237  2  0.0664152 .

Is this a theoretical issue, or is there a programmatic way that I can harmonise the outputs so that they're organised the same way? Ideally, I would prefer if summary(Model) matched the more concise car::Anova(Model, test.statistic = "Chisq", type=2).
Answer:
Why does my model output show levels within the fixed effects and interactions?

Imagine for a moment if it didn't, and to keep things simple let's suppose you have only one fixed effect, a factor, say Eye Colour, with three levels, say "Blue", "Brown" and "Other". Now, let us suppose that you are interested in the association of eye colour with some outcome variable, $Y$. Since the question has nothing to do with generalised, or mixed effects, models, you fit a model such as
lm(Y ~ EyeColour, mydata)

And, (by default in R for example), you will obtain an estimate for the intercept, which will be the expected value of $Y$ for people with blue eyes (assuming that blue is the reference level). The model will also produce estimates for people with brown eyes, which will be the estimated difference in $Y$ for people with blue and brown eyes; and another estimate for "other" which will be the estimated difference between those with blue and other coloured eyes. So there will be 3 estimates, providing inference about the the response, $Y$, for each eye colour.

Ideally, I would prefer if summary(Model) matched the more concise car::Anova(Model, test.statistic = "Chisq", type=2).

Given what I said above, this would not make sense at all. If you want to test the significance of a variable in a model via a likelihood ratio test, then you could use Anova for that, but typically you also want the estimates for the individual levels of a factor, and Anova won't give you that.
Comment: Thanks for your clear explanation! I am obviously still coming to terms with how these different models work. I think that in my previous use of `glmer`, I worked with numeric variables or factors with only two levels, hence my confusion.
Comment: You're welcome.


TypeError: Cannot read property 'join' of undefined

Question: i am trying to integrate select2 in my html template but i keep getting this error. i want to be able to select multiple options at a time
i am not sure how to go about it.
help would be appreciated.
admin.ts

import { Component, OnInit } from '@angular/core';
import { Select2OptionData } from 'ng2-select2';

@Component({
  selector: 'app-admin',
  templateUrl: './admin.component.html',
  styleUrls: ['./admin.component.scss']
})
export class AdminComponent implements OnInit {

  public candidateData: Array<Select2OptionData>;
  public options: Select2Options;
  public value: string[];
  public current: string;

  constructor() { }

  ngOnInit() { this.candidateData = [
    { id: '1', text: 'Mohammadu Buhari' },
    { id: '2', text: 'Atiku Abubakr' },
    { id: '3', text: 'John Doe'},
    { id: '4', text: 'Someone Else'}];

    this.options = {
      multiple: true
    };

    this.current = this.value.join(' | ');
  }

  changed(data: {value: string[]}) {
    this.current = data.value.join(' | ');
  }

}

      Candidates
      <select2
      [data] = "exampleData"
      [options] = "options"
      [width] = "500"
      [value] = "value"
      (valueChanged) = "changed($event)"
      ></select2>

Comment: Init the value `public value: string[] = []`.
Comment: i see. but after i init the model, the error goes away but i still cannot select multiple
Comment: [Check this working code here..almost same](https://stackoverflow.com/questions/51963048/multiple-select-dropdown-using-angular-with-input-tag)
Comment: I suggest providing a link to the library you are using, update the code to what it looks like right now, update the question title and add a question marked question at the end of your post :).
Answer: You must initialize your model because it's undefined when you try to access it in ngOnInit.
public value: string[] = [];

Comment: i see. but after i init the model, the error goes away but i still cannot select multiple
Comment: Change `[data] = "exampleData"` to `[data] = "candidateData"`


Proving that $\sigma(\tau_{\mathbb{R}}\times\sigma(\tau_{\mathbb{R}}))=\sigma(\tau_{\mathbb{R}})\otimes \sigma(\tau_{\mathbb{R}})$

Question: $\sigma(\tau_{\mathbb{R}})$ denotes the Borel $\sigma$-algebra ($\tau_{\mathbb{R}}$ is the usual topology on $\mathbb{R}$), $\sigma(\tau_{\mathbb{R}}\times\sigma(\tau_{\mathbb{R}}))$ is the $\sigma$-algebra generated by the product $\tau_{\mathbb{R}}\times\sigma(\tau_{\mathbb{R}})$ and $\sigma(\tau_{\mathbb{R}})\otimes \sigma(\tau_{\mathbb{R}})$ is the $\sigma$-algebra generated by sets of the form $B\times C$ where $B$ and $C$ are Borel sets. I'm asked to prove that
$$\sigma(\tau_{\mathbb{R}}\times\sigma(\tau_{\mathbb{R}}))=\sigma(\tau_{\mathbb{R}})\otimes \sigma(\tau_{\mathbb{R}}).$$
I would like a hint, if possible, for the inclusion "$\supseteq$", since the converse one is easy. I don't know how to show that a set of the form "borelian$\times$borelian" must belong to $\sigma(\tau_{\mathbb{R}}\times\sigma(\tau_{\mathbb{R}}))$.
Thank you!
Comment: Any luck with the exercise?
Comment: @Stefan Yes, but I didn't use the fact you stated, because that fact seems to me harder to prove than my question (I'm probably mistaken, sorry). What I used was that in general $\sigma(\mathcal{E}\cap B)=\sigma(\mathcal{E})\cap B$.
Answer: Suppose that $(X,\mathcal{E})$ and $(Y,\mathcal{F})$ are measurable spaces. Then use that if $\mathcal{C}$ and $\mathcal{D}$ are generators for the two sigma-algebras $\mathcal{E}$ and $\mathcal{F}$ respectively, i.e. $\sigma(\mathcal{C})=\mathcal{E}$ and $\sigma(\mathcal{D})=\mathcal{F}$ and furthermore that there exist sequences $(C_n)\subseteq\mathcal{C}$ and $(D_n)\subseteq\mathcal{D}$ such that
$$
X=\bigcup_{n=1}^\infty C_n\quad\text{and}\quad Y=\bigcup_{n=1}^\infty D_n,
$$
then
$$
\mathcal{E}\otimes\mathcal{F}=\sigma\left(\{C\times D\mid C\in\mathcal{C},\;D\in\mathcal{D}\}\right).
$$
Now, choose $\mathcal{C}$ and $\mathcal{D}$ wisely.


Overcome underfitting on train data using CNN architecture

Question: I use 2 layer CNN network for NLP task with triplet loss with margin 0.2. The task is to learn document embeddings to find similar docs.
My architecture is similar to this https://arxiv.org/abs/1406.3830
I use truncated_normal init function from TensorFlow to initialize conv weights. I also use AdamOptimizer with default params.
Then I subsample small (or big) dataset and use 5-10 epochs to train on it.
But the loss stays close to 0.2 all the time. I am defiantly underfitting. But the underfitting is not related to insufficient number of layers because same architecture works fine in literature. The problem I guess is the initialization function of convolution layers weights - it is just stuck in some local minimum. But I don't know for sure.
Where is my mistake? What is a good way to init my conv layers?
Comment: Does your network converge and overfit if you make the training set just one or a few datapoints? That is often a good sanity check.
Comment: no, it did not. The problem was my mistake, when I fixed it the model started to overfit as expected. You are right
Comment: If anyone wanders in to this question from Google and is looking for general suggestions of how to improve their triplet network, see: https://stats.stackexchange.com/questions/475655/in-training-i-first-have-a-solid-drop-in-loss-but-eventually-the-loss-slowly-b/475778#475778
Answer: The problem was my mistake. I did not compose triples properly, there was no anchor, positive and negative examples, they were all anchors or positives or negatives. My mistake. That is why neural net could not find no way to optimize. After I fixed this mistake the loss went down as expected and it started to overfit, luckily.


Can V65 or V6 move down to V7 or do they usually resolve to I?

Question: I understand that the bass of V65 or V6 wants to move up to I. My text book does have some double neighbor figures where V65 goes to V43 before resolving to I or I6. However, I am unable to find an example where V65 or V6 moves to V7 root position. Is this not a good chord progression or perhaps not a common one?
Answer: Moving between inversions of the same chord, or even different chords with the same function (for example - moving from III to V without sharpening the dominant with a 7) usually results with weak progressions. It is more common to sharpen the dominant by adding a 7th to the chord, or using different suspensions (or tensions).
Moving from a V6/5 to a V4/3 is possible, but to me sounds too redundant (at least with a direct progression), and it requires a weird leap in the bass. Most of the times inversion changes in the dominant would be V -> V6 (or V6/5),  V -> V4/2, V -> V7, or other suspensions like V4 -> V7 - I4 - I, or even V7(b13) - > V7 -> I, etc. Pay attention that all of those progressions suppose to sharpen the dominant, Make it more dissonance and carry more tension towards the solution, but the progression between V6/5 to V7 is weakening the dominant, and therefore should be avoided (unless this is what you're trying to do).
Comment: If we're going by "strengthens the dominant" as "the chord progression goes from less firm to more firm", then I thought V -> V6 and V7(b13) -> V7 both weaken the dominant. If we're going by "strengthens the dominant" as "the overall chord progression is firmer and stronger than the V chord alone", then I thought V -> V6 weakens the dominant (for starters, you no longer have a perfect authentic cadence and, therefore, that's a weak way to end a piece).
Comment: my bad about the V7(b13) -> V7, I meant the opposite. but V to v6 is extremely common.  Not as an ending progression, since, as you mentioned, it is not a perfect cadence, but it used to spice up cadences during the piece.
Comment: Just because a chord progression is used to spice up cadences does not mean that it strengthens them. As far as I can tell, V -> V6 increases variety in the bassline, but it goes from stronger to weaker (I find that V6 -> I has less finality than V -> I). I personally find that V6 -> V strengthens the dominant.
Comment: I dont know, it seems like you two are saying opposite things. So Dekka for you, going to V7 from V65 would be a good progression?
Comment: @armani - I wouldn't bat an eye at V6/5-V7, and I'm fairly sure I've actually used V6-V in my piano compositions before.
Comment: Dekka, are those compositions good?


Unit testing umbraco based sites

Question: Is there a way to test code which relies on umbraco. i.e. within my model i'm calling getNode or using uquery to get nodes but i can't test those classes because they rely on umbraco...
i guess one way to resolve this issue is by mocking, but how can you mock uquery or nodefactory?
Answer: I'm not convinced that CMS systems are naturally suited to unit testing as the output is not always as cut and dried as business logic should be.
But (there's always a but) you can mock the generation of nodes - manipulate them and check for errors at the least.  One example unit test would be to create a document type with some mandatory and some optional fields - then populate a content node with the mandatory fields only before then making an optional field mandatory - which should then error.  I don't think you can add properties as mandatory via code, though.
(Also see:
How to add a property to a document type in Umbraco from code?
http://our.umbraco.org/wiki/reference/api-cheatsheet/creating-a-document
http://our.umbraco.org/wiki/how-tos/working-with-document-types
)
Something like:
<code>var alias = "aliasOfOptionalField";
var value = string.Empty;
var docType = "Textpage";

DocumentType dt = DocumentType.GetByAlias(docType);
User author = User.GetUser(0);
Document doc = Document.MakeNew("My new document", dt, author, 1018);
doc.Publish(author);
umbraco.library.UpdateDocumentCache(doc.Id);

var newProperty = dt.AddPropertyType(new DataTypeDefinition(-49), alias, "test prop");

if (doc.HasProperty(alias))
{
   doc.getProperty(alias).Value = value;
   doc.Publish(new User(0));
   library.UpdateDocumentCache(doc.Id);
}
</code>

Unit testing umbraco based sites


Matrix representation of a co-domain restriction of a linear operator

Question: Consider the finite-dimensional linear operator:
$\mathcal{A}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{3},$ with $Ax=y,$
$A=\left[\begin{array}{ccc}
1 & 0 & 1\\
1 & -2 & -1\\
0 & 1 & 1
\end{array}\right].$
Let $\mathcal{A}_{1}:\mathbb{R}^{3}\rightarrow\mathcal{R}\{A\}$ be
the co-domain restriction of $\mathcal{A}$ to $\mathcal{R}\{A\}$.
Give the matrix representation $A_{1}$ of $\mathcal{A}_{1}.$
I have gotten $\mathcal{R}\{A\}$, which is 2-dimensional. So I know the matrix representation of the restricted operator will be a 2x3 matrix, which maps $\mathbb{R}^{3}$ to the plane parametrized by the 2 vectors which span the range. I am unsure of how to obtain the restricted operator. Any advice would be greatly appreciated. Thank you!
Answer: Note that the matrix representation $A_1$ of ${\cal A}_1$ will depend on which basis you take. Let's assume you take the standard basis on the left, and on the right the basis constisting of the two vectors you've just found, say $v_1$ and $v_2$.
Now express ${\cal A}(e_1)$ (i.e., the first column of the matrix $A$) as a linear combination of $v_1$ and $v_2$. Those two coefficients will be the first column of the matrix $A_1$.
Do the same for ${\cal A}(e_2)$ (the second column of $A$), putting the result in the second column of $A_1$; and for ${\cal A}(e_3)$ (the third column of $A$), putting the result in the third column of $A_1$.
Note that this is going to be easiest if $v_1$ and $v_2$ are two columns of the original matrix $A$ to start with, because then two of your new columns are just going to be $(1,0)$ and $(0,1)$.
Comment: So the columns of A1 will be the representations of the columns of A w.r.t. the basis vectors of R{A}?
Answer: The matrix representation you are asking is not unique, it will depend on the basis of $\mathcal{R}\{A\}$ you are considering. To find a basis for $\mathcal{R}\{A\}$, find $Ae_1$ and $Ae_2$: you get that $$Ae_1=\left[\begin{array}{ccc}
1 & 0 & 1\\
1 & -2 & -1\\
0 & 1 & 1
\end{array}\right]\left[\begin{array}{c}
1 \\0\\0
\end{array}\right]=\left[\begin{array}{c}
1 \\1\\0
\end{array}\right],\,\,\,\,\,Ae_2=\left[\begin{array}{ccc}
1 & 0 & 1\\
1 & -2 & -1\\
0 & 1 & 1
\end{array}\right]\left[\begin{array}{c}
0 \\1\\0
\end{array}\right]=\left[\begin{array}{c}
0 \\-2\\1
\end{array}\right].$$ Those are independent and $\mathcal{R}\{A\}$ is two dimensional, so they span it. Now, for the general $z=(a,b,c)$: $$Az=\left[\begin{array}{ccc}
1 & 0 & 1\\
1 & -2 & -1\\
0 & 1 & 1
\end{array}\right]\left[\begin{array}{c}
a \\b\\c
\end{array}\right]=\left[\begin{array}{c}
a+c \\a-2b-c\\b+c
\end{array}\right]=\left[\begin{array}{c}
a+c \\a+c\\0
\end{array}\right]+\left[\begin{array}{c}
0 \\-2b-2c\\b+c
\end{array}\right]=$$$$(a+c)Ae_1+(b+c)Ae_2.$$ So, a matrix representation (with respect to the usual basis of $\mathbb R^3$ and the basis $Ae_1, Ae_2$) is $$\left[\begin{array}{ccc}
1 & 0 & 1\\
0 & 1 & 1
\end{array}\right].$$
Comment: How did you choose Ae1 and Ae2 for the basis vectors of R{A}? It turns out that those do correspond to the two linearly independent columns of A, but your answer seems to suggest that Ae3 would work as well. Also, I still don't see how the third to last line leads to that particular A1. Is it that the rows of A1, when multiplied by the general z give (a+c) and (b+c) respectively?
Comment: You can choose anything that will end up giving you a basis for the image. Here, $Ae_1$ and $Ae_2$ worked. If $Ae_1$ and $Ae_3$ are independent, then $Ae_1$ and $Ae_3$ will work as well. Finally, you find the last matrix by seeing what $Az$ as a linear combination of $Ae_1$ and $Ae_2$: if you multiply the last matrix by $z=(a,b,c)$, you get $(a+c)Ae_1+(b+x)Ae_2$.
Comment: Thank you! However, if you multiply the last matrix by z wouldn't you get a 2x1 vector with components (a+c) and (b+c)?
Comment: Yes, but this is the matrix with respect to the basis $Ae_1, Ae_2$ of the image, so you end up with $(a+c)Ae_1+(b+c)Ae_2$.
Comment: Aha! That makes sense. Thank you.


Prove the countability of $\mathbb Q \times \mathbb Q$ and $M_{2 \times 2}(\mathbb Z)$

Question: Using the fact that $\mathbb N \times \mathbb N$ is countable, or otherwise, prove that the following sets are countable.
a) the set of all points in the $(x,y)$ plane with rational coordinates
b) the set of all $2\times 2$ matrices with integer entries.
I really don't know where to start... and I've never done matrices either!
I know that I need to prove there exists a bijection, but from there I'm lost.
Comment: If it helps, you can think of $2\times 2$ matrices as vectors in $\Bbb R^4$ as there is an easy bijection between them.  $\begin{bmatrix}a&b\\c&d\end{bmatrix}\leftrightarrow (a,b,c,d)$.  The only thing special about matrices is how they add and multiply, but that has nothing to do with the current problem.
Comment: For (a), can you use the fact that the rationals are countable?
Answer: a) There's two parts to this question. 1. Show there exists a bijection between $\mathbb{N}$ and $\mathbb{Q}$ (a famous result that hopefully you're given). 2. If $f \colon A \to B$ is a bijection and $g \colon C \to D$ is a bijection, then $f \times g \colon A \times C \to B \times D$ defined by $(f \times g)(a, c)=(f(a), g(c))$ is a bijection.
b) For your purposes, you're essentially trying to show $(\mathbb{Z}^2)^2$ is countable. Knowing that $\mathbb{Z}$ is countable, you can apply part 2. to $\mathbb{Z}^2$ to show that $\mathbb{Z}^2$ is countable, and then to show that $(\mathbb{Z}^2)^2$ is countable...well...apply part 2. again.
Fun fact: The insight you get from this problem hopefully shows you most generally that $\prod^n_{i=1} C_i$ for any finite collection $C_1, \dots, C_n$ of countable sets is countable.
Answer: For the purpose of just this problem, your $2\times 2$ matrices are just ordered tupls of $4$ integers.  That is, they are isomorphic to elements of $\Bbb{N}^4$,
which can be written as $\Bbb{N}^2\times \Bbb{N}^2$. Each of those has the required bijection, and now you are down to finding a bijection of $\Bbb{N}\times \Bbb{N}$ to $\Bbb{N}$.
Think about composiing those maps with each other.
For the rationals, it is easy to show that you can have a surjection of the integers to the rationals (choose denominators of 1, for example.  The surjection of the rationals to the integers can be made by pretending the numrator and denominator are each independent members of $\Bbb{N}$ so the fraction can be surjectively related to $\Bbb{N}^2$ which in turn is surjective to $\Bbb{N}$.
You need to flesh out these ideas a bit.
Comment: for part a) do i need to show QxQ is countable? I could probably do that, I am just a bit stuck with what it is asking me to do?


Analyzing slow mysql querries

Question: Our Drupal site has recently started to get much more traffic and often pegs the CPU and memory when we have 50-100 active connections. We have 8cpus and 16GB of RAM. I notice that when the cpu is pegged mysql is by far the largest consumer so I decided to look at the mysql-slow.log and found that some queries take over 1 second and I am wondering if this is reasonable. Here is one example:
# Time: 150330 12:01:34
# User@Host: root[root] @ localhost []
# Query_time: 1.131631  Lock_time: 0.000301 Rows_sent: 4  Rows_examined: 151102
SET timestamp=1427738494;
SELECT
  node.nid AS nid,
  node.title AS node_title,
  users_node.name AS users_node_name,
  users_node.uid AS users_node_uid,
  votingapi_cache_node_points_projects_sum.value AS votingapi_cache_node_points_projects_sum_value,
  node.created AS node_created,
  node.changed AS node_changed,
  node_revision.timestamp AS node_revision_timestamp,
  'node' AS field_data_field_screenshot_node_entity_type
FROM node AS node
INNER JOIN users AS users_node
  ON node.uid = users_node.uid
LEFT JOIN votingapi_cache AS votingapi_cache_node_points_projects_sum
  ON node.nid = votingapi_cache_node_points_projects_sum.entity_id
  AND votingapi_cache_node_points_projects_sum.entity_type = 'node'
  AND votingapi_cache_node_points_projects_sum.value_type = 'points'
  AND votingapi_cache_node_points_projects_sum.tag = 'projects'
  AND votingapi_cache_node_points_projects_sum.function = 'sum'
LEFT JOIN votingapi_vote AS votingapi_vote_node_points_vote
  ON node.nid = votingapi_vote_node_points_vote.entity_id
  AND votingapi_vote_node_points_vote.entity_type = 'node'
  AND votingapi_vote_node_points_vote.value_type = 'points'
  AND votingapi_vote_node_points_vote.tag = 'vote'
LEFT JOIN node_revision AS node_revision
  ON node.vid = node_revision.vid
WHERE node.status = '1'
  AND node.type IN ('arcade_project_container')
  AND EXISTS (
    SELECT na.nid AS nid
    FROM node_access AS na
    WHERE na.gid = '0'
    AND na.realm = 'all'
    AND na.grant_view >= '1'
    AND node.nid = na.nid
  )
ORDER BY
  node_created DESC,
  node_changed DESC,
  node_revision_timestamp DESC
LIMIT 4 OFFSET 5800;

This query comes from the front page where I have a view which shows the most recent submissions. Should this query really take 1.13 seconds?
EDIT: Here is a screenshot of the results of running EXPLAIN on this query on phpmyadmin:

Here is the EXPLAIN after switch nid and uid as suggested by mikeytown2:
Comment: What does EXPLAIN say? Add EXPLAIN before the SELECT to get the output.
Comment: @Mike2012  you can copy/paste the query in to something like phpmyadmin and add the "explain"
Comment: @mikeytown2 I wonder, how could I do that? This query is generated by Drupal core and views. Or am i missing something?
Comment: @user5482 Ok, great thanks for the tip! I uploaded a screenshot of the result of EXPLAIN from phpmyadmin.
Answer: Add a new index of type index on the node table with these 7 columns; I would label it "status_type_uid_nid_vid_created_changed"
status, type, uid, nid, vid, created, changed

Once you have done that run explain again; odds are the index will need to be re-ordered or another table will need to have an index added to it.
Try #2. Flip nid & uid.
status_type_nid_uid_vid_created_changed
status, type, nid, uid, vid, created, changed

Comment: Thanks for taking the time to look at this. I have added the index you suggested and posted the new result from explain. It doesn't seem that different.
Comment: @Mike2012 Try flipping nid & uid in the order
Comment: ok, I flipped nid and uid like you suggested and uploaded the new EXPLAIN.
Comment: What version of MySQL are you using?
Comment: I am using version: 14.14 Distrib 5.5.41, for debian-linux-gnu
Comment: You might have better luck with MySQL 5.6 or MariaDB 10.0. The [query optimizer](http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html#optimizer) is not picking the correct index. You could add a FORCE INDEX to the node table to pick the index we created but that requires a query alter to pull off.

Analyzing slow mysql querries


how can I match column fields and group their values together?

Question: I am sorting some files that I've created using pdfgrep, to list page numbers of certain PDFs that I have. it produced the following output:
./Buddhism in the Shadow of Brahmanism.pdf:111:      Then, rising from his seat, covering one shoulder with his robe, the king
./Buddhism in the Shadow of Brahmanism.pdf:182:branch who has adopted the yellow robes of Buddhism; he is sur-
./Buddhism in the Shadow of Brahmanism.pdf:229:       resolve that his body, his bowl, and his monastic robe (which had been
./Buddhism in the Shadow of Brahmanism.pdf:230:robe. In this way, Mahākāśyapa (or at least his body) is to act as a sort
./Buddhism in the Shadow of Brahmanism.pdf:230:corpse to his disciples and displays to them the Buddha's robe, and they
./Buddhism in the Shadow of Brahmanism.pdf:230:offer him the robe that the Buddha had confided to him. Only then will
./Introduction to the History of Indian Buddhism.pdf:31:the robes of a Buddhist monk in an effort to convert them, he was Sciequia. For
./Introduction to the History of Indian Buddhism.pdf:54:monks, and in particular on retreat, robes, and chastity, p. 308.—On the life of
./Introduction to the History of Indian Buddhism.pdf:97:are the Kat.hināvadāna, which deals with the bowl, the staff, and the robes of
./Introduction to the History of Indian Buddhism.pdf:111:of a sort of robe.
./Introduction to the History of Indian Buddhism.pdf:112:cover his nakedness, and who rejects all other robes as superfluous.
./Introduction to the History of Indian Buddhism.pdf:127:noon, after having taken his robe and his bowl,
./Introduction to the History of Indian Buddhism.pdf:127:bowl and his robe, he went to the place where the Cāpāla caitya6 was located,

What I'm trying to do is group together the page numbers on the second column that match the file name, I was expecting the output to look something like:
./Buddhism in the Shadow of Brahmanism.pdf:111, 182, 229, 230
./Introduction to the History of Indian Buddhism.pdf:31, 54, 97, 111, 112, 127

I've tried using awk to parse the first value and then use those results on the same file to print only the page numbers so I could grep the result and append after the file name later, like so:
awk -F : '{print $1}' parsing_file | uniq | while read line; do awk -v number="$line" -F : '$1 == "$number" { print $2 }' parsing_file; done

But that didn't go through and I am guessing uniq and while read could be cut out, and maybe use some array with awk only?
I've seen something similar done here:
https://unix.stackexchange.com/questions/167280/awk-group-by-and-sum-column-values
But instead of summing up the values on the columns, I just wanted to group them together.
Thanks
Comment: You're welcome. That part at least is easy - ask a question, give it a few hours or a day to see what answers you get (providing feedback/questions/comments to answers as appropriate you go) and **then** accept the one you're going to use.
Comment: Thanks for pointing that out, Ed. Indeed, you're right, I didn't pay much attention for the full output of that answer, and I'm glad you answered it with one other possibility even if I had already marked it as solved. I still have to better grasp the mechanics of Forums usage. Thanks for your patience and time!
Comment: The first answer you got may be the best possible answer or it may not be. By accepting it right away you discourage other people from posting answers so you may never find out a better way to do what you want.
Answer: With your shown samples, please try following. Written and tested in GNU awk.
awk  -v OFS=":" '
match($0,/^\.\/.*\.pdf:[0-9]+/){
  value=substr($0,RSTART,RLENGTH)
  split(value,arr,":")
  if(!seen[arr[1],arr[2]]++){
    name[arr[1]]=(name[arr[1]]?name[arr[1]]", ":"")arr[2]
  }
}
END{
  for(key in name){
    print key,name[key]
  }
}
'  Input_file

Output will be as follows with your shown samples:
./Buddhism in the Shadow of Brahmanism.pdf:111, 182, 229, 230
./Introduction to the History of Indian Buddhism.pdf:31, 54, 97, 111, 112, 127

Explanation: Adding detailed explanation for above.
awk  -v OFS=":" '                   ##Starting awk program from here.
match($0,/^\.\/.*\.pdf:[0-9]+/){    ##Using match function to match from starting ./ till .pdf : digits as per shown samples.
  value=substr($0,RSTART,RLENGTH)   ##Creating value with matched sub string here.
  split(value,arr,":")              ##Splitting value into array arr with : delimiter.
  if(!seen[arr[1],arr[2]]++){
     name[arr[1]]=(name[arr[1]]?name[arr[1]]", ":"")arr[2]  ##Creating name array with index of book name and its value it digits as per needed output.
  }
}
END{                                ##Starting END block of this program from here.
  for(key in name){                 ##Traversing through name here.
    print key,name[key]             ##Printing key and array value here.
  }
}
' Input_file                        ##Mentioning Input_file name here.

NOTE: previously above solution was not handling duplicate numbers coming for same paragraph so I had edited solution to take care of that case after Ed's answer.
Comment: I wasn't in any way suggesting you change SUBSEP in this case (but there's nothing wrong with setting it to the same value as OFS for example when convenient for storing and printing compound indices - `arr[$1,$2,$3]=7;... for (i in arr) print i, arr[i]`). What I was saying in that point is that you already have the string stored in `value` that contains the same string as `arr[1] ":" arr[2]` and so you could be using `value` as the index rather than creating a new almost identical compound index from `arr[1] SUBSEP arr[2]`.
Comment: If the OPs real data has `:`s in the title then your `split(value,arr,":")` will fail in exactly the same way as setting FS to `:` would on a `:` in the title (or, far less likely, page number). If the OPs real data has titles that don't match your regexp otherwise (e.g. they aren't PDFs) then its most likely they'll want them handled exactly the same way as in the sample input. If they have lines of input that look nothing like that then they'd have to provide that info in the question for us to know how to handle them.
Comment: @EdMorton, Sure sir points taken, thank you for your nice suggestions as always, I will try to learn from them, cheers, be safe, stay healthy sir.
Comment: You're welcome. The main issue though is something you can't solve without turning your answer into mine and that is that storing all of the data in memory and then printing it in a loop in the END section is always a worse approach to these types of problem in terms of memory usage and efficiency than sorting the input on the key values just printing the data every time a key value changes.
Comment: Fair enough - if you can sort your input in advance for printing one key at a time then you should do that but if you can't sort it for some reason then you have to store at least all of the output in memory for outputing in the END.
Comment: I see you've fixed the issue of printing duplicate page numbers and not printing a `:` between  the title and the page numbers (but fix the output you show under "Output will be..." to include that `:`), good.
Comment: @EdMorton, Hi sir, I had added it, please do let me know if you see any issues, thank you.
Comment: I do see a few... I'll provide comments but then please wait til I'm done with all of them before updating  your code as the final comment will affect how you address earlier ones.
Comment: @EdMorton, yeah I have tried to fix them after reading your post. Sure, please do post here, I will try to fix if I could.
Comment: These are separate from anything listed in my post I think.
Comment: 1) You use `seen[arr[1],arr[2]]` but `arr[1],arr[2]` is just a slightly less robust version of `value` (`:` changed to SUBSEP) so just use `seen[value]`.
Comment: 2) You're isolating `value` using `match()` and then spliting `value` into `arr[]` on every `:` so you can use `arr[1]` and `arr[2]` to hold the title and page strings. It'd be much simpler to not do any of that and just set `FS` to `:` and then use `$1` and `$2` instead of `arr[1]` and `arr[2]`.
Comment: Actually I think those are really the only 2 points not alread mentioned in my answer.
Comment: @EdMorton, For 1st point(I saw experts(don't remember if its you or someone else here) not to change SUBSEP so I avoided, could update it in answer). 2nd point- I also thought to set field separators first but then I thought to use regex rather than field separator(in case OP's real data is having some other stuff too but that's an assumption only), thank you sir.
Comment: @EdMorton, yeah but if data is not sorted then it could be helpful(in case someone not using sort/uniq etc), cheers and have a Good day. NOTE: I have also added a note, that changes were made after seeing your post for duplicate digits removing, I missed it :(
Answer: $ cat tst.awk
BEGIN { FS=OFS=":" }
$1 != title {
    if ( title != "" ) {
        print title, pages
    }
    title = $1
    pages = $2
    delete seen
    next
}
!seen[$2]++ {
    pages = pages ", " $2
}
END {
    print title, pages
}

$ awk -f tst.awk file
./Buddhism in the Shadow of Brahmanism.pdf:111, 182, 229, 230
./Introduction to the History of Indian Buddhism.pdf:31, 54, 97, 111, 112, 127

The differences between the above and @Ravinder's answer as I write this (which is after that answer was accepted) are:

This relies on your input being sorted by title and page number as shown in the example in your question while Ravinders doesn't. If your real data isn't sorted then run sort -t':' -k1,1 -k2,2n file | awk ...'.
This doesn't read all of the titles and page numbers into memory, it just reads the page numbers for 1 title at a time, so it'll work no matter how massive your input file is.
This will output the titles in the same order they appear in the input instead of a random order.
This produces the expected output by only outputting unique page numbers per title rather than the same
page number being output every time it appears in the input (e.g.
111, 182, 229, 230 vs 111, 182, 229, 230, 230, 230.
This produces the expected output by puting a : between the title and the first page number instead of a blank.


Generalization of any product similar to the Gamma function

Question: So, the Gamma function generalizes the following product:
$$n!=1\cdot 2\cdot 3\cdot..........n$$
Because it is only defined when $n$ is a natural number.
But I thought about the following product:
$$n{$}=1\cdot (1+d)\cdot (1+2d) \cdot..................n$$
Clearly this product is also defined only when $n$ is of the form $1+kd$ where $k$ is a natural number. Can this product also be generalized when $n$ is not of that form?
For example, the product:
$$f(n)=1\cdot 1.3\cdot 1.6\cdot 1.9\cdot 2.2......n$$
Clearly, this product is only defined when $n$ is of the form $1+0.3k$. Can this product be generalized?
Comment: is $d$ a rational number?
Comment: @mrnovice Does it matter? When $d=1$ we get our usual factorial function whose generalization is the gamma function.
Comment: https://en.wikipedia.org/wiki/Pochhammer_k-symbol
Comment: @Chappers Thanks. I think that'll do.
Answer: As Chappers commented$$P_n=\prod_{i=0}^n(1+id)=d^n \left(1+\frac{1}{d}\right)_n$$ where appears Pochhammer symbol.
This can also be expressed in terms of the gamma function $$P_n=d^n\frac{ \Gamma \left(n+1+\frac{1}{d}\right)}{\Gamma
   \left(1+\frac{1}{d}\right)}$$


remove extra letters and numbers from parenthesis in a string

Question: I have a string like:
(aeiou 123) word one

How can I remove everything from the parenthesis so that only "word one" remains?
Comment: post your attempts..
Comment: @Raj very little information provided, scope not defined
Comment: @Raj : Please provide the sample code you have attempted. This looks like a very easy problem however.
Answer: You could use a regex, e.g.:
String str = "(aeiou 123) word one";
str = str.replaceAll("\\([^\\)]*\\)", "").trim();

Answer: public class ParanthesisRemoval {

public static void main(String args[])
{
    String test="ab(xy)cd(zw)ef";

    boolean modified = true;
    while(modified)
    {
        modified = false;
        int indexOpenParanthesis = test.indexOf("(");
        int indexClosedParanthesis = test.indexOf(")");
        if(indexOpenParanthesis!=-1 && indexClosedParanthesis!=-1 && indexOpenParanthesis<indexClosedParanthesis)
        {
            int stringLength = test.length();
            test = test.substring(0, indexOpenParanthesis)+test.substring(indexClosedParanthesis+1, stringLength);
            modified=true;
        }

    }

    System.out.println(test);
}

}
note that this will not work for nested parantesis - (()), or paranthesis not properly  paired  (()