Unable to read serial data without errors/bugs

Question: I am writing a small application based on tkinter in order to read serial data from my arduino.
The arduino, when it receives a serial text (rf), it will begin sending data to the pc.
Below is the suspicious code:
def readSerial():
    ser_bytes = ser.readline()
    ser_bytes = ser_bytes.decode("utf-8")
    text.insert("end", ser_bytes)
    after_id=root.after(100,readSerial)
    #root.after(100,readSerial)

def measure_all():
   global stop_
   stop_ = False
   ser.write("rf".encode()) #Send string 'rf to arduino', which means Measure all Sensors
   readSerial() #Start Reading data

Now this does not work. The program freezes, and no info is revealed on the terminal.
When i change the line after_id=root.after(100,readSerial) to root.after(100,readSerial) then the program works, but only when i receive serial input.
For example, if there is a 5 second delay to when arduino sends serial, then the program will freeze, until it receives the data. More specificallly, if the program is minimized, and i select to view it as normal, it will not respond unless it receives input from arduino (which will display normally).
So even now, it still does not work properly.
But also have in mind, that i need to have the after_id line, so that i can have a handle, so that i can terminate the readSerial() function (for example when the user presses the 'stop measurement' button).
Can someone understand what is going on, and how i can have the after_id behaviour (so i can stop the continuous function later), while having the program behaving normal, without crashing or stuck until it receives data?
EDIT: This is the modified code after user's acw1668 suggestions. This does not work. I see nothing on the text frame of tkinter.
import tkinter as tk
import tkinter.ttk as ttk
import serial.tools.list_ports #for a list of all the COM ports
from tkinter import scrolledtext
import threading
import time
from queue import SimpleQueue

#to be used on our canvas
HEIGHT = 800
WIDTH = 800

#hardcoded baud rate
baudRate = 9600

# this is the global variable that will hold the serial object value
ser = None #initial  value. will change at 'on_select()'

after_id = None

#this is the global variable that will hold the value from the dropdown for the sensor select
dropdown_value = None

# create the queue for holding serial data
queue = SimpleQueue()
# flag use to start/stop thread tasks
stop_flag = None

# --- functions ---

#the following two functtions are for the seria port selection, on frame 1

#this function populates the combobox on frame1, with all the serial ports of the system
def serial_ports():
    return serial.tools.list_ports.comports()

#when the user selects one serial port from the combobox, this function will execute
def on_select(event=None):
    global ser
    COMPort = cb.get()
    string_separator = "-"
    COMPort = COMPort.split(string_separator, 1)[0] #remove everything after '-' character
    COMPort = COMPort[:-1] #remove last character of the string (which is a space)
    ser = serial.Serial(port = COMPort, baudrate=9600, timeout=0.1)
    #readSerial() #start reading shit. DELETE. later to be placed in a button
    # get selection from event
    #print("event.widget:", event.widget.get())
    # or get selection directly from combobox
    #print("comboboxes: ", cb.get())

    #ser = Serial(serialPort , baudRate, timeout=0, writeTimeout=0) #ensure non-blocking

def readSerial(queue):
    global stop_flag
    if stop_flag:
        print("Reading task is already running")
    else:
        print("started")
        stop_flag = threading.Event()
        while not stop_flag.is_set():
            if ser.in_waiting:
                try:
                  ser_bytes = ser.readline()
                  data = ser_bytes.decode("utf-8")
                  queue.put(data)
                except UnicodeExceptionError:
                  print("Unicode Error")
            else:
                time.sleep(0.1)
        print("stopped")
        stop_flag = None

# function to monitor whether data is in the queue
# if there is data, get it and insert into the text box
def data_monitor(queue):
    if not queue.empty():
       text.insert("end", f"{queue.get()}\n")
       if vsb.get()[1]==1.0: #if the scrollbar is down to the bottom, then autoscroll
          text.see("end")
    root.after(100, data_monitor, queue)

# this function is triggered, when a value is selected from the dropdown
def dropdown_selection(*args):
   global dropdown_value
   dropdown_value = clicked.get()
   button_single['state'] = 'normal' #when a selection from the dropdown happens, change the state of the 'Measure This Sensor' button to normal

# this function is triggered, when button 'Measure all Sensors' is pressed, on frame 2
def measure_all():
   button_stop['state']='normal' #make the 'Stop Measurement' button accessible
   ser.write("rf".encode()) #Send string 'rf to arduino', which means Measure all Sensors
   sleep(0.05) # 50 milliseconds
   threading.Thread(target=readSerial, args=(queue,)).start()

# this function is triggered, when button 'Measure this Sensor' is pressed, on frame 2
def measure_single():
   global stop_
   stop_=False
   button_stop['state']='normal'
   ser.write(dropdown_value.encode()) #Send string 'rf to arduino', which means Measure all Sensors!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   readSerial()

# this function is triggered, when button 'STOP measurement(s)' is pressed, on frame 2
def stop_measurement():
    button_stop['state']='disabled'
    ser.write("c".encode())
    if stop_flag:
        stop_flag.set()
    else:
        print("Reading task is not running")
# --- functions ---

# --- main ---
root = tk.Tk() #here we create our tkinter window
root.title("Sensor Interface")

#we use canvas as a placeholder, to get our initial screen size (we have defined HEIGHT and WIDTH)
canvas = tk.Canvas(root, height=HEIGHT, width=WIDTH)
canvas.pack()

#we use frames to organize all the widgets in the screen

'''
relheight, relwidth − Height and width as a float between 0.0 and 1.0, as a fraction of the height and width of the parent widget.
relx, rely − Horizontal and vertical offset as a float between 0.0 and 1.0, as a fraction of the height and width of the parent widget.
'''

# --- frame 1 ---
frame1 = tk.Frame(root)
frame1.place(relx=0, rely=0.05, relheight=0.03, relwidth=1, anchor='nw') #we use relheight and relwidth to fill whatever the parent is - in this case- root

label0 = tk.Label(frame1, text="Select the COM port that the device is plugged in: ")
label0.config(font=("TkDefaultFont", 8))
label0.place(relx = 0.1, rely=0.3, relwidth=0.3, relheight=0.5)

cb = ttk.Combobox(frame1, values=serial_ports())
cb.place(relx=0.5, rely=0.5, anchor='center')
# assign function to combobox, that will run when an item is selected from the dropdown
cb.bind('<<ComboboxSelected>>', on_select)
# --- frame 1 ---

# --- frame 2 ---
frame2 = tk.Frame(root, bd=5) #REMOVED THIS bg='#80c1ff' (i used it to see the borders of the frame)
frame2.place(relx=0, rely=0.1, relheight=0.07, relwidth=1, anchor='nw')

#Button for 'Measure All Sensors'
#it will be enabled initially
button_all = tk.Button(frame2, text="Measure all Sensors", bg='#80c1ff', fg='red', state='normal', command=measure_all)  #bg='gray'
button_all.place(relx=0.2, rely=0.5, anchor='center')

#label
label1 = tk.Label(frame2, text="OR, select a single sensor to measure: ")
label1.config(font=("TkDefaultFont", 9))
label1.place(relx = 0.32, rely=0.3, relwidth=0.3, relheight=0.4)

#dropdown
#OPTIONS = [0,1,2,3,4,5,6,7]
OPTIONS = list(range(8)) #[0,1,2,3,4,5,6,7]
clicked = tk.StringVar(master=frame2) # Always pass the `master` keyword argument, in order to run the function when we select from the dropdown
clicked.set(OPTIONS[0]) # default value
clicked.trace("w", dropdown_selection) #When a value from the dropdown is selected, function dropdown_selection() is executed
drop = tk.OptionMenu(frame2, clicked, *OPTIONS)
drop.place(relx = 0.65, rely=0.25, relwidth=0.08, relheight=0.6)

#Button for 'Measure Single Sensor'
#this will be disabled initially, and will be enabled when an item from the dropdown is selected
button_single = tk.Button(frame2, text="Measure this Sensor", bg='#80c1ff', fg='red', state='disabled', command=measure_single) #bg='gray'
button_single.place(relx = 0.85, rely=0.5, anchor='center')
# --- frame 2 ---

# --- frame 3 ---
frame3 = tk.Frame(root, bd=5) #REMOVED THIS bg='#80c1ff' (i used it to see the borders of the frame)
frame3.place(relx=0, rely=0.2, relheight=0.07, relwidth=1, anchor='nw')

#Button for 'STOP Measurement(s)'
#this will be disabled initially, and will be enabled only when a measurement is ongoing
button_stop = tk.Button(frame3, text="STOP measurement(s)", bg='#80c1ff', fg='red', state='disabled', command=stop_measurement)
button_stop.place(relx=0.5, rely=0.5, anchor='center')
# --- frame 3 ---

# --- frame 4 ---
frame4 = tk.Frame(root, bd=5)
frame4.place(relx=0, rely=0.3, relheight=0.09, relwidth=1, anchor='nw')

label2 = tk.Label(frame4, text="Select a sensor to plot data: ")
label2.place(relx = 0.1, rely=0.3, relwidth=0.3, relheight=0.5)

clickedForPlotting = tk.StringVar()
clickedForPlotting.set(OPTIONS[0]) # default value
dropPlot = tk.OptionMenu(frame4, clickedForPlotting, *OPTIONS)
dropPlot.place(relx=0.5, rely=0.5, anchor='center')

#CHANGE LATER
#dropDownButton = tk.Button(frame4, text="Plot sensor data", bg='#80c1ff', fg='red', command=single_Sensor) #bg='gray'
#dropDownButton.place(relx = 0.85, rely=0.5, anchor='center')
# --- frame 4 ---

#frame 5 will be the save to txt file

#frame 6 will be the area with the text field
# --- frame 6 ---
frame6 = tk.Frame(root, bg='#80c1ff') #remove color later
frame6.place(relx=0.0, rely=0.4, relheight=1, relwidth=1, anchor='nw')

text_frame=tk.Frame(frame6)
text_frame.place(relx=0, rely=0, relheight=0.6, relwidth=1, anchor='nw')
text=tk.Text(text_frame)
text.place(relx=0, rely=0, relheight=1, relwidth=1, anchor='nw')
vsb=tk.Scrollbar(text_frame)
vsb.pack(side='right',fill='y')
text.config(yscrollcommand=vsb.set)
vsb.config(command=text.yview)
# --- frame 6 ---

# start data monitor task
data_monitor(queue)

root.mainloop() #here we run our app
# --- main ---

Comment: When timeout on `ser.readline()`, it just returns an empty byte string.  Have you ever tried the timeout or using `.in_waiting`?
Comment: You need to specify a timeout for the serial read, so the function doesn't hang if there's nothing to receive.  Note that `after_id` is useless the way you have it: it's a local variable, so it's not accessible anywhere else in the program.
Comment: @jasonharper Thank you so much for your comment! I am researching this problem for about a week now.  1. You mean i should set up `global after_id` before my line so it's accessable globally? 2. How do i specify a timeout for the serial read? If my arduino spits data in an interval of five seconds, will the timeout still work, or it will stop the execution of the serial read, and then i will not be able to read the next interval of data coming in?
Comment: You can set `timeout` option when creating instance of `Serial(...)` or use `.in_waiting()` to check whether there is data in the read buffer before reading.
Comment: @acw1668 Thank you. But there is one thing i do not get. My arduino sends data in an interval of 5 seconds. If i set the timeout, then does that mean i can only get one interval of these reading (because the timeout will terminate the connection)?
Comment: The timeout does not terminate the connection, only the current reading action.
Comment: @acw1668 So if the timeout gets triggered, but five seconds later another reading is received, will it get parsed and processed normally?
Comment: You call `readSerial()` every 100ms periodically, so basically yes.
Comment: @acw1668 I am a bit confused... So i just basically set the timeout? Or there is extra logic that i have to implement? When the program is set to the 'receiving data' logic (through the press of a button in tkinter), do i have to include extra logic to restart the function when (and if) the timeout occurs?
Comment: @acw1668 Thank you for trying to help me but i confused and stuck. I am opening a bounty.
Comment: @user1584421 You can go through the docs [here](https://pyserial.readthedocs.io/en/latest/pyserial_api.html), `timeout` has been explained well.
Comment: @AST But how can i also incorporate the `id` functionality so i can terminate it with a stop button? Is the behavior described in the question normal?
Comment: @user1584421 The behaviour described is pretty normal as `ser.readline()` is blocking, and any blocking function when used in `tkinter` will freeze the mainloop. Setting the `timeout` parameter makes it non-blocking (as also described in the docs). You can use the "`id` functionality" normally then after.
Comment: @AST i tried a `timeout=0.1` and it worked great the first time, but it never worked again. Mostly the software just got stuck, no info on the terminal. However, two times i got errors on the terminal. The first time it said `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte` on this line: `ser_bytes = ser_bytes.decode("utf-8")`. The second time i got `AttributeError: 'NoneType' object has no attribute 'write'` on this line: `ser.write("rf".encode())`. Do you have any clues on these? Thank you.
Comment: Did you try to use thread instead of `.after()`?
Comment: @acw1668 no.. I haven't received this recommendation before, this is the first time i am hearing this. How exactly do i use this? Do you think this is the source of my new problems? It's weird that the program worked just fine once but never again....
Answer: In order to not blocking the main tkinter application, it is recommended to use thread to run the serial reading.  Also use queue.SimpleQueue to transfer the serial data to main task so that the serial data can be inserted into the Text widget.
Below is an example:
import threading
import time
from queue import SimpleQueue
import tkinter as tk
import serial

class SerialReader(threading.Thread):
    def __init__(self, ser, queue, *args, **kw):
        super().__init__(*args, **kw)
        self.ser = ser
        self.queue = queue
        self._stop_flag = threading.Event()

    def run(self):
        print("started")
        while not self._stop_flag.is_set():
            if self.ser.in_waiting:
                ser_bytes = self.ser.readline()
                data = ser_bytes.decode("utf-8")
                self.queue.put(data)
            else:
                time.sleep(0.1)
        print("stopped")

    def terminate(self):
        self._stop_flag.set()

# create the serial instance
ser = serial.Serial(port="COM1") # provide other parameters as well
# create the queue for holding serial data
queue = SimpleQueue()
# the serial reader task
reader = None

def start_reader():
    global reader
    if reader is None:
        # create the serial reader task
        reader = SerialReader(ser, queue, daemon=True)
    if not reader.is_alive():
        # start the serial reader task
        reader.start()
    else:
        print("Reader is already running")

def stop_reader():
    global reader
    if reader and reader.is_alive():
        # stop the serial reader task
        reader.terminate()
        reader = None
    else:
        print("Reader is not running")

# function to monitor whether data is in the queue
# if there is data, get it and insert into the text box
def data_monitor(queue):
    if not queue.empty():
        text.insert("end", f"{queue.get()}\n")
    root.after(100, data_monitor, queue)

root = tk.Tk()

text = tk.Text(root, width=80, height=20)
text.pack()

frame = tk.Frame(root)
tk.Button(frame, text="Start", command=start_reader).pack(side="left")
tk.Button(frame, text="Stop", command=stop_reader).pack(side="left")
frame.pack()

# start data monitor task
data_monitor(queue)

root.mainloop()

Update@2021-04-16: Example without using class
import threading
import time
from queue import SimpleQueue
import tkinter as tk
import serial

# create the serial instance
ser = serial.Serial(port="COM1") # provide other parameters as well
# create the queue for holding serial data
queue = SimpleQueue()
# flag use to start/stop thread tasks
stop_flag = None

def readSerial(queue):
    global stop_flag
    if stop_flag:
        print("Reading task is already running")
    else:
        print("started")
        stop_flag = threading.Event()
        while not stop_flag.is_set():
            if ser.in_waiting:
                ser_bytes = ser.readline()
                data = ser_bytes.decode("utf-8")
                queue.put(data)
            else:
                time.sleep(0.1)
        print("stopped")
        stop_flag = None

def start_reader():
    threading.Thread(target=readSerial, args=(queue,)).start()

def stop_reader():
    if stop_flag:
        stop_flag.set()
    else:
        print("Reading task is not running")

# function to monitor whether data is in the queue
# if there is data, get it and insert into the text box
def data_monitor(queue):
    if not queue.empty():
        text.insert("end", f"{queue.get()}\n")
    root.after(100, data_monitor, queue)

root = tk.Tk()

text = tk.Text(root, width=80, height=20)
text.pack()

frame = tk.Frame(root)
tk.Button(frame, text="Start", command=start_reader).pack(side="left")
tk.Button(frame, text="Stop", command=stop_reader).pack(side="left")
frame.pack()

# start data monitor task
data_monitor(queue)

root.mainloop()

Comment: Thank you. I will check the code in some hours. However, is there a way to do this without using classes? I find it complicated that we have function declarations, and then function declarations inside the class. One more question, do you think this approach will also handle the errors, that i got yesterday, that are referenced in my comment?
Comment: @user1584421 Answer updated with example not using class.  For the errors you mentioned in your comment, they are related to the data sent from the arduino.  You can try not calling `decode("utf-8")` on the data returned.
Comment: Thank you. I run some experiments today, and the behavior is completely inexplicable. I made a question here. https://stackoverflow.com/questions/67144017/inconsistent-serial-fails. Can you please take a look at it, and tell me if your implementation will solve those errors? Because if i have to do your implementation, i have to redesign the program from the ground up and make a lot of changes. If you believe those problems will be solved by your implementation, please let me know. Thank you,
Comment: Sorry but your code did not work. I updated my question with your version of the code i used.


How to get records with in fixed date interval in sql

Question: The federal aviation authority wants to know how many types of aircraft each of your pilots (employee only i.e. EMP) is currently certified (qualifies) to fly. Assume that the validity of a certification is only for one year. So, list the pilots and the number of types of aircraft each is currently qualified to fly (date_certified). Please list pilot full name, license number, and number of types s/he is currently qualified (certified) to fly. (hint: SYSDATE returns today's date like date() in MS Access. (SYSDATE +30) is 30 days from today).
HERE IS THE PILOT TABLE:
pilot_nbr, license_nbr, last_name, first_name, title, address, phone, office_nbr, contract_type, salary, manager_nbr

'701', '7111', 'Dark', 'Jack', '1st officer', '6 street', '6042233445', '789', 'PTE', '145000.00', NULL
'702', '7222', 'Mack', 'Bill', '1st officer', '7 street', '6043344556', '890', 'EMP', '155000.00', '701'
'703', '7333', 'Cheung', 'Charles', '2nd officer', '8 street', '6044455667', '503', 'PTE', '140000.00', '701'
'704', '7444', 'Gordon', 'Greg', '1st officer', '9 street', '6045566778', '123', 'EMP', '125000.00', '701'
'705', '7555', 'Basso', 'Nicki', '2nd officer', '5 street', '6046677889', '223', 'EMP', '163000.00', '701'
'706', '7666', 'Vettel', 'Sebast', '1st officer', '5 street', '6046677800', '523', 'EMP', '199000.00', '701'
'707', '7777', 'Hawke', 'Mike', '2nd officer', '7 street', '6046677326', '423', 'EMP', '139000.00', '701'

HERE IS THE QUALIFIES TABLE:
pilot_nbr, plane_type, date_certified

'701', 'DE Prop', '2013-08-14'
'701', 'SE Prop', '2013-04-10'
'702', 'DE Prop', '2013-02-22'
'702', 'SE Jet', '2013-04-10'
'702', 'SE Prop', '2013-06-21'
'703', 'DE Prop', '2013-04-10'
'703', 'SE Jet', '2013-04-10'
'703', 'SE Prop', '2013-04-10'
'704', 'DE Prop', '2013-06-12'
'704', 'ME Jet', '2013-08-21'
'704', 'SE Jet', '2013-06-15'
'704', 'SE Prop', '2013-04-10'

HERE IS WHAT I TRIED (DON'T KNOW IF IT IS CORRECT)
SELECT first_name, last_name, license_nbr, Count(first_name) As 'No of types of airplane certified to fly'
FROM pilot, qualifies
WHERE pilot.pilot_nbr=qualifies.pilot_nbr AND contract_type='EMP' AND date_certified >=  DATE_SUB(NOW(), INTERVAL 1 YEAR)
GROUP BY first_name, last_name

Your help will be appreciated.
Comment: How about posting your expected results?
Comment: Buddy, the query which I wrote will return no results and I think it should because there is no pilot which is currently certified to fly this is 2016 and their certified date if you see is in 2013 and valid for 1 year. Please correct me if I understood bad because that's why I am asking? Thanks
Answer: Hope this will solve your task...

SELECT p.first_name, p.last_name, p.license_nbr, Count(q.plane_type) As 'No of types of airplane certified to fly'
FROM pilot p inner join qualifies q on p.pilot_nbr = q.pilot_nbr
WHERE p.contract_type='EMP' AND DATE_SUB(q.date_certified, INTERVAL 1 YEAR) > CURDATE()
GROUP BY p.first_name, p.last_name, p.license_nbr;


Recursive unversion files in subversion without deleting them from the working copy

Question: I need to recursively unversion all *.pdf files in subversion
But want to keep them in the working copy.
Answer: First use delete with the --keep-local option, and then commit.
On Windows
for /R . %1 in (*.pdf) do svn delete "%1" --keep-local

On Linux
find ./ -name "*.pdf" -type f | xargs svn delete --keep-local

And if you want to ignore them:
svn propset svn:global-ignores "*.pdf" .


pyspark sql with where clause throws column does not exist error

Question: I am using pyspark to load csv to redshift. I want to query how manny rows got added.
I create a new column using the withcolumn function:
csvdata=df.withColumn("file_uploaded", lit("test"))

I see that this column gets created and I can query using psql. But, when I try to query using pyspark sql context, I get a error:
py4j.protocol.Py4JJavaError: An error occurred while calling o77.showString.
: java.sql.SQLException: [Amazon](500310) Invalid operation: column "test" does not exist in billingreports;

Interestingly, I am able to query other columnns but not just the new column I added.
Appreciate any pointers on how to resolve this issue.
Complete code:
df=spark.read.option("header","true").csv('/mnt/spark/redshift/umcompress/' +
filename)
csvdata=df.withColumn("fileuploaded", lit("test"))

countorig=csvdata.count()

## This executes without error
csvdata.write \
    .format("com.databricks.spark.redshift") \
    .option("url", jdbc_url) \
    .option("dbtable", dbname) \
    .option("tempformat", "CSV") \
    .option("tempdir", "s3://" + s3_bucket + "/temp") \
    .mode("append") \
    .option("aws_iam_role", iam_role).save()

select="select count(*) from " + dbname + " where fileuploaded='test'"

## Error occurs
df = spark.read \
.format("com.databricks.spark.redshift") \
.option("url", jdbc_url) \
.option("query", select) \
.option("tempdir", "s3://" + s3_bucket + "/test") \
.option("aws_iam_role", iam_role) \
.load()
newcounnt=df.count()

Thanks for responding.
Dataframe does have the new column called file_uploaded
Here is the query:
select="select count(*) from billingreports where file_uploaded='test'"
I have printed the schema
|-- file_uploaded: string (nullable = true)
df.show() shows that the new column is added.
I just want to add a pre determined string to this column as value.
Comment: Please add all the code to the question. How is billingreports created? What is the command that gives the error?
Comment: There must be some other transformations not shown here. The exception is on a dataframe called billingreports while you add the column to csvdata.
Comment: I have a dataframe called csvdata which I use to write to redshift. U have the count of that dataframe. I want to validate with whats written to redshift. The dataframe I am trying to construct out of the sql query select="select count(*) from billingreports where file_uploaded='test'". This is where the error is showing up.
Answer: Your Dataframe csvdata will have a new column named file_uploaded, with default value "test" in all rows of df. This error shows that it is trying to access a column named test, which does not exist in the dataframe billingreports thus the error. Print the schema before querying the column with billingreports.dtypes or better try to take a sample of your dataframe with billingreports.show() and see if the column has correct name and values.
It will be better if you share the query which resulted in this exception, as the exception is thrown for dataframe billingreports.
Comment: Thanks for responding. Dataframe does have the new column called file_uploaded Here is the query: select="select count(*) from billingreports where file_uploaded='test'"

I have printed the schema

|-- file_uploaded: string (nullable = true)

df.show() shows that the new column is added. I just want to add a pre determined string to this column as value.
Comment: withColumn expects second argument to be a Column. The way you are assigning a default column looks good to me, a string wrapped in lit function. Can you try using Spark SQL to see if it makes any difference. Something like this : df.createOrReplaceTempView("df"); csvdata=spark.sql("SELECT a.*, '{0}' AS fileuploaded FROM df a").format("test"))


Prove subspace and subsets or R are polish space

Question: "Let a,b∈R, with a < b. Prove that each discrete subspace of R and each of the spaces [a,b], (a,b), [a,b),(a,b], (−∞,a), (−∞,a], (a,∞), [a,∞), and {a}, with the topology induced as a subspace of R, is a Polish Space."
This question is a bit confusing. What exactly are discrete subspace of R? Is it just any subset or R with the discrete topology?
Also, Polish space means separable and completely metrizable, but just because R is complete under the usual Euclidean metric doesn't mean a subspace is complete so I must be confusing something.
Comment: A toplogical space is Polish if there is a metric which induces the given topology and makes it a complete, separable metric space.
Comment: Any $G_{\delta}$ subset of a  Polish space is Polish.
Comment: "Discrete subspace of $\Bbb{R}$" means a subset $X$ such that the usual topology on $\Bbb{R}$ induces the discrete topology on $X$. (The challenge here for such an $X$ is to show that it is finite or countably infinite.)
Answer: Each discrete subspace $D$ of $\Bbb R$ is at most countable, and has the discrete topology as its subspace topology (that's the definition of being discrete), and this topology can be induced by the discrete metric $d_1(x,y)=1$ whenever $x \neq y$, and this metric is complete (every $d_1$-Cauchy sequence is eventually constant, and hence converges). So $D$ is Polish (separable, due to its size) and completely metrisable.
Indeed $\Bbb R$ itself is Polish (it's usual metric will show that, and $\Bbb Q$ is countable and dense), and hence so is every closed susbet of $\Bbb R$: we just inherit the metric and the subspace is still sepaarble and compleet in the inherited metric, by standard facts. This handles sets like $[a,b]$, $[a,\infty)$ and $(-\infty,a]$ e.g. and also $\{a\}$ (it's discrete too BTW).
$[a,b) \simeq [a,\infty)$ so if one is Polish so is the other (being Polish only dpends on the topology; it's preserved by homeomorphism), etc. $(a,b) \simeq \Bbb R$ is Polish for the same reason.
Comment: Ok I understand that all intervals contain the rationals so they are separable and closed subsets of complete metric space are closed under the induced topology so that also makes sense, but I still dont get why (a,b) is a Polish space. The usual euclidean metric does not make this complete.
Comment: @William it's homeomorphic to $\Bbb R$ via a $\tan$ like function e.g. completely metrisable means there exists some metric that makes it complete; this doesn't need to be the original metric.


Recommended effect size measure for repeated measures t-tests and repeated measures contrasts on experimental data

Question: Context
This came up recently in a consulting context. A researcher was performing repeated measures t-tests based on experimental data.
Some of the analyses involved comparing one condition with another. Other analyses involved performing contrasts comparing one or more conditions with one or more other conditions.
Question

What effect size measure would you recommend using in relation to (a) repeated measures t-tests (b) repeated measures contrasts comparing one or more group means with one or more other group means?
If you were reporting a d-based effect size measure, which measure of the within group standard deviation would you use?
Are there any references that you would recommend?

I have a few thoughts, but I'm keen to get your suggestions.
Answer: The answer here depends on your situation.  Dunlap, Cortina, Vaslow, and Burke (1996) argued that the effect size should be calculated using a SD based on pooled variance from separate conditions, as is typical in independent groups studies, even with repeated measurements.  Their argument was that the study may be replicated with a between design and the effect sizes will be more comparable across studies in meta-analysis with that effect size measure. They asserted that the effect size is the effect size and shouldn't be influenced by the correlation in the measurement in a repeated measures design.
Unfortunately, this suggestion has been overgeneralized in some literatures (and in Cortina's book I believe).  When it's not possible to design an experiment any other way than repeated measures then using the between S effects size is a mistake.  It will underestimate the size of the effect and be useless in power calculations.
Imagine an attentional cueing study where you need to study a single mental state, (e.g. oriented in a direction indicated by an arrow), and have to measure the effect comparing performance at the indicated location and one that is not.  That study has to be done within and there is no other way to do it.  In that case, the need to have an effect size comparable to situations where the study is done with independent groups vanishes because the independent group study couldn't occur.  Using the between S effect size would not be a useful estimate in seeing the number of S's to replicate the study while the within would.  The between S effect size would tend to vastly underestimate what you actually needed to measure, which is the effect within.
Comment: Doesn't generalized eta-square, available via `ezANOVA` from the `ez` package, solve the problem of generating an effect size estimate that generalizes across experimental designs?
Comment: +1 @John I agree with Dunlap et al's arguments. I don't quite follow your claim that it has been over-generalised. In g-power 3 for example, if you want to calculate statistical power you supply the correlation between the two conditions and a between subjects measure of d. And even when a between subjects design is not possible, I would still find the between subjects measure of effect size a more useful standardisation metric for describing the amount of change that has been observed.
Comment: When I've seen people follow the Dunlap et al recommendation I've not seen the correlation reported.  Reporting the correlation would help.
Comment: How is between S effect size more useful when only repeated measures are possible?
Comment: @John Good point about power analysis. Having only the d without the correlation does lead to some guess work. I guess standardisation is an attempt to make a metric that lacks an intuitive or comparable interpretation have an intuitive and comparable interpretation. Thus, the sample standard deviation provides one such metric. If I want to contextualise the size of an effect, I want to be able to be able to interpret effect sizes on a common playing field. I don't want the effect size to appear larger just because it's been calculated using a repeated measures design.
Comment: I don't want it to appear larger because it happens to be repeated measures either.  That's why I don't suggest tossing Dunlap et al all together.  But if what you measure must, by it's very nature, be repeated measures, then that's going to be the effect size.  I'm trying to imagine a situation where I'd want to contextualize such an effect among between S effects.  I can think of literatures (comparing L/R ears) where that never happens but between S ES is always used following Dunlap (et al) and there's *possible* context but no immediate utility -> overgeneralization.
Comment: how?  I don't think it does.  And if it did how does it fix the problems of using a between Cohen's D on strictly with designs?


Why do Baseball players cover their bats?

Question: In Baseball, why do the batters waiting to go and bat have some kind of cover on their bat? It's not a full cover, it just seems to be something that goes around the thick part of the bat. It looks like a knee/elbow protector that someone has slid over their bat.
He wasn't even swinging at balls or anything, just swinging at air. It seemed almost like he was trying to cover up what was on the bat (like a picture or text)
Answer: Alright so these are what used to be known as 'donuts.' They are a weight sleeve or ring that the player puts on the bat to make it heavier as they warm up. They take the practice swings with the bat heavier so their muscles adjust to the weight, then remove it so the bat feels much lighter and they have a quicker swing. It is a very effective tactic that I use myself when I play.
Comment: Also just FYI - there is evidence that these are actually ineffective and even detrimental - https://en.m.wikipedia.org/wiki/Baseball_doughnut
Comment: @user1167442 Yes, I wish I could upvote your comment 100 times. This is yet another traditional baseball practice that is detrimental to player performance. Thank you for pointing that out


Compendium of cross-validation techniques

Question: I'm wondering if anybody knows of a compendium of cross-validation techniques with a discussion of the differences between them and a guide on when to use each of them. Wikipedia has a list of the most common techniques, but I'm curious if there are other techniques, and if there are any taxonomies for them.
For example, I just run into a library that allows me to choose one of the following strategies:

Hold out
Bootstrap
K Cross-validation
Leave one out
Stratified Cross Validation
Balanced Stratified Cross Validation
Stratified Hold out
Stratified Bootstrap

and I am trying to understand what stratified and balanced mean in bootstrapping, hold out or CV.
We can also turn this post into a community wiki if people want ,and collect a discussion of techniques or taxonomies here.
Comment: This great question would be even more helpful if we could link to explanations of each of the methods.
Answer: You can add to that list:

Repeated-cross validation
Leave-group-out cross-validation
Out-of-bag (for random forests and other bagged models)
The 632+ bootstrap

I don't really have a lot of advice as far as how to use these techniques or when to use them.  You can use the caret package in R to compare CV, Boot, Boot632, leave-one-out, leave-group-out, and out-of-bag cross-validation.
In general, I usually use the boostrap because it is less computationally intensive than repeated k-fold CV, or leave-one-out CV.  Boot632 is my algorithm of choice because it doesn't require much more computation than the bootstrap, and has show to be better than cross-validation or the basic bootstap in certain situations.
I almost always use out-of-bag error estimates for random forests, rather than cross-validation.  Out-of-bag errors are generally unbiased, and random forests take long enough to compute as it is.
Comment: Any advice concerning when to use each of these?
Answer: K-fold cross-validation (CV) randomly breaks your data up into K partitions, and you in turn hold out one of those K parts as a test case, and lump the other K-1 parts together as your training data. Leave One Out (LOO) is the special case where you take your N data items and do N-fold CV. In some sense, Hold Out is another special case, where you only choose one of your K folds as test and do not rotate through all K folds.
As far as I know, 10-fold CV is pretty much the de rigueur, since it uses your data efficiently and also helps to avoid unlucky partition choices. Hold Out does not make efficient use of your data, and LOO is not as robust (or something like that), but 10-ish-fold is just right.
If you know that your data contains more than one category, and one or more categories are much smaller than the rest, some of your K random partitions might not even contain any of the small categories at all, which would be bad. To make sure each partition is reasonably representative, you use stratification: break your data up into the categories and then create random partitions  by choosing randomly and proportionally from each category.
All of these variations on K-fold CV choose from your data without replacement. The bootstrap chooses data with replacement, so the same datum can be included multiple times and some data might not be included at all. (Each "partition" will also have N items, unlike K-fold, in which each partition will have N/K items.)
(I'll have to admit that I don't know exactly how the bootstrap would be used in CV, though. The principle of testing and CV is to make sure you don't test on data that you trained on, so you get a more realistic idea of how your technique + coefficients might work in the real world.)
EDIT: Replaced "Hold Out is not efficient" with "Hold Out does not make efficient use of your data" to help clarify, per the comments.
Comment: By "efficient" I mean that it does not use your data efficiently. The beauty of K-fold CV is that 100% of your data gets used for training and 100% of your data gets used for testing, which makes efficient use of your data. The key is, of course, that all of your data does not get used for testing and training at the same time, which would yield way-optimistic test results (overfitting). By making a static division, Hold Out says that, say, 1/3 of your data will never be used for training, and 2/3 of your data will never be used for testing, wasting a lot of information in your data.
Comment: When you say that "Hold Out is not efficient", I'm not sure I follow. What do you mean by efficiency here? As opposed to regular N-fold, Hold Out does not rotate through the folds, so it should be faster. Do you mean instead that it's worse at fighting overfitting than regular N-fold CV?
Comment: @Wayne Isn't the holdout estimator *asymptotically* unbiased? Also, simple k-fold CV is more prone to type II error than repeated k-fold CV.
Comment: @chl: I believe you're right on both counts. I haven't used repeated K-fold CV, but it should have lower variance, which would help.
Answer: I found one of the references linked to in the Wikipedia article quite useful
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.48.529&rep=rep1&type=pdf
"A study of cross-validation and bootstrap for accuracy estimation and model selection", Ron Kohavi, IJCAI95
It contains an empirical comparison for a subset of CV techniques. The tl;dr version is basically "use 10-fold CV".
Answer:
...and a guide on when to use each of them...

Unfortunately that problem is harder than it gets credit for. There are at least 2 main uses of cross-validation: selecting a model, and evaluating model performance.
Roughly speaking, if you use a CV variant which splits the data using a high train-to-test ratio, this can be better for evaluation. Using a larger training set will more accurately mimic the performance of the model fit on the full dataset.
But a high train-to-test ratio can be worse for selection. Imagine there really is a "best" model that you "ought" to choose, but your dataset is quite large. Then, too-large models which overfit slightly will have almost the same CV performance as the "best" model (because you'll successfully estimate their spurious parameters to be negligible). Randomness in the data and the CV/splitting procedure will often cause you to choose an overfitting model instead of the truly "best" model.
See Shao (1993), "Linear Model Selection by Cross-Validation" for older asymptotic theory in the linear regression case. Yang (2007), "Consistency of Cross Validation for Comparing Regression Procedures" and Yang (2006), "Comparing Learning Methods for Classification" give asymptotic theory for more general regression and classification problems. But rigorous finite-sample advice is hard to come by.
Comment: @paperskilltrees Asymptotically, moderate and high train-to-test ratios **do** both have this same problem. To avoid it, papers like Shao 1993 actually assume a vanishing train-to-test ratio: n_train/n_test going to 0 as n grows. This ensures that 1. the too-large model really does have a poorer fit to the training data than the true model does (the true difference b/w these two fitted models is non-negligible); and 2. the test set is large enough to detect this difference. Alas, it's nice in theory but not very practical as far as finite-sample guidance goes.
Comment: "_Randomness in the data and the CV/splitting procedure will often cause you to choose an overfitting model instead of the truly "best" model._"
I wonder, why does not this apply to moderate ($K\ll N$) as much as it applies to high train-to-test ratio splits (e.g., the leave-one-out cross-validation)?


Mysql did not start

Question: I tried to restart mysql. However it did not restart. Here is the content of mysql.log:
131217 17:01:09  InnoDB: Starting shutdown...
131217 17:01:11  InnoDB: Shutdown completed; log sequence number 153229469
131217 17:01:11 [Note] /usr/libexec/mysqld: Shutdown complete

131217 17:01:11 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
131217 17:01:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131217 17:01:11 [Note] Plugin 'FEDERATED' is disabled.
131217 17:01:11 InnoDB: The InnoDB memory heap is disabled
131217 17:01:11 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131217 17:01:11 InnoDB: Compressed tables use zlib 1.2.3
131217 17:01:11 InnoDB: Using Linux native AIO
^G/usr/libexec/mysqld: Can't create/write to file '/tmp/ibsMvrkL' (Errcode: 13)
131217 17:01:11  InnoDB: Error: unable to create temporary file; errno: 13
131217 17:01:11 [ERROR] Plugin 'InnoDB' init function returned error.
131217 17:01:11 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
131217 17:01:11 [ERROR] Unknown/unsupported storage engine: InnoDB
131217 17:01:11 [ERROR] Aborting

Can any one help me to find the reason of this? In fact mysql was working before this time and it seems that it is not a regular error!!Please let me know if any other information is nedeed.
Thanks a lot
Comment: Given the specific error - share the output of both `df -h` and `mount` without parameters. Your `/tmp` mount is probably either full or mounted read-only.
Answer: Have you checked if there is a problem about your disk partition. May be its out of space or have a problem. The error clearly shows the problem is caused by creating temp file failure. Rest is listed because InnoDB failed to initialize.
Comment: You also could suffer a run out of inodes in a partition (note you can have en enough free space, but not enough inodes). You can check this with df -ih
Answer: This error clearly mentioned that MySQL daemon unable to create a temporary file under /tmp directory.
Please check the permission of /tmp directory.
Your /tmp directory permission should have a sticky bit. below is the example for same.
drwxrwxrwt   3 root root 4.0K Feb  4 13:48 tmp

Mysql did not start


Continuous function on a closed interval is integrable proof

Question: I'm reviewing my prof's slides in which he proves that a continuous function on a closed interval is integrable.
He has defined:

$m_i=\inf\{f(x) : x\in[t_{i-1}, t_i]\}$
$M_i=\sup\{f(x) : x\in[t_{i-1}, t_i]\}$

In the proof, he says:

$\displaystyle|f(x)-f(y)|<\frac{\epsilon}{2(b-a)}\ \forall x, y\in[t_{i-1}, t_i]$
$\displaystyle\therefore M_i-m_i\le\frac{\epsilon}{2(b-a)}$

How did he get the second step from the first? In particular, how did $<$ become $\le$?
Answer: Each interval $[t_{i-1},t_i]$ is compact, and the restriction of $f$ to it is still continuous, so we can use the extreme value theorem to deduce that actually there exist $x,y \in [t_{i-1},t_i]$ such that $f(x) = M_i$ and $f(y) = m_i$.  The bound on $M_i - m_i$ then follows from the first bound, and there's no need to switch from $<$ to $\le$.  I'd guess the swap is just a typo, though it's certainly not incorrect.


i18n - Not translating Menu's or Blocks

Question: The Problem
Using the i18n module, node translations are working, however Menu and Block translations are not showing up.
For example: I created 2 identical blocks. For one i set the Show this block for these languages to English and the other to Bulgarian. When i then go mysite.com it shows me the English block, but when i go to mysite.com/bg it still shows me the English block.
I have also tried the Make this block translatable but that doesn't work either.
The setup
Drupal version: 7.14
Related modules: i18n (all enabled apart from Multilingual Forum), pathauto, localization update, administration language, locale
Background
This is my first Drupal 7 website. The website im working on is a multilingual, multi-country website which is near completion. Hopefully that gives some idea of my exerpience, so if you could tailor your answers to that please.
Thanks in advance.
Comment: What exactly doesn't work in "Make this block translatable"?
Comment: Well i checked the box, i added translations for each language, went to the page and it wasn't translating it.
Comment: Are you sure you where not seeing that previous version of the English block that you mention in your question?
Comment: Right so i've now hidden the Bulgarian copy of the block i created (in the block list), and gone into the English block. I have unchecked all the language fields and checked *Make this block translatable*. I have then gone to the *Translate* tab in the block and created a translation for Bulgaria. Then i went to my website and using the Language Switcher selected Bulgaria. On the page i can see translated versions of my nodes, but the block still shows the English copy.
Comment: So do you confirm that it doesn't work if you create a new block, make it translatable and translate it?
Comment: Yes. I can confirm that.
Answer: if anyone else is experiencing this issue, notice that it can also be caused by your configuration in yoursite/admin/config/regional/language/configure, even if you are not using Admin Language Module.
Menus are part of your interface, not of content. So, for example, if you set your interface language detection to default (only use site default language), because you want your interface to be only in english, you are not going to see your translated menu blocks when you navigate to yoursite/anotherlanguage!
Hope this helps someone!
Comment: `Menus are part of your interface, not of content.` - wow - ok - ofcourse - thanks
Answer: For those that encounter this issue, the problem is being caused by another module. Disabling the below module fixed the issue.
Drupal 7 - Admin Language Module
Comment: I had a similar side effect with this module at http://drupal.stackexchange.com/questions/37967/node-save-does-not-save-language
Answer: One thing to check regarding menu items being translated or not. Make sure that your menus are not themed via CSS with images that contain a specific language.

i18n - Not translating Menu's or Blocks


My contact form isn't working

Question: I received this code from my host for my contact form
<?php
$request_method = $_SERVER["REQUEST_METHOD"];
if($request_method == "GET")
{
    $query_vars = $_GET;
}
elseif ($request_method == "POST")
{
    $query_vars = $_POST;
}

reset($query_vars);
$t = date("U");
$file = $_SERVER['DOCUMENT_ROOT'] . "\ssfm\gdform_" . $t;
$fp = fopen($file,"w");

while (list ($key, $val) = each ($query_vars))
{
    fputs($fp,"<GDFORM_VARIABLE NAME=$key START>\r\n");
    fputs($fp,"$val\r\n");
    fputs($fp,"<GDFORM_VARIABLE NAME=$key END>\r\n");
    if ($key == "redirect")
    {
        $landing_page = $val;
    }
}

fclose($fp);

if ($landing_page != "")
{
    header("Location: http://".$_SERVER["HTTP_HOST"]."/$landing_page");
}
else
{
    header("Location: http://".$_SERVER["HTTP_HOST"]."/");
}
?>

I DECIDED TO USE  THIS
<?php
if ($_POST['parse_Var'] == "test_contact") {
    $email_title = 'New Email From Your Website';
    $my_email = 'email@example.com';

    $email_field = $_POST['email'];
    $name_field = $_POST['name'];
    $message_field = $_POST['message'];

    $body = <<<EOD

    Email: $email_field

    Name: $name_field

    Message: $message_field

EOD;

    $headers = "From: $email_field\r\n";
    $headers .= "Content-type: text/html\r\n";
    mail("$my_email", "$email_title", "$body", "$headers");

    header("location: http://www.phweshdeals.com/thanks.html");

}

?>

HERE IS MY HTML FORM WITH PHP EXTENSION
<form action="config/test_contact.php" method="post">

Name:

<input id="name" class="field" name="name" value="<?php print "$name_field"; ?>" placeholder="enter your name..." type="text" required>*required

Email:

<input id="email" class="field" name="email" value="<?php print "$email_field"; ?>" placeholder="enter your email..." type="email" required>*required

                        Your Message

                        <textarea id="message" name="message" value="<?php print "$messgae_field"; ?>" autocomplete="on" placeholder="type your message here..."></textarea>*required

<input type="checkbox" required>

Confirm you are NOT a spammer

<input type="submit" value="submit" style="background-color:#287472;
                        color:#fff;text-shadow:1px 1px #000; text-transform:uppercase;
                         font-weight:bold; border-radius:5px;"
                      id="submit_button">

                      <input type="reset" value="clear!" style="background-color:#287472;
                        color:#fff;text-shadow:1px 1px #000; text-transform:uppercase;
                         font-weight:bold; border-radius:5px;"
                      id="reset_bg">

                <input type="hidden" name="parse_Var" id="parse_Var" value="test_contact">

        </form>

IF I USE My host's php code, it doesn't capture all the input fields that I would like to capture if a client fills the form.
I decided to create mine, which doesn't even work at all. if I click on submit, I get this message in the browser; "No input file specified.".
I have tried different php form formats, yet non worked for me. Could the problem be from my host? this is my host by the way; www.websitepalace.com . I have contacted them, but I couldnt get any tangible solution. I also heard some hosts purposely to this to control / prevent their customers from spamming people or something. not quit cleared.
Please, I need your help.
Answer: There might be more issues but one I see first is: in all the input fields, update the value attribute to remove quotation marks from the PHP variables:
print "$name_field";

to
print $name_field;

My contact form isn't working


Pandas DataFrame Transpose multi columns

Question: I have a dataframe as below.
a   x   10
b   x   11
c   x   15
a   y   16
b   y   17
c   y   19
a   z   20
b   z   21
c   z   23

and I want to transform it as below:
    x   y   z
a   10  16  20
b   11  17  21
c   15  19  23

Currently I am making the original DF into multiple Data Frames (each for "a", "b" and "c") and then transposing and merging back.
I am sure there would be an optimum solution. Hence looking for help.
Answer: Use pivot:
print (df)
   A  B   C
0  a  x  10
1  b  x  11
2  c  x  15
3  a  y  16
4  b  y  17
5  c  y  19
6  a  z  20
7  b  z  21
8  c  z  23

df = df.pivot(index='A', columns='B', values='C')
print (df)
B   x   y   z
A
a  10  16  20
b  11  17  21
c  15  19  23

Or set_index + unstack:
df = df.set_index(['A','B'])['C'].unstack()
print (df)
B   x   y   z
A
a  10  16  20
b  11  17  21
c  15  19  23

If duplicates use pivot_table with aggregate function like mean, sum...:
print (df)
   A  B   C
0  a  x  10 <-same a,x different C = 10
1  a  x  13 <-same a,x different C = 13
2  b  x  11
3  c  x  15
4  a  y  16
5  b  y  17
6  c  y  19
7  a  z  20
8  b  z  21
9  c  z  23

df = df.pivot_table(index='A', columns='B', values='C', aggfunc='mean')

Or groupby + aggregate function + set_index:
df = df.groupby(['A','B'])['C'].mean().unstack()
print (df)
B     x     y     z
A
a  11.5  16.0  20.0 <- (10 + 13) / 2 = 11.5
b  11.0  17.0  21.0
c  15.0  19.0  23.0


Array-3 fix45 coding bat

Question: i am trying to solve an Array problem on coding bat the assignment is

Return an array that contains exactly the same numbers as the given array, but rearranged so that every 4 is immediately followed by a 5. Do not move the 4's, but every other number may move. The array contains the same number of 4's and 5's, and every 4 has a number after it that is not a 4. In this version, 5's may appear anywhere in the original array.

my code passes all the tests except the one below
([4, 9, 4, 9, 5, 5, 4, 9, 5]) → [4, 5, 4, 5, 9, 9, 4, 5, 9]

here is my code
public int[] fix45(int[] nums) {
      int val =0, lastF = 0, var;
      for (int i =0; i < nums.length-1; i++)
        if (nums[i] == 4){
          var = i+1;
          for (int j = 0; j < nums.length; j++)
            if ((nums[j] == 5) && (j >= lastF && j != var)){
              lastF =j;
              val = nums[i+1];
              nums[i+1] = nums[j];
              nums[j] = val;
            }
        }
      return nums;
    }

my code returns
[4, 5, 4, 5, 9, 5, 4, 9, 9]

if anyone can help me figure out what i need to tweak to make it pass this last test thank you
Comment: Run your code step-by-step under debugger and see for yourself.
Comment: If you know what's wrong, what stops you from fixing it?
Comment: i know what the issue is that lastF variable becomes greater that j which is the index  that searches for the number 5 i try setting a break statement and even making lastF = 0 each loop but that does not work @PM77-1
Answer: It can be done in one while loop just by using two flags and two pointers!
public int[] fix45(int[] nums) {

      int start = 0;
      int end = nums.length-1;

      boolean is4 = false;
      boolean is5 = false;

      while( start < nums.length ){

        if(nums[start] == 4 && nums[start+1]!=5){
          is4 = true;
        }

        if(nums[end] == 5 && (end == 0 || (end-1>=0 && nums[end-1]!=4))){
          is5 = true;
        }

        if(is4 && is5){
          int temp = nums[start+1];
          nums[start+1] = nums[end];
          nums[end] = temp;

          is4 = false;
          is5 = false;
          end = nums.length-1;
        }

        if(is4){
          end--;
          continue;
        }

        start++;
      }
      return nums;

    }

Answer: You can use a flag so your second loop stops iterating when it finds the next element 5. Also instead of iterating from 0 on the second loop you could iterate from i instead so it becomes more efficient.
    boolean isFound = false;
    for (int i = 0; i < nums.length - 1; i++) {
        if (nums[i] == 4) {
            var = i + 1;
            for (int j = i; j < nums.length && !isFound ; j++) {
                if ((nums[j] == 5) && (j >= lastF && j != var)) {
                    lastF = j;
                    val = nums[i + 1];
                    nums[i + 1] = nums[j];
                    nums[j] = val;
                    isFound = true;
                }
            }
            isFound = false;
        }
    }

Answer: You are setting lastF to j when you should set it to i+2. You are replacing i+1 with j so you should set lastF to one after i+1(i+2) instead of j. Also, you have to start the j-forloop at lastF.
It should look like this:
int val =0, lastF = 0, var;
for (int i =0; i < nums.length; i++)
if (nums[i] == 4){
          var = i+1;
          for (int j = lastF; j < nums.length; j++)
            if ((nums[j] == 5) && (j >= lastF && j != var)){
              lastF =i+2;
              val = nums[i+1];
              nums[i+1] = nums[j];
              nums[j] = val;
            }
        }

Comment: hey your code pass that particular test but fails these two test case            5, 5, 4, 1, 4, 1 it returns  1, 5, 4, 5, 4, 1 instead of 1, 1, 4, 5, 4, 5  the 2nd case is 5, 4, 5, 4, 1 it returns 5, 4, 5, 4, 1 instead of 1, 4, 5, 4, 5 but it passes the rest
Comment: Did you try the other answer?
Comment: No I mean @anto004 also answered has answered did you try his code?
Comment: I tried the other way it did not work. I will keep trying though.
Comment: What I would suggest is to move all data other than 4 or 5 to the end and then from there you can sort the 4 and 5s easier.

Array-3 fix45 coding bat


ClassCastException onAttach() when unit testing Android Fragment

Question: I'm trying to test a Fragment that has in interface that must be implemented by the hosting Activity and gets casted to that specific interface's type through onAttach().
Problem: I'm not sure how to implement the necessary interface methods within an Android Unit Test or if it's even necessary to do so. Surprisingly, I haven't found any posts or forums that address this issue.
Test:
public class FragmentTest {

    private ActivityForUnitTesting fragmentHostActivity;
    private ExampleFragment fragmentToTest;

    @Rule
    public ActivityTestRule activityTestRule = new ActivityTestRule<>(ActivityForUnitTesting.class);

    @Before
    public void setUp() {
        fragmentHostActivity = (ActivityForUnitTesting) activityTestRule.getActivity();
        fragmentManager = fragmentHostActivity.getSupportFragmentManager();
        fragmentToTest = new ExampleFragment();
    }

    @Test
    public void testExample() {
        fragmentManager.beginTransaction()
                .replace(R.id.frame_layout_container, fragmentToTest)
                .commit();
    }

}

Fragment:
public class ExampleFragment extends Fragment {

    private ExampleFragmentListener exampleFragmentListener;

    ...

    @Override
    public void onAttach(Context context) {
        super.onAttach(context);
        try {
            exampleFragmentListener = (ExampleFragmentListener) context;
        } catch (ClassCastException e) {
            throw new ClassCastException(context.toString() + " must implement ExampleFragmentListener");
        }
    }

    ...

}

but when I try running a simple test I get:

java.lang.ClassCastException: com.example.package.ActivityForUnitTesting@1234567 must implement ExampleFragmentListener
  at com.example.package.ExampleFragment.onAttach(ExampleFragment.java:)

I know that the issue is that my Unit Test ActivityForUnitTesting object does not implement the required interface methods. My question is, how do I safely implement those methods within my Unit Test. I haven't had any luck finding a similar question or a solid example.
Comment: https://medium.com/@aitorvs/isolate-your-fragments-just-for-testing-ea7d4fddcba2
Answer: I didn't find a solution to this, but I did find a "workaround". Instead of using onAttach(), explicitly set your listener through a public method.
public class ExampleFragment extends Fragment {

    private ExampleFragmentListener exampleFragmentListener;

    ...

    //@Override
    //public void onAttach(Context context) {
    //    super.onAttach(context);
    //    try {
    //        exampleFragmentListener = (ExampleFragmentListener) context;
    //    } catch (ClassCastException e) {
    //        throw new ClassCastException(context.toString() + " must implement ExampleFragmentListener");
    //    }
    //}

    public void setExampleFragmentListener(ExampleFragmentListener exampleFragmentListener) {
        this.exampleFragmentListener = exampleFragmentListener;
    }

    ...

}

then, you should already have ExampleFragmentListener implemented in your host Activity. Just call
setExampleFragmentListener(ActivityOrClassThatImplementsExampleFragmentListener)
from wherever you perform your Activity setup. As a result, the test shouldn't complain about unimplemented methods.


Creating a News Blog in Drupal

Question: Can anybody help me with this, any help is appreciated.
I want to create a news blog where any anonymous user can put the news post, but for posting news article,
the user must provide their valid email, the verification mail should be sent to that user. News should only be posted after only the email is verified.
Answer: This has been discussed at Use email to validate post, not unlike craigslist.
A module that is still in alpha is also available:
https://drupal.org/project/anonymous_publishing
The module is not widely used, thus not extensively tested. It might be safer to write your own rules as discussed in the first link.

Creating a News Blog in Drupal


Dúvida sobre Zend Framework

Question: Desenvolvi uma aplicação na versão 2.5 da ZF2, e me veio uma dúvida caso não atualizar para versão ZF3 minha aplicação poderá ocorrer algum problema?
Comment: Primeiramente, está pegunta é muito ampla e pode ter várias respostas. Mas na verdade, não vai acontecer nenhum problema diretamente. O que pode acontecer é vc precisar de alguma funcionalidade nova do ZF3, ou mesmo algum conserto. Ou ainda quiser utilizar uma lib que dependa das versões mais novas do ZF. Caso vc não atualize não poderá utiliza-los.
Answer: Com certeza ocorrerá problemas, atualizei o meu do 2.5 para o 2.7 e apresentou erro no servicemanager, na verdade seria um aviso de que a forma como eu estava usando estaria obsoleto e que no ZF3 não iria funcionar.
Me desculpe mas não tenho maiores detalhes, mas faça uma busca e com certeza terá maiores detalhes.
Comment: Na verdade ele perguntou o contrário. Se **não** atualizar poderá ocorrer algum problema?
Comment: Verdade, eu interpretei o inverso. ;) @ViníciusFagundes

Dúvida sobre Zend Framework


Possible to specify min/max length of a string in JSON schema (v4)?

Question: I'm a non-developer and new to JSON and have created a JSON schema to validate JSON code when troubleshooting customer issues to insure syntactics against our defined data elements.
I noticed that I can only verify min/maxLength if the field type is an integer.  If the type=string, min/maxLength does not work.
JSON Schema code snippet:
<code>"LastName": {
    "type" : "string",
    "optional": false,
    "minLength": 1,
    "maxLength": 254,
    "description": "Last Name of Insured"
},
</code>
In JSON code, if "LastName": "" it still validates, but our WS/JSON call will fail because this field is required and requires data.
Thanks in advance!
Answer: It should work per the spec. Reference and quote below. Please elaborate "If the type=string, min/maxLength does not work." - what unexpected exacting behavior do you get?
https://json-schema.org/understanding-json-schema/reference/string.html

Length
The length of a string can be constrained using the minLength
  and maxLength keywords. For both keywords, the value must be a
  non-negative number.

<code>{
  "type": "string",
  "minLength": 2,
  "maxLength": 3
}
</code>


.Include() não carrega classe filha - Lazyloading C#

Question: Eu estava usando o Modelo DDD com Repositorios e  com 3 camadas de dados, (Serviço,Repositorio e Dominio), com a ajuda de alguns usuários aqui eu resolvi mudar o projeto e agora acabei por remover 2 camadas a serviço e a repositório do meu projeto.
Estou ainda estudando sobre este assunto que esta cada vez ficando mais claro para mim (agora entendo porque meu codigo era prolixo, como dito pelo usário @jbueno), eu tenho meu contexto declarado e estou usando a versão do 6.0 do EntityFramework.
Eu criei uma controller na minha camada Web e fiz a seguinte pesquisa :
 List<MensagemUnidade> mensagens = this.Contexto.MensagemUnidade
            .Include(c => c.Cliente).Where(l => l.UnidadeId == unidade.UnidadeAtual && l.OrigemId == (int)enumOrigemMensagem.ADMIN)
            .OrderByDescending(l => l.DataEnvio).Skip(mensagemModel.PaginaAtual * 20)
            .Take(20)
            .ToList();

Ela não trouxe o cliente,
Essa e minha classe Cliente:
public class Cliente : Pessoa
{
    public string FacebookFoto { get; set; }

    public ICollection<TokenCliente> TokensCliente { get; set; }
}

Minha classe MensagemUnidade:
 public class MensagemUnidade
  {
     public virtual Cliente Cliente { get; set; }
     public virtual Unidade Unidade { get; set; }
  }

O LazyLoading esta ativado,ele ate carrega o ClienteId Correto mas o Cliente fica null.
Eu estava com o modelo DDD e usando Repositorio, o meu projeto agora possui a seguinte estrutura, uma camada Dominio, uma  Camada Infra(onde fica meu contexto e minhas camadas Web/API , quando eu mudei o projeto fiquei em duvida sobre UnitofWorke coisas do tipo, agora esta tudo se encaixando melhor, mas ainda tenho esse problema.
Answer: Você só precisa explicitar na sua classe de relação que Cliente é uma entidade que será carregada a partir da chave ClienteId. Como Cliente é uma classe que herda de Pessoa e quem tem a PrimaryKey é esta segunda, rola uma certa confusão.
public class MensagemUnidade
{
    public int ClienteId { get; set; }

    [ForeignKey("ClienteId")]
    public virtual Cliente Cliente { get; set; }
 }

Uma observação importante sobre a seguinte frase

O LazyLoading esta ativado,ele ate carrega o ClienteId Correto mas o Cliente fica null.

O LazyLoading não é necessário para se usar o Include. Na verdade, quando se usa Include se está falando de eager loading (ou carga antecipada) e não de lazy loading (carga preguiçosa).
Em suma, a principal diferença entre os dois é que usando técnicas de eager loading (Include, Load) a entidade é carregada em memória no momento que query é materializada (veja mais sobre isso aqui), isto é, a consulta que é executada no banco de dados, já contém as informações referentes às entidades explicitadas (no seu caso, usando o método Include). Já no lazy loading, as entidades relacionadas são apenas proxies e a consulta delas só será executada no banco quando você tentar acessar a entidade.
Aqui tem uma publicação interessante sobre o assunto: O que é Lazy Loading e Eager Loading?


Given constrain $m=a_1>a_2>...>a_n$ and the elements are integer prove $\sum \frac{a_i-a_{i+1}}{a_i} < H_m$

Question: For decreasing positive integers $a_1>a_2>...>a_n>0$ when $a_1=m$.
Mark $a_{n+1}=0$,
Prove that $\sum_{k=1}^n \frac{a_i-a_{i+1}}{a_i} < H_m=\sum_{k=1}^m \frac{1}{k}$
Might add that $n$ can be chosen by us, as long as $a_1=m$.
I got the problem from a different problem and if I prove this I solved the problem, so I'm not sure if it's true.
I know the harmonic number is achieved if we choose $a_i=m-i+1$ i.e  $a_m=1<a_{m-1}=2<...<a_1=m$
My attemp was to prove that the question is equivalent to proving
$$\min \sum_{k=1}^n \frac{a_{i+1}}{a_i} $$ is achieved by $a_i=m-i+1$ i.e  $a_m=1<a_{m-1}=2<...<a_1=m$.
Showing the equivalent is simple, but proving that this is the best  I havn't manage to do.
Comment: Unless I am mistaken is $\sum_{i=1}^n \frac{a_i-a_{i+1}}{a_i} =\sum_{k=1}^m \frac{1}{k}$ if $n=m$ and $a_i = m+1-i$, so the inequality may not be strict.
Comment: For $a_i-a_{i+1}=d$ you can write $(a_i-a_{i+1})/a_i=\sum_{j=1}^d1/a_i$. Try using this for each contribution on the left and then compare it to the right. Work with examples. Feel free to ask if you're stuck.
Comment: do you mean $a_i-a_{i+1}=d_i$?, im not sure how you got $(a_i-a_{i+1})/a_i=\sum_{j=1}^d 1/a_i$
Comment: As I said, work with examples. If $a_1=4$ and $a_2=1$, then $(a_1-a_2)/a_1=1/4+1/4+1/4$. Now look at a few examples for your $a_1,\dots,a_n$, use this trick and then compare it to $H_m$.
Comment: I get $\sum_{j=1}^n \frac{d_j}{a_j}$, I tried this approach with sadly no success.

Is there another trick I'm missing to compare it to $H_m$?
Comment: @Itsme: There seems to be a typo in your question, $\sum_{k=1}^n \frac{a_i-a_{i+1}}{a_i}$ should probably be $\sum_{i=1}^n \frac{a_i-a_{i+1}}{a_i}$.
Answer: A formal proof is a bit challenging to write and to read, so here's an example.
Say we have $a_1=8$, $a_2=5$ and $a_3=1$. This gives us $m=8$ and $n=3$. So, we consider the sum \begin{align*}S&=\sum_{i=1}^{3}\frac{a_i-a_{i+1}}{a_i}\\&=\frac{8-5}8+\frac{5-1}5+\frac{1-0}1\\&=\frac38+\frac45+\frac11.\end{align*}
Now, we rewrite the fractions to obtain ones in the numerators, yielding
\begin{align*}S&=
\frac18+\frac18+\frac18+\frac15+\frac15+\frac15+\frac15+\frac11\\
&<\frac18+\frac17+\frac16+\frac15+\frac14+\frac13+\frac12+\frac11\\&=H_8.
\end{align*}
I hope this helps to establish the general case.
Comment: That demonstrates the idea nicely. For a formal proof of the general case you can write $$ \frac{a_i - a_{i+1}}{a_i} \le \sum_{k=a_{i+1}+1}^{a_i} \frac 1k $$ and then sum over $i$.
Answer: The idea has been nicely demonstrated in Matija's answer. Here is a formal proof for the general case.
For $1 \le i \le n$ and $a_{i+1}+1 < k \le a_i$ is $1/a_i \le 1/k$, so we have
$$
 \frac{a_i - a_{i+1}}{a_i} = \sum_{k=a_{i+1}+1}^{a_i} \frac 1{a_i} \le \sum_{k=a_{i+1}+1}^{a_i} \frac 1k \, ,
$$
with equality if $a_i - a_{i+1} = 1$, and strict inequality otherwise.
It follows that
$$
 \sum_{i=1}^n \frac{a_i - a_{i+1}}{a_i} \le \sum_{i=1}^n \sum_{k=a_{i+1}+1}^{a_i} \frac 1k
\overset{(*)}{=} \sum_{k=1}^m \frac 1k = H_m \, .
$$
$(*)$ holds because every integer $k$ between $1$ and $m$ is in exactly one half-open interval $(a_{i+1}, a_i]$.
Equality holds if $a_i - a_{i+1} = 1$ for all $i$, that is if $n = m$ and $a_i = m+1-i$ for $1 \le i \le m$. Otherwise the inequality is strict.


DataTable Join all Rows Data

Question: Actually,I have a DataTable containing 2000 rows with a ColumnName EMAIL.
Now,I want to show all emails from the DataTable and adding them to a TextBox by separating with a ,.
I am achieving this by a for loop,But,it is taking a long time.
Actually,I have used for getting the ColumnNames :
string[] columnNames1 = (from dc in table.Columns.Cast<DataColumn>()
                                select dc.ColumnName).ToArray();

Is there any thing like this,for joining all rows data to a single string ?
Answer: var text = string.Join(",", table.AsEnumerable()
                                 .Select(x=>x["EMAIL"].ToString())
                                 .ToArray());

Answer: suppose any one value is NULL means, Get the NULL Object reference error occurs
var text=string.Join(",", table.AsEnumerable().Select(x =>
x["EMAIL"].ToString()).ToArray());

so check the field value and after that you add the values
var text=string.Join(",", table.AsEnumerable().Select(x =>
((x["EMAIL"]) ? x["EMAIL"] : "").ToString()));

Comment: what is ColumnName here ?
Comment: ColumnName is your field name "EMail" @RealSteel

DataTable Join all Rows Data


Pass a preprocess variable to a template?

Question: In subtik.theme, I have:
/**
 * Implements template_preprocess_field().
 */
function subtik_preprocess_field(&$variables) {
  if ($variables['field_name'] == 'field_language') {
    $variables['language'] = $variables['element']['#object']->get('field_language')->getString();
    ksm($variables['language']);
  }
}

All good. It dumps 'php' to my page, which is correct.
I also have, in field--node--field-body.html.twig, I have:
...
{{ kint(language) }}
...

Not so good. It dumps 'null' to my page. Why not 'php'?
Answer: At a glance it looks like you have a condition checking for:
if ($variables['field_name'] == 'field_language') {
but the template you are using is field--node--field-body.html.twig instead of field--node--field-language.html.twig.
To get it to pass to field body twig, you would need:
  if ($variables['field_name'] == 'field_body') {
    $variables['language'] = $variables['element']['#object']->get('field_language')->getString();
  }

Comment: My brain is fried. I need to take a holiday :-) Thanks.


WP v4.7 REST API v2 endpoints stopped filtering

Question: After updating to WordPress 4.7 WP REST api v2 endpoints stopped filtering. Any one else experiencing this? Any solutions?
Answer: Yes, same here after updating to 4.7 alla endpoints stop working, with a little digging i found this, https://developer.wordpress.org/rest-api/reference/posts/
WordPress 4.7 comes with REST API endpoints for posts, comments, terms, users, meta, and settings.
So i change for example /wp-json/wp/v2/posts?filter[name]=something to /wp-json/wp/v2/posts?slug=something


$\mathbb{Q}(\sqrt{p^*})$ is the only quadratic extension of $\mathbb{Q}$ contained in $\mathbb{Q}(\zeta_{p^n})$.

Question:
Let $p$ be an odd prime and let $p^*=(-1)^{\frac{p-1}{2}}p$. Let $n\geqslant 1$. Show that $\mathbb{Q}(\sqrt{p^*})$ is the only quadratic extension of $\mathbb{Q}$ contained in $\mathbb{Q}(\zeta_{p^n})$. (Hint. Consider the Gauss sum $g=\sum_{a=1}^{p-1}\left(\frac{a}{p}\right)\zeta_p^a$, where $\left(\frac{a}{p}\right)$ is the Legendre symbol.

My try: Since $g^2=\left(\frac{-1}{p}\right)p=p^*$, we have $\sqrt{p^*}=g\in\mathbb{Q}(\zeta_p, \zeta_p^2, \dotsc, \zeta_p^{p-1})=\mathbb{Q}(\zeta_p)$. I want to know what does $\mathbb{Q}(\zeta_{p^n})$ do and how to prove the uniqueness?
Any hints are appreciated, thanks in advance.
Comment: @DietrichBurde I think we only get other 2-torsion in the Galois group of $\Bbb{Q}(\zeta_{m})$ iff $m$ has more odd prime factors or is a power of two $>4$. When $p$ is an odd prime $Gal(\Bbb{Q}(\zeta_{p^n})/\Bbb{Q})$ is cyclic of order $(p-1)p^n$, so has a unique element of order two. And, more relevantly, a  unique subgroup of index two.
Comment: For $n=1$ see [here](https://math.stackexchange.com/questions/30111/unique-quadratic-subfield-of-mathbbq-zeta-p-is-mathbbq-sqrtp-if?rq=1).
Comment: Is there an essential difference between "case $\mathbb{Q}(\zeta_{p})$" and "case $\mathbb{Q}(\zeta_{p^n})$"?
Comment: Maybe it is a printing mistake. Could you please give a more detailed explanation?
Comment: Thanks for your help! I still need some time...
Answer: One could note that there is a counterexample for $p=2$ and, for example, $n=4$. Then $\Bbb Q(\zeta_{p^n})$ contains three different quadratic subfields:
Determining subfields of cyclotomic field extension
Otherwise the above answer and the comment by Jyrki show that the claim is true for $p>2$ and $n\ge 1$.
Answer: The Galois group of $\mathbb{Q}(\zeta_{p^n})/ \mathbb{Q}$ is isomorphic to $(\mathbb{Z}/p^n\mathbb{Z})^\times$, which is cyclic for all $n\geq 1$ since $p$ is odd.
In particular, it has a unique subgroup of order $\varphi(p^n)/2$, hence the Galois extension $\mathbb{Q}(\zeta_{p^n})/\mathbb{Q}$ has a unique quadratic subextension, by Galois correspondence.
Comment: Thanks! Could you please help to explain the DietrichBurde's comment?


How to simulate more data for machine learning?

Question: I am attempting to analyze a small dataset using machine learning (SVM, binary problem). There are $103$ samples and $215$ variables (all variables are real numbers). Some of the variables (around half) are moderately/highly correlated with each other.
Knowing the empirical distribution of each variable, would it be somehow possible to generate more data by a simulation? I guess I should do it only for the training data. I am using nested cross-validation for picking parameters / estimating performance if it matters.
Comment: If you already knew the joint distribution of Y and X, why would you want to fit a SVM?
Comment: Generate more data for what purpose?
Comment: I don't think you need more data, really. SVMs have good regularization, you should be safe. Also something like Attribute Bagging is worth a try.
Answer: You can ignore some of the correlated attributes and use Cross validation, instead of "generating" more data. Cross validation is a common approach in situations where you have more attributes than observations.
Answer: Similar to what stan0 said, I would use cross validation, but more specifically repeated k-fold Cross Validation. For a 'smaller' data set of 103 by 215, I would perhaps use 10x10 (10 repeats of 10 fold cross validation, so 100 model fits in total). If you're using R, I would highly suggest the caret package which is able to handle this for you using the trainControl() function.
Good resources for the underlying funciton calls and algorithum:
https://topepo.github.io/caret/training.html
https://topepo.github.io/caret/splitting.html
Answer:
Knowing the empirical distribution of each variable, would it be
  somehow possible to generate more data by a simulation?

Yes, you can always generate more data, the question is of its quality! If you know the empirical distributions of the individual variables, this is not enough to generate more data. If you generated data on each of the variables independently, then you would end up with dataset where the features are unrelated to themselves and your target variable. So if your purpose is to check how does your model behaves with the data where the variables are unrelated to your target variables (i.e. how much is it prone to produce false positives), then this would help. Otherwise, such approach does not make sense. As others said, $k$-fold cross-validation and similar methods may help if size of your sample is limited (see also questions tagged as small-sample for other suggestions).
However, there are situations where you can produce artificial data for your model. It can be done, and often is done, in image processing where you can start with small dataset and then add noise to it (rotate images, add white noise, mix them with other pictures, change the backgrounds, make them blurry, change their sizes, resolution etc.). Unfortunately, this does not easily translate to other kinds of data.


Is it possible to expand $\sin(2x+1)\cdot\sin(2x+1)$?

Question: Is it possible to treat it as a binomial?
Comment: What about $\sin^2(2x+1)$?
Comment: I need to expand if it's even possible.
Comment: Would you accept $\sin^2(2x + 1) = 1 - \cos^2\left(2x + 1\right)$ ?
Comment: Yeah. I'm just trying to find the derivative of sine using limit defination. I'm also a bit curious to know where you derived cos2x=cossquared x - sinessquared x.
Comment: @KapookyHandy: It can be easily derived using $\cos(A+B) = \cos A\cos B - \sin A\sin B$. Put $\theta = A = B$ . Then use $\cos^2x = 1 - \sin^2x$. Presto, you've derived an equation for $\cos2\theta$, I've put a link to a video in my answer which will hopefully be helpful.
Answer: We know that $\cos2\theta = 1 - 2\sin^2\theta$ $\implies \sin^2\theta = \frac{1}{2}\left(1-\cos2\theta\right)$
Putting $\theta = 2x + 1$,
$$
\sin^2(2x + 1) = \frac{1}{2}\Big( 1 - \cos(4x + 2) \Big)
$$
Comment: That was awesome. Trig is by far my favorite subject in math by a long shot!


MongooseIM Push Notifications for MUC`s messages (not MUC light)

Question: I was trying to use MongooseIM with mod_muc (just mod_muc, not mod_muc_light) and found that users do not receive Push Notifications for messages that they could miss in MUC room while being offline.
I am using MongooseIM with mod_event_pusher_push_plugin_defaults and MongoosePush service. Push Notification works fine for MUC light but doesn`t work for MUC messages. Is it possible to configure MongooseIM to send Push Notifications to users for missed MUC messages?  Or how can I implement it? Please, help me
Answer: Currently it's not possible to have push notifications for the regular MUC in MongooseIM. What do you need from MUC that you couldn't use MUC light?
Comment: Thanks for the answer! I need Moderated Room when only some members are allowed to send messages to all occupants
Comment: I read information about "XEP-0045: Multi-User Chat" and there are a lot of features with roles and permissions that is necessary for me. Does mongooseim support it?
Comment: Yes, MongooseIM supports XEP-045. It only does not provide push notifications integration for the regular MUC
Comment: Four years later, is this still true?


Show translated content in view listing page when accessing translated site

Question: I have a view that filters for a specific content type (accommodation) and shows a paginated list for the user.
The site has a second language (Welsh) and a few of the content nodes have Welsh translations.
In theory, I should be able to visit the accommodation view, and if I was on the Welsh site (http://www.example.com/welsh/accommodation) I should see the Welsh nodes listed.
Sadly, this is not the case exactly.
My view is able to filter the Welsh translated nodes, but the view is spitting out the English text (title, URL, description etc) and not the Welsh versions.
This makes it pretty pointless, especially the fact that the URLs are rendered as the 'normal' English URLs and do not point to the Welsh site.
I've tried a tons of various view options but cannot fathom how to render the translated information.
Answer: Have you added a filter to the view for the translation language?
You should be able to set one to only show content in the language of the current page (Which should be welsh if your language detection is setup and working correctly)
Comment: I do have a filter for language added - 'Content: Translation language (= Content language selected for page)'. Tried various seetings, still only seeing the English versions of the Welsh entries that should be showing.
Comment: I'm sure my language detection is working, as I can visit /welsh and get the interface in welsh, just not the view listings :/
Comment: That should be all you need. Do your accommodation nodes definitely have languages set and aren't language neutral?
Comment: They definitely have translations (maybe half do to actually, not 100% of them). If I visit the Welsh view, I get the right number of items listed (the nes that are translated) but it shows the English title, URL etc.
Comment: Do I have to have separate displays for the two languages?
Comment: What would you do differently in the 2 displays as you can only apply the filter like you already have. You could try changing the filter to specifically only show welsh. If that doesn't work and only show welsh translated items then im not sure what's gone wrong
Comment: Even changing the language filter to welsh, the correct nodes are shown, but the actual fields brought through are not the translated fields :(
Comment: What sort of translation have you got? Have you setup individual fields to be translatable? Or have you set the whole node to be translated? The whole node translated is the way to do it rather than individual fields
Comment: The whole node is set to be translated. We have investigated, and whilst it looks like the View is doing the correct query and getting the correct node IDs based on language, the next query along is clearly fetching the default language node, English. Currently trying to locate this query.
Answer: Turns out is was Panelizer causing issues. The site was using v3.0-beta, and Panelizer is currently on v4.0.
I also found a ticket about Panelizer not supporting translations in older versions.
For now, I have removed Panelizer and I am now able to get the correct results.


VBA: Select rows below cell with specific value

Question: I'm trying to select a set of rows below the cell with the text "City"  and copy and paste them in different columns. Once the word "City" is found then copy all the next rows until the word Bird_type is found, copy and paste the rows and repeat the steps for all the cities in column A, something like this

I tried to code it in VBA but I'm only getting the first found row:
Sub test()

Dim city_find As String
Dim result As Integer
result = 2

Dim i As Integer

a = Worksheets("sheet1").Cells(Rows.Count, 1).End(xlUp).Row

For i = 2 To a

    If Worksheets("sheet1").Cells(i, 1).Value = "city" Then

      Range(Cells(i + 1, 1), Cells(i + 1, 6)).Copy

      Range("E100").End(xlUp).Offset(1, 0).PasteSpecial xlPasteFormulasAndNumberFormats

       result = result + 1

    End If

Next i

End Sub

How can I do?
Answer: I will use VBA .Find():
Sub Calc()
  Dim s As Range, e As Range, nS As String
  Set s = ActiveSheet.UsedRange.Find("City")
  If Not s Is Nothing Then
    Set e = ActiveSheet.UsedRange.Find("City",s)
    If Not e Is Nothing Then
      Range(s,e).EntireRow.Copy
    End If
  End If
  Range("E100").End(xlUp).Offset(1, 0).PasteSpecial xlPasteFormulasAndNumberFormats
End Sub


Deploy de Aplicação Django com Apache

Question: fiz uma aplicação Django e estou fazendo uns testes de deploy em uma máquina virtual com Ubuntu Server 18.04, antes de partir para o deploy em um VPS. Fiz as configurações corretamente, mas estou tendo um probleminha o Django não está reconhecendo os templates que estão na pasta templates. Alguém já teve esse problema?
Comment: E o template `blog/home` existe?
Comment: existe, ele está na pasta templates
Comment: Ele parece estar tentando carregar em `django/contrib/auth/templates/blog/home.html` dentro do Django instalado. Poderia [edit] a pergunta e adicionar um [mcve] que reproduza o problema?
Answer: No seu settings.py, na variável "TEMPLATES", faça o seguinte:
<code>TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]
</code>
Tive o mesmo problema utilizando o apache em um servidor dedicado na Hostdime e em um vps na Kinghost, resolvi assim.
Comment: deu certo aqui, muito obrigado ^-^

Deploy de Aplicação Django com Apache


What does RW mean in the FLIR Spinnaker software?

Question: In the FLIR documentation of PySpin/Spinnaker there is no reference to PySpin.RW. What does it mean?
It is used as follows:
if cam.TriggerMode.GetAccessMode() != PySpin.RW:
Obviously its there for test/comparision reason.
Answer: From the FLIR website I can find the following here:
Read-only nodes
For some features, the node is read-only and cannot be unlocked under any circumstances. The node is provided for information purposes.
To determine if a node is read-only:
In SpinView, double-click on the feature to open the Node Information window. Look for the AccessMode row.

Access    Description

RO     Read-only
RW     Read / Write
N/A    Not Available

Using Spinnaker API, you can poll the Access Mode with the following:
//! Checks if a node is readable
inline bool IsReadable (const IBase* p)
{
   return (p != NULL) && IsReadable (p->GetAccessMode());
}


B2C - Unable to output custom attribute in SAML response

Question: I'm trying to connect to a SAML SP, who require us to send them 4 claims. 3 of them are standard claims (surname, email, and objectId) but with different names, which I've been able to do just by using the OutputClaim's PartnerClaimType attribute in the RelyingParty. The other claim is one I've had to create a custom attribute for in the B2C portal, which I've have named employeeId, and the SP needs it to be called employeenumber.
I've read that custom attributes need to be preceeded with extension_, so I have added the following to our custom polcies:
SignUpOrSignInSaml:
<OutputClaim ClaimTypeReferenceId="extension_employeeId" PartnerClaimType="employeenumber" />

TrustFrameworkBase:
  <ClaimType Id="extension_employeeId">
    <DisplayName>EmployeeID</DisplayName>
    <DataType>string</DataType>
    <UserHelpText>Used when authenticating using SAML</UserHelpText>
  </ClaimType>

I've added the below into the AAD-UserReadUsingObjectId technical profile.
<OutputClaim ClaimTypeReferenceId="extension_employeeId" />

I've been using Microsoft's SAML test app to see what's in the response. The first 3 claims are fine, but the custom attribute never. If I give it a default value then it is in the response, so I guess for whatever reason the actual value, which I've only setup on one user account so far, cannot be found. Is there something that I'm missing?
Comment: Did you enable extension attribute support? https://learn.microsoft.com/en-us/azure/active-directory-b2c/user-flow-custom-attributes?pivots=b2c-custom-policy#modify-your-custom-policy
Comment: @JasSuri-MSFT I do have the section to enable custom attributes in the policy, although after checking it again it wasn't quite right. However, after correcting it it's still not working.
Comment: Last thing to check is: the extension attribute you are targeting, does it contain the same GUID as the AAD-Common ApplicationId GUID?
Comment: @JasSuri-MSFT I don't understand. Are you asking if the custom attribute name in the custom policy contains the same GUID?
Comment: Not in the custom policy, but the raw attribute name on the user object as returned by graph api - does that raw attribute name contain the same GUID as the one you've put in AAD-Common. It needs to match.
Comment: @JasSuri-MSFT Sorry, my mistake. The employeeId attribute we're trying to use is actually one that's already built into B2C. So I've created this as a new claim in our base policy and also added it as an output claim in the AAD-UserReadUsingObjectId technical profile, which uploads fine. I've added it to our signupsignin policy as an output claim, but when I upload it I receive this error - "Output Claim 'employeeId' is not supported in Azure Active Directory Provider technical profile 'AAD-UserReadUsingObjectId' of policy B2C_1A_signup_signin_saml". Is there something else I need to do?
Comment: You cannot use the built in employeeId attribute. Supported list is here https://learn.microsoft.com/en-us/azure/active-directory-b2c/user-profile-attributes
Comment: @JasSuri-MSFT OK, so we've created a new attribute called EmployeeNumber using the Graph API, so the full name is extension__EmployeeNumber. Using Graph Explorer this is visible amongst all the data and has the correct value we set for it. The custom policies have been updated to use extension_EmployeeNumber, but when using the Microsoft test SAML app the attribute still isn't in the response.
Answer: I finally managed to get this working by creating a new Technical Profile to output the extension_EmployeeNumber. This Technical Profile is referenced by a ValidationTechnicalProfile inside the existing SelfAsserted-LocalAccountSignin-Email Technical Profile, which also has an OutputClaim for the extension_EmployeeNumber.


Expose custom DateTime property to user control

Question: I am developing a calendar where my days are aligned like this
<code>    <Label Grid.Row="1" Grid.Column="1" Content="{Binding Path=SundayLabel, FallbackValue=Sunday}" Style="{StaticResource ResourceKey=DayLabelStyle}" />
    <custom:DayDisplay Grid.Row="2" Grid.Column="1" x:Name="SundayDisplay" />

    <Label Grid.Row="1" Grid.Column="2" Content="{Binding Path=MondayLabel, FallbackValue=Monday}"  Style="{StaticResource ResourceKey=DayLabelStyle}" />
    <custom:DayDisplay Grid.Row="2" Grid.Column="2" x:Name="MondayDisplay" />
</code>
and so forth. DayDisplay is a custom WPF control.
The control containing all theese DayDisplays needs some way to set the individual days with a DateTime preferably from code behind.
Is there some way to expose a custom property in my DayDisplay view so i could do something like:
<code>    <custom:DayDisplay Grid.Row="2" Grid.Column="2" x:Name="MondayDisplay" x:Day="{MondayDateTime}"/>
</code>
Preferably i would like to store all custom:DayDisplay pairs in a Dictionary and have them autogenerated also if it is possible. Any help is greatly appreciated :)
Thanks,
Niklas
Answer: you should expose a DateTime dependency property on the control, done like so:
<code>    public DateTime Day
    {
        get { return (DateTime)GetValue(DayProperty); }
        set { SetValue(DayProperty, value); }
    }

    // Using a DependencyProperty as the backing store for Day.  This enables animation, styling, binding, etc...
    public static readonly DependencyProperty DayProperty =
        DependencyProperty.Register("Day", typeof(DateTime), typeof(DayOfWeek), new UIPropertyMetadata(DateTime.MinValue));
</code>
if you want this to be auto generated, what you should do is have an ItemsControl that contains this dayControl of yours, like so:
<code>    <ItemsControl x:Name="DaysItemsControl">
        <ItemsControl.ItemTemplate>
            <DataTemplate>
                <custom:DayDisplay Day="{Binding}"/>
            </DataTemplate>
        </ItemsControl.ItemTemplate>
    </ItemsControl>
</code>
and then DataBind it from code behind:
<code>        DateTime[] week = new DateTime[] { new DateTime(2000,1,1), new DateTime(200,1,2) };
        DayItemsControl.ItemsSource = week;
</code>
Comment: This is perfekt. Thank you very much :D One follow question. I want this change to be propagated to my DataContext on my DayView. Is there some way to constrain this by using a OneWay binding? Thanks
Comment: I couldn't understand what it is you want to do, can u explain more in depth? (also, mark question as answered :)
Comment: I have a CalendarView containing a list of DayViews (Thanks to you :D). The CalendarView sets the day of the DayView with a binding like . Now i want to be able to change this day from CalendarView code behind and have that change trigger an property update on my DayView.DataContext. Like if i have a Property, CurrentDay, in DayViewModel I want this to update when changing the Day DependencyProperty. I guess i could make a new question of this :) Thanks. I will mark this as the answer. I just wanted to wait a bit to see if i got more answers :)