@Async not working in Spring API rest with Interfaces

Question: I'm working with @Async to stored some data in parallel in the database with hibernate. I need to do that because before saving the information to the database I need to run some task that takes several minutes. So I implemented @Async.
The issue is that @Async seems to not be working. Please find the code below:
WebConfig
@Configuration
@EnableAsync
@EnableWebMvc
public class WebConfig extends WebMvcConfigurerAdapter {

}

StudentServiceImpl:
@Autowired
RunSomeTaskService runSomeTaskService;

@Override
Transactional
public Response saveWithoutWaiting(StudentBO[] students, String username) throws Exception {
...
for (StudentBO student : students) {
    ....
    Future<Response> response = runSomeTaskService.doTasks(student);
    //Finish without waiting for doTasks().
}

@Override
    Transactional
    public Response saveWithWaiting(StudentBO[] students, String username) throws Exception {
    ...
    for (StudentBO student : students) {
        ....
        Future<Response> response = runSomeTaskService.doTasks(student);
        //Finish and wait for doTasks().
        response.get();
    }

RunSomeTaskService:
public interface RunSomeTaskService{
    @Async
    public Future<Response> doTasks(Student student);
}

RunSomeTaskServiceImpl:
public class RunSomeTaskServiceImpl extends CommonService implements RunSomeTaskService{

Student student;
@Override
    public Future<Response> doTasks(Student student) {
          Response response = new Response();
          this.student = student;
          //do Task
          return new AsyncResult<Response>(response);
       }
}

web.xml
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
          http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
    version="3.0">

    <display-name>Sample Spring Maven Project</display-name>

    <servlet>
        <servlet-name>mvc-dispatcher</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>/WEB-INF/spring-config.xml</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
        <async-supported>true</async-supported>
    </servlet>

    <servlet-mapping>
        <servlet-name>mvc-dispatcher</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>

<filter>
    <filter-name>encodingFilter</filter-name>
    <filter-class>
            org.springframework.web.filter.CharacterEncodingFilter
        </filter-class>
    <init-param>
      <param-name>encoding</param-name>
      <param-value>UTF-8</param-value>
    </init-param>
  </filter>
  <filter-mapping>
    <filter-name>encodingFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>
  <filter>
    <filter-name>jwtTokenAuthFilter</filter-name>
    <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
  </filter>
  <filter-mapping>
    <filter-name>jwtTokenAuthFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>
</web-app>

spring.config.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
    xmlns:util="http://www.springframework.org/schema/util"
    xmlns:mvc="http://www.springframework.org/schema/mvc"
    xmlns:tx="http://www.springframework.org/schema/tx"
    xsi:schemaLocation="http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc.xsd
  http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.2.xsd
  http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
  http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd">

    <context:annotation-config  />
    <context:component-scan base-package="com.app.controller" />
    <tx:annotation-driven transaction-manager="transactionManager"/>
    <mvc:annotation-driven />

    <bean id="dataSource"
        class="org.springframework.jdbc.datasource.DriverManagerDataSource">
        ...
    </bean>

    <bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">
...
    </bean>

    <bean id="sessionFactory"
        class="org.springframework.orm.hibernate5.LocalSessionFactoryBean">
        <property name="dataSource" ref="dataSource" />
        <property name="annotatedClasses">
            <list>
                <value>//every model generated with Hibernate</value>
            </list>
        </property>
        <property name="hibernateProperties">
            <props>
                <prop key="hibernate.dialect">org.hibernate.dialect.MySQL5Dialect</prop>
                <prop key="hibernate.show_sql">true</prop>
            </props>
        </property>
    </bean>

    <bean id="transactionManager"
        class="org.springframework.orm.hibernate5.HibernateTransactionManager">
        <property name="sessionFactory" ref="sessionFactory" />
    </bean>

    <bean id="persistenceExceptionTranslationPostProcessor"
        class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor" />

    <bean id="studentService" class="com.app.services.StudentServiceImpl"></bean>
    <bean id="studentDao" class="com.app.dao.StudentDaoImpl"></bean>
    ...

    <bean id="jwtTokenAuthFilter" class="com.app.security.JWTTokenAuthFilter" />
</beans>

So, could you please help me to understand why @Async is not working?
UPDATE: @Async is now working, but I'm not getting the expected results.
For the case, that I have to wait for the result (sync case)  CompletableFuture.get() is not waiting for the response and I'm getting and error:
My code:
CompletableFuture<Response> res = extractDataService.doTask(student);

The error:
org.hibernate.HibernateException: Illegal attempt to associate a collection with two open sessions. Collection : <unknown>
Collection contents: [[]]
        at org.hibernate.collection.internal.AbstractPersistentCollection.setCurrentSession(AbstractPersistentCollection.java:627)
        at org.hibernate.event.internal.OnUpdateVisitor.processCollection(OnUpdateVisitor.java:46)
        at org.hibernate.event.internal.AbstractVisitor.processValue(AbstractVisitor.java:104)
        at org.hibernate.event.internal.AbstractVisitor.processValue(AbstractVisitor.java:65)
        at org.hibernate.event.internal.AbstractVisitor.processEntityPropertyValues(AbstractVisitor.java:59)
        at org.hibernate.event.internal.AbstractVisitor.process(AbstractVisitor.java:126)
        at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.performUpdate(DefaultSaveOrUpdateEventListener.java:293)
        at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.entityIsDetached(DefaultSaveOrUpdateEventListener.java:227)
        at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:92)
        at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:73)
        at org.hibernate.internal.SessionImpl.fireSaveOrUpdate(SessionImpl.java:648)
        at org.hibernate.internal.SessionImpl.saveOrUpdate(SessionImpl.java:640)
        at org.hibernate.internal.SessionImpl.saveOrUpdate(SessionImpl.java:635)
        at com.app.dao.CommonDaoImpl.addOrUpdate(CommonDaoImpl.java:28)
        at com.app.services.ExtractDataServiceImpl.doExtraction(ExtractDataServiceImpl.java:361)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:108)
        at org.springframework.aop.interceptor.AsyncExecutionAspectSupport$CompletableFutureDelegate$1.get(AsyncExecutionAspectSupport.java:237)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

When I don't have to wait and I execute several cases in Async way I get:
12:17:44.040 [DEMO-4] DEBUG o.h.r.t.b.j.i.JdbcResourceLocalTransactionCoordinatorImpl - JDBC transaction marked for rollback-only (exception provided for stack trace)
java.lang.Exception: exception just for purpose of providing stack trace
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.markRollbackOnly(JdbcResourceLocalTransactionCoordinatorImpl.java:265) [hibernate-core-5.0.6.Final.jar:5.0.6.Final]
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:156) [hibernate-core-5.0.6.Final.jar:5.0.6.Final]
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$100(JdbcResourceLocalTransactionCoordinatorImpl.java:38) [hibernate-core-5.0.6.Final.jar:5.0.6.Final]
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:231) [hibernate-core-5.0.6.Final.jar:5.0.6.Final]
        at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:65) [hibernate-core-5.0.6.Final.jar:5.0.6.Final]
        at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:581) [spring-orm-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:108) [spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.aop.interceptor.AsyncExecutionAspectSupport$CompletableFutureDelegate$1.get(AsyncExecutionAspectSupport.java:237) [spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
12:17:44.043 [DEMO-4] DEBUG o.s.o.h.HibernateTransactionManager - Initiating transaction rollback after commit exception
org.hibernate.AssertionFailure: null id in com.app.model.FieldValue entry (don't flush the Session after an exception occurs)

Comment: Could you please add log with thread name in `doTasks` and `saveWithoutWaiting` method, just to check if they are called in the same thread. And are you sure the `WebConfig ` gets scanned when you start your application?
Comment: `@Async` is working perfectly fine actually. The problem is you haven't specified a specific `TaskExecutor` to launch your threads with. Also async request processing (`true`) is something then async executing a method. Those are different beasts. Also your implementation of `RunSomeTaskServiceImpl` is flawed as it isn't thread safe. Another question next to the `DispatcherServlet` do you have a `ContextLoaderListener` which is loading th services? As that would make your `@EnableAsync` pretty much useless as that lives in a different context.
Comment: What is this line `future.get()` ?
Comment: @ErvinSzilagyi as far as I understand is to force to wait for the task to be completed... But I'm using it just in the saveWithWaiting mehtod.
Comment: Yes, but `future` is not declared in your code, ignoring the syntax error after.
Comment: Also, it is enough to have `@Async` only for the interface. Btw, how can you tell that it is not working?
Comment: @ErvinSzilagyi I fixed the errors in the text (anyway in the app was ok). I removed the Async from the implementation and I still get the same result.
How I know that it is not working, because the process takes some minutes, so my intention is that the API rest generate a response to the user without waiting for that process, but the app wait those minutes and when all the process are done, the API generate the response. Also debugging, It didn't move from doTasks line to the next one until doTasks is complete.
Comment: `@Async` isn't tagged with `@Inherited` it needs to be placed on the method that needs to be async. It also needs to avoid any proxied methods .
Comment: @DarrenForsythe could you please explain? Where should I add Inherited? In the interface or in the implementation? Also, how can I check if the method is proxied or not?
Comment: `get()` is a blocking method, when you call it in a loop it will wait until the current `Future` is completed and moves to the next one.
Comment: @ErvinSzilagyi But in saveWithoutWaiting I'm not calling get() and it block it anyway
Comment: I made a mistake for the `@Async`, as Darren Forsythe said, it has to be on the implementation, not on the interface. I don't see anything other wrong with your code.
Comment: MM but it keeps without working... maybe something in the spring-config.xml ?
Comment: in saveWithoutWaiting how do you deal with future responses?
Comment: @Rohit I don't deal with it.. I don't need to know which is the response, for that reason I'm interested in using something really Async
Comment: Is it possible to share project on github?I suspect there may be problem in code other than the code shared here. @grape_mao is correct, debugging can be misleading in multithreaded app.
Comment: @grape_mao I printed 'Thread.currentThread().getName()' inside the Async function and I everytime I got http-nio-8080-exec-3 as Thread Name. I think that WebConfig is scanned because when I did what Faiz in his answer suggested, I get and error because the context was declared twice.
Comment: @Rohit the code is from a medium/large project.. I cannot share.. for that reason I shared the minify version of the classes... But there I'm doing some image processing and OCR in Command Line and saving the result in MySQL with Hibernate. When I use Thread directly, it works fine (works in ASync way) but I cannot stored objects in the database in parallel.
Comment: @Faabass better add log in `saveWithoutWaiting` as well, otherwise we cannot compare. And the answer provided by Faiz is in conflict with your web.xml, it has nothing to do with `WebConfig`. Do make sure your configuration class is scanned by Spring.
Comment: @grape_mao When I do what Faiz said, I get java.lang.IllegalStateException: No unique WebApplicationContext found: more than one DispatcherServlet registered with publishContext=true?
Comment: @Faabass can you share your spring-config.xml file?
Comment: @Rohit done! could you please take a look at it?
Comment: As far as I know, asynchronous REST API calls can only be implemented with Jersey: https://jersey.github.io/documentation/latest/async.html. I have implemented these with Spring boot + Jersey.
Comment: @Async should be on the impl , not on the interface , that will solve it, and remember it will only happen when u will call this method using the framwork .
Comment: The latest exception stack looks like transaction boundary and cascade issue. We would need to see the `Student` entity and what you do in the task. I would again ask to create a new question rather than updating this question.
Comment: One quick thing to remember, transactions don't move from thread to thread. If you are taking this async, you need to be careful with your use of Cascade. As @Rohit mentioned, Please present the entity in your post.
Comment: Thanks! I created a new post https://stackoverflow.com/questions/51831843/hibernate-two-open-sessions-when-working-with-future-and-aync
Answer: Just use:
servlet.setAsyncSupported(true);

For Example
public class WebAppInitializer implements WebApplicationInitializer {
    @Override
    public void onStartup(ServletContext servletContext) throws ServletException {
        AnnotationConfigWebApplicationContext ctx = new AnnotationConfigWebApplicationContext();
        ctx.register(WebConfig.class);
        ctx.setServletContext(servletContext);
        ServletRegistration.Dynamic servlet = servletContext.addServlet("dispatcher",
            new DispatcherServlet(ctx));
        servlet.setLoadOnStartup(1);
        servlet.addMapping("/");
        servlet.setAsyncSupported(true); //Servlets were marked as supporting async
        // For CORS Pre Filght Request
        servlet.setInitParameter("dispatchOptionsRequest", "true");
    }
}

Comment: I implemented that in WebConfig.java and I got and error saying that I have two ApplicationContext.. So, then I found that in the web.xml I have to add true for doing the same config that you suggested, but it's also not working... I added the definition of web.xml to the post so you can check
Comment: This is the error I get
java.lang.IllegalStateException: No unique WebApplicationContext found: more than one DispatcherServlet registered with publishContext=true?
Answer: More Sophisticated way would be to implement AsyncConfigurer and set the AsyncExecutor to threadPoolTaskExecutor.
Sample Code below
@Configuration
@EnableAsync(proxyTargetClass=true) //detects @Async annotation
public class AsyncConfig implements AsyncConfigurer {

 public Executor threadPoolTaskExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(10); // create 10 Threads at the time of initialization
        executor.setQueueCapacity(10); // queue capacity
        executor.setMaxPoolSize(25); // if queue is full, then it will create new thread and go till 25
        executor.setThreadNamePrefix("DEMO-");
        executor.initialize();//Set up the ExecutorService.
        return executor;
    }

    @Override
    public Executor getAsyncExecutor() {
        return threadPoolTaskExecutor();
    }

    @Override
    public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
        return new YOUR_CUSTOM_EXCEPTION_HANDLER();
    }

}

The above configuration will detect @Async annotation wherever mentioned
Comment: This was the issue... I created a new class in the same package than WebConfig and added the package in the base-package, but I still keep getting error for the database session... I got that the session is duplicated and things like that...
Comment: Is it possible for you to share the stack trace? Thanks.
Comment: Sure! added.. I getting hopeless about this. What I'm doing is I'm doing some process and then I'm updating those students in the database in parallel way
Comment: This make the app to execute the methos in async way!
Answer: Well, finally I make it work...
I used Executors in the following way:
ExecutorService executor = Executors.newFixedThreadPool(students.size());
for (StudentBO student : students) {
    executor.submit(() -> extractDataService.doTask(student));
}

Where doTask is a regular function, that when I don't need it to work in a different thread, I just call it as it is. When I need the threads, I use the code above.
Comment: The code is dangerous as you are creating thread pools without ever cleaning them up. Eventually you will run out of resources and a lot of idle threads. If you want this you have to `shutdown` the `executor` in a `finally` block. Also your want to limit the number of threads and don't want to create 100 threads if you only have 16 cores in your system being able to do the work.
Comment: This is only a workaround. Of cause using your own executor will do the work.
Comment: With this you do not need `@Async`.
Comment: @M.Deinum and the finally will be fired AFTER all the threads are done? Or how do I make sure they are completed before triggering the finally block?
Comment: The `shutdown` closes as soon as there are no more tasks to process.
Answer: There is possibility that the @EnableAsync annotation in WebConfig.java is never scanned. The web.xml points to the spring-context.xml.
You can change the DispatcherServlet definition in web.xml to:
<servlet>
    <servlet-name>mvc-dispatcher</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
    <init-param>
        <param-name>contextClass</param-name>
        <param-value>
            org.springframework.web.context.support.AnnotationConfigWebApplicationContext
        </param-value>
    </init-param>
    <init-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>
            com.yourpackage.WebConfig
        </param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
    <async-supported>true</async-supported>
</servlet>

And include all configuration from spring-config.xml to this class.
Or
Add <task:annotation-driven> in spring-config.xml.
Updated
Currently, com.app.controller package is scanned in spring-config.xml. Make sure the WebConfig.java is in this package or one of it's sub-package. If not add WebConfig's package to base package attribute separated by comma.
Additionally, you can control the thread pool used by async task. Create a executor bean
@Bean
public Executor asyncTaskExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(5);
    executor.setMaxPoolSize(10);
    executor.setThreadNamePrefix("asynctaskpool-");
    executor.initialize();
    return executor;
}

And in your async method use the bean name like this
@Async("asyncTaskExecutor")
public Future<Response> doTasks(Student student);

This will ensure all task will be executed in this thread pool.
Comment: I added the spring-config.xml could you please take a look and share your opinion?
Comment: @Faabass Is your WebConfig.java in the package com.app.controller or it's subpackage? If not then it is not scanned and it's package needs to be added to base packages. Base-package attribute can have multiple comma separated packages like `base-package="x.y.z.service, x.y.z.controller"`.
Comment: Thank! I added that to the base-package and now it is working in @Async, anyway I0m having other issues. please see my post
Comment: I would strongly suggest to ask a new question and mark this as answer for the original question.
Comment: done! https://stackoverflow.com/questions/51831843/hibernate-two-open-sessions-when-working-with-future-and-aync
Answer: You can do CompletableFuture , with this you know when all your tasks are complete
List<CompletableFuture<T>> futureList = new ArrayList<>();

for(Student student:studentList){

CompletableFuture<T> returnedFuture = CompletableFuture.supplyAsync(() -> doSomething(student),executor).exceptionally(e -> {
        log.error("Error occured in print something future",e);
        return 0;
    });

futureList.add(returnedFuture);
}

Completable.allOf(futureList);

Then you can pipeline with thenCompose or thenApply (to take consumer) to have complete control on the task pipeline. you can shutdonw executors when you are done safely.
CompletetableFuture.allOff javadoc for more info
Comment: I updated the code to use CompletableFuture.. but I'm getting errors with the hibernate session, I added the stracktrace in my post


I can hear the melody, but I can't hear the chords (Lift Every Voice example)

Question: I have been charting chords to gospel songs lately. I did fine on a few but I came up short on "Lift Every Voice" by James Weldon Johnson, version in the key of G. The intro lick is a good example: I hear a walk up of "G A G B, B A G A, G G" (emphasis on what I would consider the chord tones). These are the "loud" notes to my ear. However, the chords to the progression are obviously not what I emphasized in bold, when I play along with the recording. The closest I have gotten to these chords is G/B, E/A, G, but that still feels wrong. The verse is the same (taking the first line for example): "Lift ev - 'ry voice and sing / 'til earth and hea - ven ring", I hear as the melody of "F# G A B B B / B C D B G A", but there is definitely something else going on with the chords than what I emphasized in bold here as my instinct on the chord choices. I just can't seem to hear through the mental "mud" to discover it. Any tips to training my ear for chord roots is really appreciated. Since this song has challenged me the most to date, specific advice pertaining to it is likely to help a lot.
Comment: There's a vote to close, but I think this is on topic. It's not asking users to provide the chords to Lift Every Voice. (Those chords are readily available online.) The question is asking for tips on transcribing chords. Some answers to [**this question**](https://music.stackexchange.com/questions/9806/trying-to-find-chord-progressions-of-a-song-how/9843#9843) might be helpful.
Comment: Thank you. You're right, my question is generally about how to identify chords when transcribing a song. Yesterday I had a small breakthrough by realizing that if I go down from the note I first hear by 5 or 8 semitones, I usually find the root. But this only works about half of the time. If the answer is just lots of ear training, I've taken up a software that challenges users to identify chords, hoping that will work.
Answer: Here are a couple of tools that work for me.
If I am looking for the root movement of chords I play just that: disregarding the nature of the chord, major, minor etc, I play a simple bass line either on guitar or piano. I treat this line as if I were a bass player looking only for root movement.
When I think I am close I play a simple root - V bass line. This will usually confirm things. Then I can start work on the function of the chords.
Most gospel music uses pretty straight forward harmony. If I can figure out the melody I can often work out the chord changes using a 'what fits?' approach.
A combination of these methods will usually reveal the chord changes.
Comment: Thanks. When I posted this, I was just doing the original approach (bass line) but because bass reharmonization is so common, it could mislead me if I misheard the real chord. Moving then to I-V-I-V is probably a great way to rule out any major goof-ups in my chord choices before picking out the chord qualities. Thanks!


Loading a form via AJAX with drupal_get_form()

Question: I am trying to load a form via ajax, here is the code i use:
JS:
bigPopupBut.click(function () {
        jQuery.ajax({
            url: "http://site/group_create",
            dataType: "html",
            success: function (data) {
                bigPopup.append(data);
            }
        });
})

The group_create is a PHP script that returns me a form, here is the code i used:
$node_type = 'group';
    $form_id = $node_type . '_node_form';
    $node = new stdClass();
    $node->uid = $user->uid;
    $node->name = $user->name ;
    $node->field_city['und']['0']['value'] = $user->field_city['und']['0']['value'];
    $node->field_country['und']['0']['value'] = $user->field_country['und']['0']['value'];
    $node->type = $node_type;
    $node->language = LANGUAGE_NONE;
    $node->path = '';
    node_object_prepare($node);
    $return=drupal_get_form($form_id,$node);
return $return;

The Problem i have is that the PHP callback doesn't just return me the form, but a whole page, with headers, body, so when i append it with JS i get a duplicate of the whole page in my bigPopup DIV.
What am i doing wrong ? how could i just get the form without the surounding page?
Thanks all for the help.
Answer: Try just to do echo render($return); exit();.
Otherwise, you should change de delivery callback from the path of your page, and I'm pretty sure that exists one one Drupal for Ajax.
Comment: Thanks yvan, that worked though i had to echo a render array so i added: echo render($return);
Comment: Now that you have the rendered form, does submitting it work for you? How do you submit the form when it's created via AJAX?
Comment: Submitting should work fine. You're $_POST'ing the data back to the server, including the form id and form build id, so Drupal should process the form just fine and place you right back at the URL you were at.
Comment: You should use drupal_exit() instead


How to add values depending on length of an array

Question: Not sure if I am over thinking this but I have this section of code:
struct.pack("b", this_or_address_send_array)

This section of code simply creates a packet and populates it with the value of the array, the number of b is number of characters in the array list ideally.
I want to have as many b's as I do elements in an array.
So if len( this_or_address_send_array) gave me 6 I want:
struct.pack("bbbbbb", this_or_address_send_array)

What is the most effective way of doing this ?
Answer: You can just multiply by the literal "b" by the array length
"b" * len(this_or_address_send_array)

Example:
>>> this_or_address_send_array = [5,4,3,3,6,6,2,2,3]
>>> "b" * len(this_or_address_send_array)
'bbbbbbbbb'

Answer: Construct the format dynamically, according to the length:
struct.pack("b" * len(this_or_address_send_array), this_or_address_send_array)


Deriving non-Gaussian uncertainties using Gaussian process regression

Question: I came across this work on "Gaussian Process Regression with
Heteroscedastic or Non-Gaussian Residuals" which is intriguing, but I believe I am failing to properly interpret its results. Namely, it appears non-Gaussian uncertainties can be derived using Gaussian process regression (GPR), but I am not recognizing how that is possible.
For example, in Figure 2, a plot is produced whereby the residuals are clearly asymmetric and non-Gaussian after running multiple samples ($n = 19$ to be specific). But how can this be the case in general (i.e. $n \rightarrow \infty$)? Intuitively, how does a latent or unobserved variable permit non-Gaussian uncertainties in GPR? The conditional mean, $\mu_{y^* \lvert x,y,x^*}$, and variance, $\Sigma_{y^* \lvert x,y,x^*}$, of the prediction is given by (13) and (14), respectively. Is not the fundamental assumption of GPR that:
$$y^* \lvert x,y,x^* \sim N(\mu_{y^* \lvert x,y,x^*},\Sigma_{y^* \lvert x,y,x^*})?$$
Therefore, in Figure 2, how is it possible that the predicted output, $y^* \lvert x,y,x^*$, can have residuals on average that are non-Gaussian? In general, can any data set with non-Gaussian errors be modeled using this approach or must the hidden variable, $w$, be known?
Any references to GPR codes/numerical packages incorporating non-Gaussian residuals and further literature on the topic would be appreciated as I cannot find anything explicitly beyond this author's own work.
Comment: In the cases in the paper, the expected function for $y$ appears to always be known. But given observed data from an unknown sample with possibly non-Gaussian errors on the observed data, could this paper's method be applied as GPR normally can if the observed errors were Gaussian?
Answer:
The conditional mean, $\mu_{y^* \lvert x,y,x^*}$, and variance,
  $\Sigma_{y^* \lvert x,y,x^*}$, of the prediction is given by (13) and
  (14), respectively. Is not the fundamental assumption of GPR that:
$$y^* \lvert x,y,x^* \sim N(\mu_{y^* \lvert x,y,x^*},\Sigma_{y^* \lvert x,y,x^*})?$$

The model presented in Wang & Neal is not GPR. It's one of the hundreds of models in use for GPR with heteroskedastic residuals, and in particular it's the GPLV model. The fundamental assumption of GPLV is that
$$y^* \lvert \omega, \theta, x,y,x^* \sim N(\mu_{y^*,\omega, \theta \lvert x,y,x^*},\Sigma_{y^*,\omega, \theta
\lvert x,y,x^*}) \tag{a}\label{a}$$
to put it in your terms (to be more precise, the fundamental assumptions are actually eq. (11) of Wang & Neal, and an uncorrelated multivariate standard normal prior on $\omega_1,\dots,\omega_n$). When you integrate $\ref{a}$ with respect to $p(\omega,\theta\vert x,y)$, there's no reason why $y^*$ should still have a normal distribution. You can indeed compute mean and variance of $y^*$ with the expressions (13) and (14) of Wang & Neal, but there's no reason why the higher order moments should be 0. See the example in Fig.1 of the paper.
You furthermore ask in the comments whether this GPLV model may or may not be applicable to cases where the "expected function" for $f$ is unknown. The answer is simple: as long as your data generating process can be modeled in the following hierarchical way
$$\begin{align}
y&\sim f(x) + \epsilon \\\\
f(x)&\sim\mathcal{GP}(\mu(x),k(x,x'; \theta,\omega)),\quad \epsilon \sim\mathcal{N}(0,\sigma^2)  \\\\
 \theta &\sim ??, \quad \omega \sim \mathcal{N}(0,1)
\end{align}$$
Note that I haven't specified the prior for $\theta$, because it's the usual one for a SE kernel. Also, for simplicity I wrote the kernel as $k(x,x';\theta,\omega)$, but the GPLV model actually uses a well-defined, specific kernel.
If you know how data are generated for your problem, you can decide if this model is realistic or not. If you don't, then you have a bigger problem than not simply ignoring whether GPLV is the right tool for you or not. You could still try to use the GPLV model, and then apply the vast array of posterior predictive checks developed for Bayesian models in the last 20 years, to verify if the underlying assumptions are reasonable.


Where do pending drush database updates live?

Question: I'm using drush to update the modules on our drupal site. I updated the video_embed_field module to the latest version.
This update required 3 database updates to occur, 7006, 7007, and 7008. For some reason 7006 failed as documented here. That link includes a patch which should fix the problem, the only issue is I have no idea where the patch needs to be applied. i've applied patches to drupal modules before, but video_embed_field does not appear to have a directory in sites/all/modules which is where I think the patch ought to go.
When I run drush updb it wants to apply those three database updates, but I can't find them stored anywhere in the file system (or any references to video_embed_field at all). Are these database updates stored in a table in the database, is their a folder somewhere I'm not looking at? Is there another folder where modules get placed?
Answer: Drupal will look in /modules and /sites/*/modules if the site was installed on top of the default profile. If you're using a distribution or other contrib/custom profile then it'll also look under /profiles/[name-of-profile].
SELECT filename FROM system WHERE name = 'video_embed_field'

will tell you exactly where Drupal is looking for the module you're trying to find.

Are these database updates stored in a table in the database[?]

I haven't looked at Drush, but as far as core's update.php goes, no. The .install file of each enabled module is checked to see if it has any update functions with a schema greater than the currently installed schema. I'd imagine Drush does the same thing, no real reason for it to do things differently.
If that assumption holds, then you basically couldn't be seeing those updates as options unless the module is somewhere in the file system (or at least a PHP file that will be included in a normal bootstrap containing a function called video_embed_field_update_7123() or similar).
So keep looking, it's likely to be there somewhere!
Comment: Drush calls through to Drupal for the updatedb operation, as is the case for most things it does.


FS register access violation on Windows 7

Question: Crashing instructions (any one of these will cause an access violation):
<code>LODS BYTE PTR FS:[ESI]
LODS WORD PTR FS:[ESI]
LODS DWORD PTR FS:[ESI]
</code>
I am trying to fix a crash for a client trying to port a Windows 95 game but as there is no source code available, I am currently debugging in assembly. The application frequently and randomly crashes on Windows 7 but works fine on Windows XP. Debugging it indicates an Access Violation and it occurs when trying to access the FS register.
I have debugged and compared on both XP and Windows 7 and found that the access violation occurs because when accessing the FS register using ESI, the address is always out of range on Windows 7. I also tried adding an SEH exception handler to the assembly code but the access violation happens even when reading FS:[0] (the data there is shown as unknown):
PUSHAD
PUSH 004DE918
MOV EAX,DWORD PTR FS:[0]
PUSH EAX
Could the application be causing a register corruption but what could be the likely cause of this happening only on Windows 7 but not on XP? Does the different FS register selector of 53 on Windows 7 have anything to do with this? Any suggestions or solutions for me to try?
Thank you!
Comment: How about simply running the app in WinXP compatibility mode? Or in XP Mode (XP VM in Win7)?
Comment: i tried that and the other compatibility settings as well but the crashing still occurs..
Comment: Even the XP VM known as XP Mode?
Comment: the VM works but that is not the solution... the game works in XP and I need to get it working on Windows 7 without crashing, I have been trying to step-through the code in assembly to try and fix the problem but with my limited knowledge, I am unable to figure out why ESI ends up with an invalid address...
Comment: The program apparently has a bug dependent on OS behavior. It could just as well be an uninitialized variable, use of a bad pointer/index/size, use of memory after it's been freed, memory corruption, failure to check some error code, an interesting race condition, etc. One thing that may be useful is running this app under `app verifier` + `windbg` in Windows XP and seeing what settings in the app verifier (if any) cause the program to crash even in Windows XP. You can probably do the same under Windows 7, but the search for what makes the game to stop crashing may be longer or even fruitless.
Comment: hmm.. i did already try app verifier on both xp and windows 7, it does not crash on XP and there are no problems but on windows 7 i get warnings about "Used a Windows System directory path that was not obtained using an approved method.".. and another problem is that I cannot get the game to run with WinDbg due to an assertion error that prevents the game from executing. I can only run it with OllyDbg and IDA Pro..
Comment: You should probably disable `first-chance exceptions` in WinDbg. Also, you should really play with memory allocations in AV, how they are made (see special pool, pool tracking).
Comment: Perhaps you could edit it out (in the disassembly)?
Comment: You mentioned "memory allocations in AV", what is AV?
Answer: FS register is used by Windows for quick access to the TIB (Thread Information Block) that is also called TEB (Thread Environment Block). FS store selector of the segment that is cover TIB data structure in memory. I expect that this segment have very small size and due to this access on any address out of the TEB size will lead to the General Fault exception. In fact, TEB data structure layout and storing of the address of the TEB in the FS is undocumented and can be changed by Microsoft from version to version. Maybe the TEB size was reduced in Windows 7.
In any case to discuss the issue more productively, please, provide the value of FS and ESI on the crash moment.
Comment: The value at FS when it crashes is always ????? on OllyDbg and it says Access Violation when reading [FFFFFFFF]... as for ESI, the address is always changing but mostly out of the range of my data segment (I will need to debug again to check the values), sometimes it is within the range of my data segment but the instruction it points to is invalid
Comment: Try to use windbg to get values of FS and ESI. It is important to know this values to understand which from it is wrong.


looking for text of an element or source of current page

Question: I am doing the following in selenium 2/webdrive using python and firefox...
I am opening some web pages that I need to check for a specific string - which, if present, means it is a good page to parse.
The phrase I am looking for is an h2 element similar to this:

Worlds Of Fantasy : Medieval House

If that h2 is missing, I know I don't need to work on it, just return and get the next in line.
In the code I have a try/exception/else block that looks for the phrase, if it sees it it passes on to the next part of the function. If not, it should go to the else, which tells it to return.
There are 2 pages called in this test - the first has the phrase, the second does not.
The first page is opened, and passes the test.
The second page is opened, and I get an exception report - but it never returns to the calling code in main...it just stops.
Why isn't the exception fallowing the proper path to return?
Here is the code:
    #!/usr/bin/env python

from selenium import webdriver
from selenium.webdriver import Firefox as Browser
from selenium.webdriver.support.ui import WebDriverWait

browser = webdriver.Firefox()

def call_productpage(productlink):
    global browser

    print 'in call_productpage(' + productlink + ')'
    browser.get(productlink)
    browser.implicitly_wait(8)

    #start block with

    product_block = browser.find_element_by_xpath("//div[@class='page_content']");

    #
Worlds Of Fantasy : Medieval House

    try:
        product_name = product_block.find_element_by_xpath("//h2[@class='page_title']");
    except Exception, err:
        #print "Failed!\nError (%s): %s" % (err.__class__.__name__, err)
        print 'return to main()'
        return 0
    else:
        nameStr = str(product_name.text)
        print 'product_name:' + nameStr
    finally:
        print "test over!"
        return 1

test1 = call_productpage('https://www.daz3d.com/i/3d-models/-/desk-clocks?spmeta=ov&item=12657')
if test1:
    print '\ntest 1 went OK\n'
else:
    print '\ntest 1 did NOT go OK\n'

tes2 = call_productpage('https://www.daz3d.com/i/3d-models/-/dierdre-character-pack?spmeta=ov&item=397')
if test2:
    print '\ntest 2 went OK\n'
else:
    print '\ntest 2 did NOT go OK\n'

And here is a screenshot of the console showing the exception I get:

One other option I thought about using was to get the source of the page from the webdriver and do a find to see if the tag was there - but apparently there is no easy way to do THAT in webdriver!
Comment: Is this for testing purpose ?  Is there any reason of avoiding python testing modules like `pytest` or `unittest`? I had similar situation, and selenium wasnt running all testcases unless they all passed. With pytest i could do that.besides there other benefites too.
Comment: Actually, no - his is not for testing. I am using Webdriver to revise all my custom written web scrapers - that keep breaking because the web site keeps changing the HTML...sometimes I think they do it just to bug me. Webdriver should be easier to keep current since it is more modular in nature, and I don't have to wade through as much code to find the problem. Plus, one of my clients can't get her network to access the web with the mechanize code I have been using. Hopefully she will b able to use this version.
Answer: That's the solution! Thanks!
Here is the final code, cleaned up a bit to make the result more readable:
    #!/usr/bin/env python

from selenium import webdriver
from selenium.webdriver import Firefox as Browser
from selenium.webdriver.support.ui import WebDriverWait

browser = webdriver.Firefox()

def call_productpage(productlink):
    global browser

    print 'in call_productpage(' + productlink + ')'
    browser.get(productlink)
    browser.implicitly_wait(1)
    product_block = ''
    try:
        product_block = browser.find_element_by_xpath("//div[@class='page_content']");
    except:
        print 'this is NOT a good page - drop it'
        return 0
    else:
        textStr = str(product_block.text)
        #print 'page_content:' + str(textStr)
        print '\nthis is a good page - proceed\n'

    print 'made it past the exception!\n'

    product_name = product_block.find_element_by_xpath("//h2[@class='page_title']");
    nameStr = str(product_name.text)
    print '>>> product_name:' + nameStr + '\n'
    print "test over!"
    return 1

test1 = call_productpage('https://www.daz3d.com/i/3d-models/-/desk-clocks?spmeta=ov&item=12657')
print '\nTest #1:\n============\n'
if test1:
    print '\ntest 1 returned true\n'
else:
    print '\ntest 1 returned false\n'

print '\nTest #2:\n============\n'
test2 = call_productpage('https://www.daz3d.com/i/3d-models/-/dierdre-character-pack?spmeta=ov&item=397')
if test2:
    print '\ntest 2 returned true\n'
else:
    print '\ntest 2 returned false\n'
print '\n============\n'

And that works just as I need it to.
Again, thanks.
Answer: If you don't know which exception to expect, use empty except and traceback:
import traceback

try:
    int('string')
except:
    traceback.print_exc()
    print "returning 0"

# will print out an exception and execute everything in the 'except' clause:
# Traceback (most recent call last):
#   File "<stdin>", line 2, in <module>
# ValueError: invalid literal for int() with base 10: 'string'
# returning 0

But from the stack trace you already do know the exact exception name, so use it instead:
from selenium.webdriver.exceptions import NoSuchElementException

try:
    #...
except NoSuchElementException, err:
    #...

UPDATE:
You just get an exception before the try ... except, here:
product_block = browser.find_element_by_xpath("//div[@class='page_content']");

and not here:
product_name = product_block.find_element_by_xpath("//h2[@class='page_title']");

Comment: The traceback gave me the same result. Adding the import gave me a new error: No Module named exceptions...


Extract data from power BI pbix file using SQL and R

Question: I want to extract data from a Power BI pbix file using R. The recently released pbixr package seems to offer the functions to do this but unfortunately there is hardly any documentation on how the extract data tables from a pbix file. I experimented with several functions and managed to retrieve the data tables. However, I fail to extract tables that are generated by Power BI using formulas (e.g. SUM etc). These are marked in Power BI with a calculator symbol in front.
At the office we have a simple piece of software that works very similar to the pbixr package when linking to and collecting data from a pbix file (i.e. using localhost: correct_port in combination with an SQL command), that is able to extract all data including the Power BI generated ones. This makes me think that this also should be possible by using the pbixr package but perhaps I am wrong.
Possibly the OlapR package from Microsoft also might do the trick but as this is not freeware and probably needs a specific version of R I do not want to use it. But I do welcome any other approaches to access the content of a pbix file.
A reproducible example:
install.packages("pbixr")
library(pbixr)

# Download example pbix file (also see pbixr vignette)
temp_dir <- "C:/users/dijk158/temp" # set your own temp dir
dir.exists(temp_dir)
sample_file_name <- "sample_vig.pbix"
path_file_sample <- file.path(temp_dir, sample_file_name)
url_pt1 <- "https://github.com/KoenVerbeeck/PowerBI-Course/blob/"
url_pt2 <- "master/pbix/TopMovies.pbix?raw=true"
url <- paste0(url_pt1, url_pt2)
req <- download.file(url, destfile = path_file_sample, mode = "wb")

At this point Power BI should be open and the example file sample_vig.pbix opened.
# Find port
connections_open <- f_get_connections()
connections_open$pbix <- gsub(" - Power BI Desktop", "",
                              connections_open$pbix_name)
correct_port <- as.numeric(connections_open$ports)

# Create the connection
connection_db <- paste0("Provider=MSOLAP.8;Data Source=localhost:",
                        correct_port, ";MDX Compatibility=1")

# Read tables
sql_table <- "evaluate TopMovies"
get_tables <- f_query_datamodel(sql_table, connection_db)
names(get_tables)
summary(get_tables)

get_tables only contains the pure data tables but not the Power BI generated ones such as Avg Rating, Avg Votes, etc.
Answer: The author of the pbixr package has provided a solution on the issues tracker section of the package website. See here.


Python script failing under pythonw

Question: I'm running Python 3.3 on Windows 7. I have a script that succeeds when I call it like this:
c:\python33\python.exe my_script.py

But fails when I call it like this:
c:\python33\pythonw.exe my_script.py

I want to be launching it regularly using pythonw, since it's supposed to run once every hour and I don't want to see the ugly console window every time it's launched.
(I know the script fails under pythonw because (a.) it exits immediately, when it should take about two minutes, and (b.) it's supposed to send an email and it doesn't.)
How do I even debug it? Since pythonw doesn't show any output, I have no idea what to do.
Comment: @vaultah: yes, I'm sure the OP knows the difference. Why that link, exactly?
Comment: [pythonw.exe or python.exe?](http://stackoverflow.com/q/9705982/2301450)
Comment: @vaultah Not seeing anything helpful there.
Answer: You could wrap your top-level code in a try-except handler that logs the exception:
import logging

logging.basicConfig(filename=r'C:\TEMP\debug.log', level=logging.DEBUG)
try:
    # top-level code
except:
    logging.exception('Danger, Robinson!')
    raise

You can add other logging calls, you could create a logging object to give you more fine-grained control over the configuration, etc, but as a first step that's what I'd do.
If the program still insta-exits, start bisecting. Remove half the code, compare that removing the other half (allowing for dependencies). If one half breaks and the other half works (within the constraints of half the code missing) then you know where to continue your search.
Comment: @J.F.Sebastian: I had a `as exc` target there at first until I edited out the need for one.

Python script failing under pythonw


Create a discrete variable from continuous one

Question: Have any sense, in order to solve a classification problem (only 2 possibles target classes) to create a new discrete column from a continuous one?
For example, I have one column that is Age, that has integer numbers. I have created Age_Over variable with 2 possible values 1 or 0.
Is it useful for use it in a machine learning problem maybe with certain models or just a a bad idea?

Let me explain more. I have data from workers of a company. I'm trying to solve a classification problem to predict the employee attrition.
For example I'm tempted to do this with one variable called YearsAtCompany. I have calculated the median of this column and created a new column with 1 if the value is grater than the median and a 0 otherwise.
I have noticed that the rate of attrition in the employees with less years at company than the median is significantly greater than employees with more years than the median.
What do you think?
Comment: What is your motivation for doing this. How is age related to the classification problem?
Comment: Are you asking if you should dichotomize age so that predicting age is a classification problem? If so, I would say not to do this. You lose a lot of valuable information in dichotomization, and you could always use a regression approach instead of a classification one.
Comment: Obligatory [link](http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous).
Comment: Please [merge](http://stats.stackexchange.com/help/merging-accounts) your two accounts [acct1](https://stats.stackexchange.com/users/160280/asierrayk) ... [acct2](https://stats.stackexchange.com/users/160282/asierrayk).
Comment: Thank you, I guess I was missunderstanding the concept of features tuning
Comment: Where you say "tented" do you mean "tempted" or something else?
Comment: Exactly. I'm not an english native speaker, and more often than I wish, I have some spelling mistakes. Sorry for the hassle
Answer: Unless the variable in question is strongly bimodal, I would suggest against dichotomising it using some arbitrary threshold.
Some of the obvious issues of dichotomising is that the estimated values will have reduced precision as well as that the dichotomisation assumes that the relationship between the dichotomised variable and other variables is constant within dichotomisation intervals (an obviously strong assumption that usually is unrealistic).
There is a significant body of literature presenting problems deriving from such dubious discretisation approaches. See for example: Ragland (1992) Dichotomizing continuous outcome variables: Dependence of the magnitude of association and statistical power on the cutpoint, Buettner et a. 1997 Problems in defining cutoff points of continuous prognostic factors: Example of tumor thickness in primary cutaneous melanoma and Altman et al. (1994) Dangers of using 'optimal' cutpoints in the evaluation of prognostic factors. F. Harrell's RMS book has the aptly name subsection 2.4.1: "Avoiding Categorization".
It could potentially be OK to partially dichotomising a variable into Low-/Medium-/High- value ranges and use only the Low- and High- value ranges if the variable in question is really noisy and contains a high degree of error but this is an edge case and should rely on particular domain knowledge. M. Kuhn discusses this shortly in his APM book in Chapt. 20.4 "Discretizing Continuous Outcomes".
Comment: Cool! You draw attention to an issue that a lot of novices discard as insignificant quickly only to confront it in later stages and realise that a lot of work might be invalidated. Best of luck with the rest of your analysis and welcome to the CV community!


Why my published var is not being updated by my view?

Question: I am making a personal project to study SwiftUI. All was going well, the I noticed a bug on my app.
I have the simple view bellow, that saves a description, a value and some tags on my ViewModel. I am having an issue with the $viewModel.value. That variable is not being filled with values from the view.
I supose that my @Published var value: Double? from my ViewModel should be updated whenever the user types some value. Thing is, it is not updating on any iPhone 11 and up, but it works perfectly on the iPhone 8.
public struct AddBillView: View {

    @ObservedObject private var viewModel: AddBillViewModel

    @Environment(\.presentationMode) var presentationMode

    public let onExpenseCreated: ((_ expense: Expense)->Void)

    public var body: some View {
        Text("Add Expense")
        VStack {
            TextField("Descrição", text: $viewModel.name)
            HStack {
                Text("Valor \(NumberFormatter.currency.currencySymbol)")
                CurrencyTextField("Value", value: $viewModel.value)
                            .multilineTextAlignment(TextAlignment.leading)
            }

            HStack {
                Text("Tags")
                TextField("car pets home",
                          text: $viewModel.tags)
            }

            Picker("Type", selection: $viewModel.type) {
                Text("Paid").tag("Paid")
                Text("Unpaid").tag("Unpaid")
                Text("Credit").tag("Credit")
            }
        }.navigationTitle("+ Expense")

        Button("Adicionar") {
            if !viewModel.hasExpense() {
                return
            }
            onExpenseCreated(viewModel.expense())
            self.presentationMode.wrappedValue.dismiss()
        }
    }

    public init(viewModel outViewModel: AddBillViewModel,
                onExpenseCreated: @escaping ((_ expense: Expense)->Void)) {
        self.viewModel = outViewModel
        self.onExpenseCreated = onExpenseCreated
    }
}

And I have a ViewModel:
public class AddBillViewModel: ObservableObject {
    @Published var name: String = ""
    @Published var type: String = "Paid"
    @Published var tags: String = ""
    @Published var value: Double?

    init(expense: Expense?=nil) {
        self.name = expense?.name ?? ""
        self.type = expense?.type.rawValue ?? "Paid"
        self.tags = expense?.tags?.map { String($0.name) }.joined(separator: " ") ?? ""
        self.value = expense?.value
    }

    func hasExpense() -> Bool {
        if self.name.isEmpty ||
            self.value == nil ||
            self.value?.isZero == true {
            return false
        }
        return true
    }

    func expense() -> Expense {
        let tags = self.tags.split(separator: " ").map { Tag(name: String($0)) }
        return Expense(name: self.name, value: self.value ?? 0.0 ,
                       type: ExpenseType(rawValue: self.type)!,
                       id: UUID().uuidString,
                       tags: tags)
    }
}

Then I use my view:

                AddBillView(viewModel: AddBillViewModel()) { expense in
                    viewModel.add(expense: expense)
                    viewModel.state = .idle
                }

I already google it and spend a couple of hours looking for an answer, with no luck. Someone have any ideas?
Edited
Here is the code for the CurrencyTextField. I`m using this component:
https://github.com/youjinp/SwiftUIKit/blob/master/Sources/SwiftUIKit/views/CurrencyTextField.swift
But the component works perfectly fine on iPhone 8 simulator and with a @State property inside my view. It does not work only with my ViewModel
Comment: @ikkarion create a property for AddBillViewModel in the class that own AddBillView - and - change StateObject to EnvironmentObject. My guess is that nothing in the domain is hanging on to the reference when the View's object (StateObject) is instantiated in the call-tree. Give that ViewModel a home!
Comment: After watching this video I started passing unbound data down the call stack [SwiftUI Best Practice Review](https://www.youtube.com/watch?v=X4hxE7VQvTA). Just something else to consider. Instead of having @State in views, have lets, and keep the property wrappers toward the top of the call tree.
Comment: @Helperbug

I figured it out! The problem was that my AddBillViewModel is an ObservableObject and I was marking each property with Published. This was causing some kind of double observable object.

I removed the Published and it started working again.
Comment: Is it limited to that one field? In other words, does the name field work, but the value one doesn't? My suspicion is that you'll need to show the code for `CurrencyTextField`
Comment: As jnpdx said, you will need to define CurrencyTextField for us
Comment: Try either an EnvironmentObject or Binding. How is AddBillView being used? The lifecycle of addBillViewModel depends a lot on how it is instantiated.
Comment: I added the source code for CurrencyTextField
Comment: @Helperbug I updated the question with the viewModel constructor. I`m only instantiating the ViewModel with  an empty constructor
Comment: @Helperbug I did the EnrionmentObject, but the problem continues... I think this maybe is linked with some issue during build time for the iPhone 11 and above. My viewModel is not used in any other place, so there is no cross line..
Comment: Even using EnvironmentObject it matters how it is instantiated. There needs to be a property on a class to hold on to the reference.
Answer: I figured it out! The problem was that my AddBillViewModel is an ObservableObject and I was marking each property with @Published. This was causing some kind of double observable object.
I removed the @Published and it started working again.


Removing required attribute on user entity fields not working

Question: I have added fields to user entity and make them required and show on registration page. So the user signing up have to fill all the fields. On the admin side these all fields are coming but i don't want to make them required when admin is creating any user. I thought it will be solved if i hook_form_alter  the user registration form and remove the #required attribute. I did this for admin and dpm($form). I set #required => 0 also unset #required but it keep on showing error of required field. I uncheck the required checkbox in field settings from admin panel and it does not show validation errors on both Admin and Frontend.
What i want is when admin create a user he just enter username and password and leave the rest of custom fields blank and it should register the user. But on the frontend users have to fill the whole form.
Below is the code:
    // If this is admin creating user
     foreach($form as $key => $value){
                if(strpos($key, 'field_') !== false){
                    $value['und']['#required'] = 0;
                    unset($value['und']['#required']); // This isn't working either
                }
            }
    // Endif

Updated Full Code:
/**
 * Implements hook_form_alter()
 */
function paypal_signup_form_alter(&$form, &$form_state, $form_id) {
  if($form_id == 'user_register_form'){
      global $user;
      if(!in_array('administrator', array_values($user->roles))){
        //Submit handler for redirecting user to Paypal.
        $form['#submit'][] = '_paypal_signup_redirect_paypal';
    } else {
        // Remove validation when admin is creating user.
        foreach (element_children($form) as $key) {
            if (strpos($key, 'field_') !== FALSE) {
              $form[$key][LANGUAGE_NONE][0]['#required'] = FALSE;
              $form[$key][LANGUAGE_NONE]['#required'] = FALSE;
            }
         }
        dpm($form);
    }
  }
}

Answer: Drupal has element_children() to help you safely loop over render arrays. Looping over such arrays manually will inevitably causes problems. Your code should look more like this:
foreach (element_children($form) as $key) {
  if (strpos($key, 'field_') !== FALSE) {
    $form[$key][LANGUAGE_NONE]['#required'] = FALSE;
  }
}

Incidentally, your current code doesn't work because you're modifying a variable that has only ben declared within that function's private scope.
Comment: `LANGUAGE_NONE` == `'und'`. I'm afraid I can't debug your site from here, you'll need to do that yourself. If you have more information about the problem, just edit the question. This code example is based on the assumption that you've already identified exactly where in the form array you need to manipulate the `#required` property. If you haven't already identified that, again you'll need to do so and update the question with those details
Comment: It might be something else - read [this](http://drupal.stackexchange.com/questions/27767/why-is-hook-form-alter-so-messy-in-d7) and see if that helps explain it
Comment: but i dpm the $form after this loop and for every custom fields my code removed the #required key.
Comment: Which is what you asked for...and if not, are you saying that the change you've made in the loop are not being persisted immediately _after_ the loop? I highly doubt that, but if so it indicates you have a problem with your PHP installation. That isn't standard behaviour
Comment: I tried your code and its not working. By the way in dpm i don't see [LANGUAGE_NONE] any where. The structure i am gettting is `key => array ('etc..','und' => array('#required => 1'));`
Comment: What i personally think is that the #required attribute for the form is removed with my code but there is some other #required thing in the field_api system. Since the #required is removed from the FAPI.
Comment: I am updating the screenshot of dpm. please wait.
Comment: You've got that wrong - `hook_field_widget_form()` is invoked for the fields (by `field_attach_form()`), and only then is the form sent for altering. So unless you have another, non-core, module affecting the `#required` attribute after your form_alter has run, the changes will definitely persist. That's what you need to establish, really
Comment: I think you are saying right :S i was about to send you screenshots of before and after loop dpm. But when in the loop i am setting it to `false` or `0` its not changing. However if i unset the #required it got vanished from the `und array`. So may be unsetting does not make the `#required = false ?`
Comment: Your code is setting #required to False for each field but on submit form i get the required field errors :(
Comment: In my `hook_form_alter` i took `&$form` as param. Is this wrong ?
Comment: No that's fine. Might be a good idea to edit the question and post the whole code, would be easier to spot errors that way
Comment: i've updated the full function. The dpm below my foreach tells that the #required = FALSE on both.
Comment: Yeah that's the problem - function signature needs to take `$form` by reference: `function paypal_signup_form_alter(&$form, &$form_state, $form_id) { `
Comment: I removed it just for try. I put it by reference and still getting required errors
Comment: I tried with 0 and false both. But even the form isn't submitted the * signed shows on all of the custom fields. Which means they are still required BEFORE and AFTER the form submission.
Answer: Before writing any custom code, I think you should try Profile2 Module and in addition Profile2_regpath Module provides you some Extra Options
Answer: I had the same issue with textfield. Drupal put the textfield added through GUI under a container and the rendered arrays look like this:
Array
(
    [#type] => container
    [#attributes] => Array
        (
            [class] => Array
                (
                    [0] => field-type-text
                    [1] => field-name-field-first-name
                    [2] => field-widget-text-textfield
                )

        )

    [#weight] => 5
    [#tree] => 1
    [#language] => und
    [und] => Array
        (
            [0] => Array
                (
                    [#entity_type] => user
                    [#entity] => stdClass Object
                        (
                            [uid] => 0
                            [hostname] => 0.0.0.0
                            [roles] => Array
                                (
                                    [1] => anonymous user
                                )

                            [cache] => 0
                        )

                    [#bundle] => user
                    [#field_name] => field_first_name
                    [#language] => und
                    [#field_parents] => Array
                        (
                        )

                    [#columns] => Array
                        (
                            [0] => value
                            [1] => format
                        )

                    [#title] => First Name
                    [#description] =>
                    [#required] => 1
                    [#delta] => 0
                    [#weight] => 0
                    [value] => Array
                        (
                            [#entity_type] => user
                            [#entity] => stdClass Object
                                (
                                    [uid] => 0
                                    [hostname] => 0.0.0.0
                                    [roles] => Array
                                        (
                                            [1] => anonymous user
                                        )

                                    [cache] => 0
                                )

                            [#bundle] => user
                            [#field_name] => field_first_name
                            [#language] => und
                            [#field_parents] => Array
                                (
                                )

                            [#columns] => Array
                                (
                                    [0] => value
                                    [1] => format
                                )

                            [#title] => First Name
                            [#description] =>
                            [#required] => 1
                            [#delta] => 0
                            [#weight] => 0
                            [#type] => textfield
                            [#default_value] =>
                            [#size] => 60
                            [#maxlength] => 255
                            [#attributes] => Array
                                (
                                    [class] => Array
                                        (
                                            [0] => text-full
                                        )

                                )

                        )

                )

            [#theme] => field_multiple_value_form
            [#field_name] => field_first_name
            [#cardinality] => 1
            [#title] => First Name
            [#required] => 1
            [#description] =>
            [#prefix] =>

            [#suffix] =>

            [#max_delta] => 0
            [#after_build] => Array
                (
                    [0] => field_form_element_after_build
                )

            [#language] => und
            [#field_parents] => Array
                (
                )

        )

    [#access] => 1
)

The element_children() doesn't help as it is not recursive. It loops only the immediate children whereas the textfield is rendered with three/four levels deep of arrays. The solution was to add this:
unset($form['field_first_name']['und']['#required']);
unset($form['field_first_name']['und'][0]['#required']);
unset($form['field_first_name']['und'][0]['value']['#required']);


Assign result of executing text/template template into a variable

Question: type Inventory struct {
    Material string
    Count    uint
}

sweaters := Inventory{"wool", 17}
tmpl, err := template.New("test").Parse("{{.Count}} items are made of {{.Material}}")
err = tmpl.Execute(os.Stdout, sweaters)

Instead of writing to os.Stdout, how can I save the result of the template execution in a golang variable?
Answer: as you may see here https://golang.org/pkg/text/template/#Template.Execute, there is an io.Writer arg in the execute method, so you may pass any io.Writer
i hope this will help. https://play.golang.org/p/kXRQ7G3uO20
package main

import (
    "fmt"
    "bytes"
    "text/template"
)

type Inventory struct {
    Material string
    Count    uint
}

func main() {
    var buf bytes.Buffer
    sweaters := Inventory{"wool", 17}
    tmpl, _ := template.New("test").Parse("{{.Count}} items are made of {{.Material}}")
    _ = tmpl.Execute(&buf, sweaters)

    s := buf.String()
    fmt.Println(s)
}


Why are there no notes called 3rd, 5th, 6th, 7th, 9th, 10th, 11th etc?

Question: I'm studying Time Signature and I understand that the top number indicates the amount of beats per measure and the bottom note indicates the note value of those beats. But it got me thinking about this. Why don't we use irrational time signatures that would contain 3,5,6,7,9,10,11 for a bottom note and how would we count those note values in comparison to 4 (quarter note beats), 8 (eighth note beats), 16 (16 note beats). Why are there no 3rd, 5th, 6th, 7th, 9th, 10th, 11th notes, etc? Is it simply because it is easier to sense the lengths of sounds if they are a power of two?
Comment: I was afraid this question would arise [from this one](https://music.stackexchange.com/q/135378/78419). There are many places in which someone might observe "This would be better/clearer/more efficient if we did it differently. Why do we do it this way?" The only answer is always "these practices are not usually invented in one day by one person with an end goal in mind. They evolve over time." And to "so why don't we change it now," the answer is "there's an inertia of practice that is hard to shift." Look at George Bernard Shaw's attempts to reform English phonetics and spelling.
Comment: Does this answer your question? [What is a rational rhythm?](https://music.stackexchange.com/questions/42987/what-is-a-rational-rhythm)
Comment: So your question relates more to the history of the notation and why it developed along a powers-of-two trajectory?
Comment: @gidds you could invent new symbols.  For example, you could use triangles, or a different shape of flag, or what have you.   Or you could use red ink to indicate division by a factor of three (not exactly a new concept, but it fell out of favor maybe 600 years ago).
Comment: @Aaron umm, actually yes I guess that would be correct or more accurate then asking it the way I did
Comment: As Todd Wilcox indicates: there's no point having a denominator which isn't a power of 2, since you can't write down notes that aren't binary fractions of a semibreve.  For example: in 3/3, each beat would be ⅓ of a semibreve, or 1⅓ crochets.  How would you write that down?  You'd have to tie a crotchet, a semiquaver, a hemidemisemiquaver, a demisemihemidemisemiquaver, and so on ad infinitum.  In practice, you'd have to truncate that sequence (and hence round off the value) — but it would still be almost impossible to read!
Comment: @gidds 3/3 is easy, you can write it as a triplet. 2/3 or 4/3 would be worse. Can you write incomplete triplets? Or, in case of 4/3, overgenerous triplets? - Of course, writing 3/3 as a triplet would still be stupidly complicated when the only difference between that and 3/2 is the triplet mark over every bar.
Comment: There are no 1/3 notes or 1/5 notes. Instead we usually group 1/8 or 1/16 notes differently so the unusual beat lengths are built from groups of 1/8 or 1/16 notes.
Comment: There are quite a few posts on this. Search for "irrational".
Comment: @Aaron Not quite! What Todd has said is in the direction of what I was thinking on. The Tuples aspect is something I need to look into understanding, I did read it partially here under irrational meters  -> https://en.wikipedia.org/wiki/Time_signature. I'm trying to conceptualize what those time signatures would sound like, and why they aren't common/why we don't have notes of those lengths, but I guess the best way would be to look up examples.
Comment: Because it would clash with the naming of intervals :o))
Comment: The questions "Why do time signatures mainly use powers-of-two?" and "Why are there no notes called 3rd, 5th, 6th, 7th, 9th, 10th, 11th etc?" are just different enough that they risk making the greater question too broad.
Comment: @phoog 600 years ago, you say... that would be sometime around the 15th century. I just wonder if it by any chance had something to do with printing in two colours being twice as complicated as printing in black.
Comment: @Divizna perhaps, though an alternative for different colors (usually red as the "special" color) was to use opened and closed note heads (this sometimes means that a note is 2/3 as long as the unfilled note and sometimes half as long, as it does today).  There's a decent discussion at https://en.wikipedia.org/wiki/Mensural_notation#Proportions_and_colorations.  The hypothesis that printing led to (or at least hastened) the end of the practice is quite plausible.
Answer: Technically, third, fifth, sixth, etc. notes do exist - only they aren't usually called that. Here is what some third, sixth, twelfth and fifth notes look like:

More commonly refered to with terms such as "a triplet of half notes" or "a quintuplet of quarter notes", but as you can see, they do have the length of 1/3, 1/6, 1/12 and 1/5 respectively.
What they don't appear as is the beat of a time signature.
The reasons why it's so lie, of course, in the long history of evolving notation. But the reason we don't miss such beats and don't feel the need to invent them is pretty much that the choice of the beat note is largely arbitrary. The only thing that really differs 3/4 from either 3/2 or 3/8 is the choice of which symbol we're going to use as the base one. A quarter note, or a half, or an eighth? If you adjust the tempo accordingly, they mean the same thing. So there's absolutely no reason to throw a 3/3 time into the mess. Let's just call it a 3/4 time at a slightly slower tempo, that's a much simpler frame of description.
Comment: Yes! Indeed, in many of Chopin's solo piano works, we have divisions by 7, 11, 13, and I think 17, etc. Maybe a 23. And quite a few bits of the transcriptions of Art Tatum's stuff has similar (not toooo surprising, since, apparently, he did study Chopin's music, whether from the score or from recordings, I have no idea... but evidently like the fluid sound).
Answer: When using time signatures in a straightforward way, it makes sense to choose ones that can be conveniently notated!   We have quarter notes, so lower figure '4' is sensible.  We have eight notes, so '8' is good.   We don't often count a bar as a certain number of quintuplets!
When we DO use 'irrational' time signatures, it is to indicate a tempo relationship.  '4/3' indicates a bar containing four notes if the length that  were previously notated as triplets.
Answer: Little-known factoid - you can take the reciprocal of the bottom number of a time signature and get the fraction of a whole note each component note of the time signature takes up.
For example:

4/4 and 3/4 time involve 1/4 = quarter notes
6/8 and 12/8 time involve 1/8 = 8th notes
Taking compound interpretations, 6/8 can be termed 2/(8/3) time, so it involves 3/8 = dotted quarter notes (i.e. 3/2 * 1/4 notes)

Pretty much only an implied tempo shift or tuple spam will therefore convince people to put odd numbers on the bottom of the time signature instead of powers of 2 or even fractions. Even then, a lot of people are going to want the tempo shift up front instead.
If you want odd numbers on the bottom of your time signature, then all of a sudden, you're going to want stuff like 1/3 notes or 1/5 notes:

3/3 time involves 1/3 notes (component notes of a half-note triplet)
4/5 time involves 1/5 notes (component notes of a quarter-note quintuplet)

This greatly increases the difficulty of your piece. To put this in perspective, I recall my clap-rhythms-by-sight-reading exercises to only introduce (8th-note) triplets and regular 8th notes in the same exercise in Grades 9-10. Trying to make people intuitively count out quintuplets or even large triplets is going to be heck, let alone septuplets.
Answer:
Why don't we use irrational time signatures that would contain 3,5,6,7,9 for a bottom note and how would we count those note values in comparison to 4 (quarter note beats), 8 (eighth note beats), 16 (16 note beats)?

I feel compelled to note that only a musician (a species that is renowned for being unable to count higher than four) could call something like 4/7 "irrational."
It is also worthwhile to note that the constraint on the bottom number of time signatures is not simply that they be even, but that they be a power of two.  The question recognizes this implicitly by including 6 in the list of unused time signature denominators.
There are two reasons for this.  The most obvious is that all of the notes we have in our notational system are half as long as the next longer note and twice as long as the next shorter note.  Yes, you can reduce the value of a note by using a tuplet, but that reduction applies to all note values within the tuplet, which continue to be related to one another only by factors of a power of two.  It might make sense to have a time signature such as 5/7 if there were some note that by its form was intrinsically worth one seventh of a whole note, but there is no such note.  In other words, implementing such time signatures would imply a much more radical overhaul of the metrical notation system, probably with greater cost in complexity than benefit in flexibility.
Second, and perhaps more importantly, what would be the point of such a system?  The current system is already full of ambiguity.
Waltzes are written in 3/4 by convention, but they can equally be written in 3/8, 3/16, or 3/2 (or even 6/8, 12/8, etc., etc.).  A piece in duple time can be in 4/4, 2/2, 4/8, 4/2, and more.  Meter nominally comes from dividing up time, but practically it comes from adding time units together.  You can put three of them in a bar, or four, five, six, seven, etc., but it doesn't really make a difference what the note value is, the bottom number of the time signature.  Because of conventions, people may find certain choices easier or harder to read, but mathematically it doesn't matter.  In Monteverdi's time -- the dawn of metrical time signatures -- you often find a fast 3/1 that works well at dotted-semibreve=80 or so.  You could copy it out with the note values divided by eight and mark it dotted-quarter=80; some modern musicians will find it easier to read, but no listener will be able to tell the difference.  What purpose would it serve to add more options of things-we-can-put-three-of-into-a-measure?  None, really, but it would make the system significantly more complicated.
Time signatures developed from mensural proportions, which worked a bit differently -- for example, they were true fractions, so there was no room for something such as 6/8.  The main reason I mention this, however, is that in this system, each note duration could be divided into two or three of the next shorter duration.  This was specified not by numbers but by other symbols: a circle meant three semibreves per breve, while a broken circle meant two (the forerunner of the C time signature for 4/4 time).  Adding a dot in the center indicated that the semibreves was to be divided into three minims; without the dot, there were to be two.  Putting a vertical stroke through the sign indicated that the basic unit of time was to be the breve rather than the semibreve, making everything twice as fast (the forerunner of both the term alla breve and the C-slash sign used to denote it).
One complication of this system is that even when the semibreve was to be divided into three minims, the composition of two minims was also represented by a semibreve.  There were rules to determine whether a semibreve was longer or shorter. For example, in a series of alternating semibreves and minims, the semibreves would be worth two minims, but in a series of semibreves without minims, they would be worth three.  In cases where the rules would imply that a semibreve should be two minims but it needed to be three, one would add a dot.
The numerical proportions began more as tempo marks.  They couldn't be "time signatures" in the modern sense, specifying the organization of each bar, because there were no bars (bar lines were sometimes used to set off sections of a piece, rather like a modern double bar line, but we're not yet used to denote measures of regular length.)  The slash for "twice as fast" wasn't enough.  You start to see 3 and 3/2, which are not only faster but also organized in groups of 3, and I think this is where the idea that the number tells you about the metrical organization began.  At the same time, the system whereby an undotted note could have the duration of three of the next smaller value fell out of use; people started using the dot always.
We're talking late 1500s here, and early 1600s.  In the next decades, the modern meaning of the time signature began to assert itself as bar lines began to be used with increasing regularity.  At this point, there was no interest in anything other than duple and triple divisions of time.  The upper number in a time signature always had factors of only two and three: 2, 3, 4, 6, 8, 9, or 12.  The lower number always had factors of two only: 2, 4, 8, or 16.  In the 19th century, triplets and other tuplets, often with other prime factors, came into wider use.  The larger tuplets were not necessarily strictly timed but were ornamental flourishes.  They certainly were not a regular subdivision of a beat into 5 or 7 parts.  The notation system represents them as exceptions because that is what they are.
As mentioned above, a system that has note divisions of other than two doesn't really get you much.  One thing it does get you is that you would use the whole note differently in meters whether the top number is greater than or equal to the bottom number.  For example, if you have a piece in 5/5, a whole note would last for an entire measure, whereas in 5/4 you need to tie a dotted half and half note or something like that.  But it raises more questions: what symbol do you use to represent 1/5 of a measure?  What about 2/5, 3/5, or 4/5?
I see that the question has been edited since I started writing this answer:

Why are there no notes called 3rd, 5th, 6th, 7th, 9th, 10th, 11th etc?

I think the above discussion still answers the new question, but this suggests other questions: What musical phenomenon would you use this system to describe?  Would this really be the best way to describe it?  Is there music that you want to play, or music that you've heard, that can't be rendered in western notation because of the absence of third, fifth, sixth, seventh notes, etc.?
Comment: Thank you also for the historical notes! :)
Comment: I learned those symbols as "common time" and "cut time". I had no idea those were just post hoc mnemonics. Although now I think of it, it makes sense
Comment: @NoName indeed, especially since the words for "common" and "cut" don't start with "c" in most languages!  But I think "common time" is as much a coincidence as a post hoc mnemonic; for example, in German it is known by the Italian term [tempo ordinario](https://de.wikipedia.org/wiki/Tempo_ordinario).
Comment: Mathematics is my day job and music just a hobby yet I do not object to "irrational" having a different meaning in music.  It has a specific meaning within mathematics but I don't feel any need to impose that meaning onto other subjects.
Comment: Irrational is not the correct terminology. 5 and 7 time is irregular time signatures. I don't know where people got irrational from.
Answer: The system in common use has evolved over centuries not all in one go by any individual or committee.  Could it be better - almost certainly yes.  Is it likely to change significantly anytime soon - almost certainly no.  So, if you want to be able to read and play existing music then just accept how it is and learn it.
If we were to design a new better and more logical system then I don't think that we would need to use a bottom number in the time signature at all.  Being able to reset the unit of the beat does not have an obvious value.  Pick one symbol, eg. the crotchet, to be the beat and stick to it.  I don't plan to do this redesigning.
Answer: That would be the best way for the simple reason that it is very easy to play a fastest tap rate and then count down from that. You could have a 64th note, then 2 64th notes (a 32nd), then 3 (a dotted 32nd), then 4 (a 16th) then 5 (a doted 64th tied to a 32nd) then 6 (a dotted 16th) then 7 and so on, as here:
Answer: The composer Henry Cowell describes a system using different noteheads for 1/3 and 1/5 notes etc. in his 1930 book "New Musical Resources". This was never taken up by other composers.


query API calling db_query result a non-object, but SQL works

Question: I query my database using the following query API code, but Drupal tells me that the result is a non-object. I can't find any reason to it
in what situation does db_query return anything but?
The error:

Fatal error: Call to a member function execute() on a non-object

the code:
for ($counter = 1; $counter <= 12; $counter++) {
  $query[$counter] = db_query("SELECT `foo` FROM `{bar}` WHERE DATE(FROM_UNIXTIME(`thing_$counter`)) = CURRENT_DATE");
  $results = $query['$counter']->execute()
    ->fetchAll();
}

The SQL works and produced results when I ran it in PHPmyAdmin.
The answer in this exchange looked tantalizing:
$date = date_create();
$date_to_test_against =  $date->format('U');
 $query[$counter] = db_query("SELECT `foo` FROM `{bar}` WHERE `thing_$counter` = $date_to_test_against");

however that produced the same result, calling execute on a non-object.
I know I don't have enough experience with this API yet. Can someone please point me to what I'm missing?
Comment: You have single quotes around ['$counter'], $counter won't evaluate. Also, try using db_select instead.
Answer: In the case of db_query, what you get back from it is a resource (it is just a wrapper around php's mysql_query.
But the answer depends on if you're in Drupal 6 or 7
Drupal 6:
In order to turn that into an object, you need to run db_fetch_object on the result, and if you have multiple results you'd need to loop through it.
For example:
<?php
    $sql = "SELECT buckles FROM foo WHERE llama = 'cheese'";
    $result = db_query($sql);

    while ($rowObj = db_fetch_object($result)) {
        echo $rowObj->buckles;
    }

Drupal 7
Use the nifty db_select api! The examples on that page pretty good, but basically it builds a query and then returns something that's very similar to a PDO object (see SelectQuery::execute for the following example)
<?php
  $matches = array();
  $query = db_select('table')
     ->fields('table', array('field_name'));

  foreach ($query->execute() as $row) {
     $matches[$row->relation_type] = $row->label;
  }

Comment: Thank you, that's very helpful cuz I'll have to do this for Drupal 6 in the future! I'm in Drupal 7 now.

I'm using db_query b/c my queries are static, and b/c it let me test my mysql to make sure it wasn't the problem. The documentation suggests db_query for a static query like this; do you think db_select is actually a better pick in this case?
Comment: I personally prefer to use the select api since I might wind up having to change it later, but you certainly can use db_query if that's what you want to do.
Answer: First, in the third line of your first snippet, you're using $query['$counter']. Remove the quotes.
Second, a db_query return value is a query already executed, so you want $results = $query[$counter]->fetchAll();
Comment: First, AAUUUGH! I thought I'd corrected that. This actually points out that my larger problem was having a large function of code written while learning this API, and all the resulting Frankencode. Thanks, too, to @david-thomas for pointing that out.

Second -- awesome! I was getting an entirely different impression from the documentation.

I've pared down my work to a test function, and it works. Now I'll break up my function & rewrite.


Breakpoint for bivariate data

Question: The breakpoint(s) estimation approach implemented in the strucchange package (Zeilei & al) seems to work very well (based on my little experience with this package on real case studies).
Is there an existing similar approach for multivariate regression models ? Otherwise, would it be easy to generalize the strucchange method ? (Unfortunately I do not have time to study this question myself).
Update
Here is an animated example (thanks to Yihui Xie's animation package). I have a bivariate dataset, each pair $(x,y)$ has been recorded at some date:
         x        y       date
1 3.690131 6.797999 25/01/2012
2 3.552278 7.055363 07/02/2012
3 3.623821 6.745984 10/02/2012
4 3.600735 6.847450 10/02/2012
5 3.726609 6.894321 14/02/2012
6 3.578204 6.823344 17/02/2012
...

Sarting $n_1=40$, I estimate the shape of the isodensity ellipses of the first $n_1$ observations as well as the shape of the isodensity ellipses of the remaining $n_2=n-n_1$ observations. These are simulated data. For some $n_0>40$ I have simulated Gaussian bivariate i.i.d pairs $(x_i,y_i)$, $i=1, \ldots, n_0$, and then I have simulated Gaussian bivariate i.i.d pairs $(x_i,y_i)$, $i=n_0+1, \ldots, n$ with different parameters. The question is: find $n_0$, which is here a breakpoint for the model lm(cbind(x,y)~1) in R syntax.
Comment: Do you mean multiple linear regression (or GLMs) with more than one independent variable?
Comment: @COOLSerdash No, I mean regression models for a multivariate dependent variable. Like `lm(cbind(x,y)~time` in R.
Answer: Stephane, I answer this thread because you linked it in another post.
Sadly I have not come across calculations for a QLR test with a set of dependent variables instead of one. Seeing as it (at least it used to be) considerable hard to calculate a good table of critical values for the QLR test, I don't know if such a thing exists.
Now you say that your variables are i.i.d. If they are jointly normal but independent, you can go ahead and do a split regression on both regressands and QLR test this. I don't know if you'd theoretically needed different critical values for this as well, but I suggest you err on the side of caution when it comes to significance. Other than that this should at least give a good indication where the structural break might be.
If the variables are not independent I'd probably also do this, and maybe build a normal multiple regression model out of the thing, for example x=y+t or x-y=t or something and run a few simulations on how this tests QLR wise, but this is of course trial and error if you don't want to create a whole new table on your own.
I am sorry I couldn't be of further help, I am not versed enough to give you a definite solution on the problem as I doubt one would get around having to calculate the fitting statistic yourself.
Hope this gives you some ideas.
When if comes to implementing the QLR test, just loop F-Tests over the model with break at t/vs. model without, while cutting off ~20% of the dataset at both sides (depends on the table of critical values you use, it will be stated in the table). Display the F-Values and pick the one which seems most likely, pretty straightforward.
Comment: Hello IMA. Thank you. There's no set of dependent variables. Data are independent pairs $(x_i,y_i)$ from a bivariate Gaussian (the pair $(x_i,y_i)$ is independent of the pair $(x_j,y_j)$, but $x_i$ is not independent of $y_i$). In my example above, there's no dependent variable. I didn't have the time to study the method for the univariate case, but if it is likelihood-based it should be straightforward to generalize it to the multivariate case.
Comment: Oooppss sorry... I always confound "dependent" and "independent" variables (I don't like these words). So you're right, there are two dependent variables.
Comment: It's shit terminology I know, sorry. Just read it as "regressands". In any case if you are up to it, this is the article you can work through: http://www.jstor.org/discover/10.2307/2951764?uid=3737864&uid=2129&uid=2&uid=70&uid=4&sid=21102259593731 -- I know that there are some updated tables with better values out there now, so maybe check newer research.
Comment: They are, actually, right here: http://www.pipedreamer.org/~tim/Thesis/2013-03/thesis/andrews2003.pdf -- the Andrews article should guide you towards the papers you'll need to generalize this for your case of a bivariate regressand.

Breakpoint for bivariate data


lilypond: can I combine \autochange and \change Staff?

Question: With a PianoStaff, and having all the melody in the upper staff, if we put \autochange before the melody, the notes will be displayed in the upper or the lower staff depending on their height. I would like to be able to interrupt briefly this behaviour, for a couple of notes that I want to set explicitly into some staff. I was thinking of using \change Staff = "down" but this does not work.
Answer: Well, an \autochange plays its contents in a sandbox in order to calculate the change requests, then does it "for real".  But the sandbox does not actually have the staves where a \change could have an effect.
So you need to put the \change in the "for real part", and provide a \context Voice where you'll be able to capture the autochange voice produced during the "for real" phase.  This could look like
<<
  \context Staff = "up"
  <<
    \autochange { g4 c' e' g c' e' g c' e' }
    \context Voice { s1 \change Staff = "up" }
  >>
  \context Staff = "down" { \clef bass s1*4 }
>>

Comment: For the next wayfairing stranger, `autochange` has been renamed to `autoChange`.