Trouble simplifying GDT on x86

Question: I'm trying to simplify a GDT table with contains 6 segments but in which 2 are really necessary (from what I gather). I cannot make the changes work.
The code is from Cromwell, a Xbox (Original) bootloader. The CPU is a Pentium III
. There is no concept of userspace so everything should run on segments with privilege level 0. I want to begin with a flat model with a single code32 and a single data32 segment.
Here's the relevant original working code:
<code>    .code32

.section .text, "ax"
     .org 0x00
     jmp    start_linux

.global Cromwellconfig
Cromwellconfig:
    .org 0x0c
    // Space for the SHA1 checksum
    .org 0x20

    // The Value positions are fixed, do not change them, used everywhere
    .long 0x0   // 0x20 if XBE, then this bit is 0, if Cromwell mode, the bit is set to 1 by the Startuploader
    .long 0x0   // 0x24 ImageRetryLoads
    .long 0x0   // 0x28 Bank, from where Loaded
    .long 0x0   // 0x2C 0 .. Bios = 256 k, 1 .. Bios = 1MB
    .long 0x0   // 0x30 free
    .long _end_complete_rom       // 0x34 free
    .long 0x0       // 0x38 free
    .long 0x0   // free

.align 16
tableGdt:
    .byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 // 0x00 dummy
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x9a, 0xcf, 0x00 // 0x08 code32
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x9a, 0xcf, 0x00 // 0x10 code32
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x92, 0xcf, 0x00 // 0x18 data32
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x9a, 0x8f, 0x00 // 0x20 code16 (8f indicates 4K granularity, ie, huge limit)
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x92, 0x8f, 0x00 // 0x28 data16

tableGdtDescriptor:
    // This is the GDT header having 8 bytes
    .word tableGdtDescriptor-tableGdt  // 0x30 byte GDT
    .long GDT_LOC                      // GDT located at 0xA0000
    .word 0                            // Padding
tableGdtEnd:

.align 16
tableIdtDescriptor:

    .word 2048
    .long IDT_LOC                      // IDT located at 0xB0000
    .word 0     // fill Word, so we get aligned again

        // We are dword aligned now

.align 16
    .globl start_linux
start_linux:

    // Make SURE the IRQs are turned off
    cli

    // kill the cache  = Disable bit 30 + 29 = CD + NW
    // CD = Cache Disable (disable = 1)
    // NW Not write through (disable = 1)
    // Protected mode enabled
    mov     $0x60010033, %eax
    mov %eax, %cr0
    wbinvd

    // Flush the TLB
    xor %eax, %eax
    mov %eax, %cr3

    // We kill the Local Descriptor Table
    xor %eax, %eax
    lldt    %ax

    // DR6/DR7: Clear the debug registers
    xor %eax, %eax
    mov %eax, %dr6
    mov %eax, %dr7
    mov %eax, %dr0
    mov %eax, %dr1
    mov %eax, %dr2
    mov %eax, %dr3

    // IMPORTANT!  Linux expects the GDT located at a specific position,
    // 0xA0000, so we have to move it there.

    // Copy the GDT to its final location
    movl $GDT_LOC, %edi
    movl $tableGdt, %esi
    movl $(tableGdtEnd-tableGdt)/4, %ecx
    // Moving (tableGdtEnd-tableGdt)/4 DWORDS from &tableGdt to &GDT_LOC
    rep movsl

    // Load the new GDT (bits0-15: Table limit, bits16-47: Base address)
    lgdt GDT_LOC+(tableGdtDescriptor-tableGdt)

    // Kill the LDT, if any
    xor %eax, %eax
    lldt %ax

    // Reload CS as 0010 from the new GDT using a far jump
    jmp $0x010, $reload_cs

reload_cs:

    // CS is now a valid entry in the GDT.  Set SS, DS, and ES to valid
    // descriptors, but clear FS and GS as they are not necessary.

    // Set SS, DS, and ES to a data32 segment with maximum limit.
    movw $0x0018, %ax
    mov %eax, %ss
    mov %eax, %ds
    mov %eax, %es

    // Clear FS and GS
    xor %eax, %eax
    mov %eax, %fs
    mov %eax, %gs
</code>
Changing the far jump in the code above to
<code>jmp $0x008, $reload_cs
</code>
also works fine by the way.
As you can see, protected mode is enabled at the start.
I want to trim the GDT to have a code32 segment at 0x08 and a data32 segment at 0x10. Here's my take on this; which isn't working:
<code>    .code32

.section .text, "ax"
     .org 0x00
     jmp    start_linux

.global Cromwellconfig
Cromwellconfig:
    .org 0x0c
    // Space for the SHA1 checksum
    .org 0x20

    // The Value positions are fixed, do not change them, used everywhere
    .long 0x0   // 0x20 if XBE, then this bit is 0, if Cromwell mode, the bit is set to 1 by the Startuploader
    .long 0x0   // 0x24 ImageRetryLoads
    .long 0x0   // 0x28 Bank, from where Loaded
    .long 0x0   // 0x2C 0 .. Bios = 256 k, 1 .. Bios = 1MB
    .long 0x0   // 0x30 free
    .long _end_complete_rom       // 0x34 free
    .long 0x0       // 0x38 free
    .long 0x0   // free

.align 16
tableGdt:
    .byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 // 0x00 dummy
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x9a, 0xcf, 0x00 // 0x08 code32
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x92, 0xcf, 0x00 // 0x10 data32

tableGdtDescriptor:
    // This is the GDT header having 8 bytes
    .word tableGdtDescriptor-tableGdt  // 0x18 byte GDT
    .long GDT_LOC                      // GDT located at 0xA0000
    .word 0                            // Padding
tableGdtEnd:

.align 16
tableIdtDescriptor:

    .word 2048
    .long IDT_LOC                      // IDT located at 0xB0000
    .word 0     // fill Word, so we get aligned again

        // We are dword aligned now

.align 16
    .globl start_linux
start_linux:

    // Make SURE the IRQs are turned off
    cli

    // kill the cache  = Disable bit 30 + 29 = CD + NW
    // CD = Cache Disable (disable = 1)
    // NW Not write through (disable = 1)
    // Protected mode enabled
    mov     $0x60010033, %eax
    mov %eax, %cr0
    wbinvd

    // Flush the TLB
    xor %eax, %eax
    mov %eax, %cr3

    // We kill the Local Descriptor Table
    xor %eax, %eax
    lldt    %ax

    // DR6/DR7: Clear the debug registers
    xor %eax, %eax
    mov %eax, %dr6
    mov %eax, %dr7
    mov %eax, %dr0
    mov %eax, %dr1
    mov %eax, %dr2
    mov %eax, %dr3

    // IMPORTANT!  Linux expects the GDT located at a specific position,
    // 0xA0000, so we have to move it there.

    // Copy the GDT to its final location
    movl $GDT_LOC, %edi
    movl $tableGdt, %esi
    movl $(tableGdtEnd-tableGdt)/4, %ecx
    // Moving (tableGdtEnd-tableGdt)/4 DWORDS from &tableGdt to &GDT_LOC
    rep movsl

    // Load the new GDT (bits0-15: Table limit, bits16-47: Base address)
    lgdt GDT_LOC+(tableGdtDescriptor-tableGdt)

    // Kill the LDT, if any
    xor %eax, %eax
    lldt %ax

    // Reload CS as 0008 from the new GDT using a far jump
    jmp $0x008, $reload_cs

reload_cs:

    // CS is now a valid entry in the GDT.  Set SS, DS, and ES to valid
    // descriptors, but clear FS and GS as they are not necessary.

    // Set SS, DS, and ES to a data32 segment with maximum limit.
    movw $0x0010, %ax
    mov %eax, %ss
    mov %eax, %ds
    mov %eax, %es

    // Clear FS and GS
    xor %eax, %eax
    mov %eax, %fs
    mov %eax, %gs
</code>
Can anybody spot why it wouldn't work?
Bonus questions I cannot find answers on my own:

First of all, at "tableGdtDescriptor:", shouldn't the limit value
(first word) be the size of the table - 1? So should the value here
be "tableGdtDescriptor-tableGdt - 1"? If so, why is it working in
the original code? (My assumption is that this value is that any
value above 47 bytes (6 segments - 1 byte) just get brought back to
47 bytes.
Why does the "tableGdtDescriptor" field have padding at the end if
there's a forced 16 bit align right after? It doesn't seem
necessary. Purely for good practice?
Why are FS and GS cleared and not set to the same values as for SS,
DS and ES? All examples online sets these registers to the same
segment offset. Why was it done differently here?
Comment: What happens exactly when you run your code? As for the side questions, yes the size is decurted by 1 byte, so it should be -1 (the CPU doesn't care about invalid lengths apparently). The padding is there because the size is 2+4=6 which is an "odd" number in CS. `fs` and `gs` are used however you like, but if you need to conform to an ABI of an OS, you need to set them properly.
Comment: Modified code halts that for sure. I cannot be more specific, this was run on the target itself (Xbox console) which lacks JTAG interface (TRST tied to GND permanently).

Is ignoring GDT length value a "normal" behavior for this architecture? I figure that's why the null entry is there...
Comment: The GDT length is checked when a selector is loaded into a selector register (the behaviour for lengths not multiple of 8 is not specified). I was asking because the first thing to do is check if the code that is failing is yours or the kernel loaded after that (in which case you may be violating some assumption by changing the GDT).
Comment: There is no kernel loading after that. The code just continues init (IDT, MTRR, etc.) and then jumps to C code and stays there. The fact that there is mentions of Linux in the ocde snippets above is merely because Cromwell is basically a frontend to manually select a Linux Kernel to load.
Answer: Well turns out the issue was in populating the IDT. I was pointing every IDT entries to the code segment at offset 0x10 in the GDT, hence why I needed the coded segment at offset 0x10.
Here's the fixed code which I simplified a bit:
<code>    .code32

.section .text, "ax"
         .org 0x00
         jmp    start_linux

.global Cromwellconfig
Cromwellconfig:
    .org 0x0c
        // Space for the SHA1 checksum
        .org 0x20

        // The Value positions are fixed, do not change them, used everywhere
        .long 0x0   // 0x20 if XBE, then this bit is 0, if Cromwell mode, the bit is set to 1 by the Startuploader
        .long 0x0   // 0x24 ImageRetryLoads
        .long 0x0   // 0x28 Bank, from where Loaded
        .long 0x0   // 0x2C 0 .. Bios = 256 k, 1 .. Bios = 1MB
        .long 0x0   // 0x30 free
        .long _end_complete_rom       // 0x34 free
        .long 0x0       // 0x38 free
        .long 0x0   // free

.align 16
tableGdt:
    .byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 // 0x00 dummy
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x9b, 0xcf, 0x00 // 0x08 code32
    .byte 0xff, 0xff, 0x00, 0x00, 0x00, 0x93, 0xcf, 0x00 // 0x10 data32

tableGdtDescriptor:
    // This is the GDT header having 8 bytes
    .word tableGdtDescriptor-tableGdt - 1  // Size - 1byte
    .long tableGdt                      // GDT location
    .word 0                            // Padding
tableGdtEnd:

.align 16
tableIdtDescriptor:

    .word 2048
    .long IDT_LOC                      // IDT located at 0xB0000
    .word 0     // fill Word, so we get aligned again

        // We are dword aligned now

.align 16
    .globl start_linux
start_linux:

        //Make SURE the IRQs are turned off
    cli

    // kill the cache  = Disable bit 30 + 29 = CD + NW
    // CD = Cache Disable (disable = 1)
    // NW Not write through (disable = 1)
       //   mov     %cr0, %eax
    //orl   $0x60000000, %eax
    mov     $0x60010033, %eax
    mov %eax, %cr0
    wbinvd

    // Flush the TLB
    xor %eax, %eax
    mov %eax, %cr3

    // We kill the Local Descriptor Table
    xor %eax, %eax
    lldt    %ax

    // DR6/DR7: Clear the debug registers
    xor %eax, %eax
    mov %eax, %dr6
    mov %eax, %dr7
    mov %eax, %dr0
    mov %eax, %dr1
    mov %eax, %dr2
    mov %eax, %dr3

    // Load the new GDT
    lgdt tableGdtDescriptor

    // Kill the LDT, if any
    xor %eax, %eax
    lldt %ax

    // Reload CS as 0008 from the new GDT using a far jump
    jmp $0x0008, $reload_cs

reload_cs:

    // CS is now a valid entry in the GDT.  Set SS, DS, and ES to valid
    // descriptors, but clear FS and GS as they are not necessary.

    // Set SS, DS, and ES to a data32 segment with maximum limit.
    movw $0x0010, %ax
    mov %eax, %ss
    mov %eax, %ds
    mov %eax, %es

    // Clear FS and GS
    xor %eax, %eax
    mov %eax, %fs
    mov %eax, %gs
</code>
Code modified above now sets the correct GDT size in descriptor (total size minus 1 byte). Also, the GDT is no longer copied at offset 0xA0000 in memory. The GDT register is now pointing to the original location of the GDT.
Every IDT entry now sets its selector to 0x08 to match the only code32 segment location.

Trouble simplifying GDT on x86


How to test deferred action - EventMachine

Question: I have a Sinatra app that runs inside of EventMachine. Currently, I am taking a post request of JSON data, deferring storage, and returning a 200 OK status code. The deferred task simply pushes the data to a queue and increments a stats counter. The code is similar to:
class App < Sinatra::Base
  ...

  post '/' do
    json = request.body.read

    operation = lambda do
      push_to_queue(json)
      incr_incoming_stats
    end
    callback = lambda {}

    EM.defer(operation, callback)
  end

  ...
end

My question is, how do I test this functionality. If I use Rack::Test::Methods, then I have to put in something like sleep 1 to make sure the deferred task has completed before checking the queue and stats such that my test may look like:
it 'should push data to queue with valid request' do
  post('/', @json)
  sleep 1
  @redis.llen("#{@opts[:redis_prefix]}-queue").should > 0
end

Any help is appreciated!
Answer: The solution was pretty simple and once I realized it, I felt kind of silly. I created a test-helper that contained the following:
module EM
  def self.defer(op, callback)
    callback.call(op.call)
  end
end

Then just include this into your test-files. This way the defer method will just run the operation and callback on the same thread.


how to make java decompiler generate an error

Question: I was curious about this question.
Is there a line of code or some way that I can make my program generate an error when being decompiled?
Its not a huge problem since usually it cannot read the code correctly anyways but it would be interesting to completely hide your source code.
Comment: There is the possibility to generate bytecode that simply has no valid java equivalent (e.g. a duff-device). Some imaginative, nested loop constructs making liberal used of break/continue (label) are also prone to break when decompiled (more a sign of the limits of the decompiler than an impossibility).
Answer: First off, obfuscation is never the answer. Whatever you think you need obfuscation for, you don't. And it won't stop anyone knowledgeable and determined from reverse engineering your app anyway.
But as for the actual question, it depends. Some decompilers are dumb and easy to break. Some, like Krakatau, are much more robust. There are still ways to prevent Krakatau from decompiling, but nothing that would stop a skilled reverse engineer for long.
There are also a lot of interesting tricks you can play with bytecode and the classfile format, though some of them are specific to Oracle's JVM and thus marginally reduce portability. I've written crackmes demonstrating some interesting bytecode tricks I discovered (particularly this, this, this, and this). Most of those will foil any tool other than Krakatau. But this is mostly of academic interest.
Comment: like i said, i was just interested in knowing if there is a way or not ;) thanks for the reply though
Answer: This is probably not possible, however, if you wish to protect your intellectual property, you may use ProGuard to obfuscate class, method and field names to make decompiled code hard to understand.
Answer: Well, depending on how the decompiler works, you can use something like hosemocha. But that is really really old java.
http://mguessan.free.fr/java/securite/HoseMocha.java


How to order by in eager loading on order field in sequelize?

Question: here is my code:- in sequelize how to order in eager loading.
by order field
<code>.findAll({
  include: [{
    model: db.tab,
    include: [{
      model: db.column,
      include: [{
        model: db.tabelement
      }]
    }, {
      model: db.tabbrand
    }]
  }],
  order: [
    [db.Sequelize.col('order'), 'ASC'],
    [db.tab, db.column, db.tabelement, 'order', 'ASC']
  ]
})
</code>
Answer: <code>return db.menu
.findAll({
  include: [{
    model: db.tab,
    include: [{
      model: db.column,
      include: [{
        model: db.tabelement
      }]
    }, {
      model: db.tabbrand
    }]
  }],
  order: [
    [db.Sequelize.col('order'), 'ASC'],
    [db.tab, 'order', 'ASC'],
    [db.tab, db.column, 'order', 'ASC'],
    [db.tab, db.column, db.tabelement, 'order', 'ASC']
  ]
})
</code>
Comment: I got the answer for eager loading


Como compilar e executar programa em C utilizando o Sublime Text no Ubuntu 12.04?

Question: É possível configurar o Sublime Text 2 para que ele compile um código em C e depois execute e mostre o resultado no terminal?
Comment: Dê uma olhada no [tour]. Você pode aceitar uma resposta se ela resolveu seu problema. Você pode votar em todos os posts do site também. Alguma lhe ajudou mais? Precisa que algo seja melhorado?
Answer: Sim, é possível configurar o sistema de build dele. Imagino que você já conheça a documentação. O que provavelmente te falta é como criar um arquivo de configuração no Linux. Provavelmente isto funciona:
<code>{
    "linux":
    {
        "cmd": ["cc","-std=c99" ,"$file_name","-o", "${file_base_name}", "-lm", "-Wall", ";", "./${file_base_name}"]
    },
    "selector" : "source.c",
    "shell": true,
    "working_dir" : "$file_path",
}
</code>
Retirado dessa resposta no SO.
Isto pode ter que ser adaptado dependendo de como está sua instalação.
Coloquei no GitHub para referência futura.
Comment: Eu comecei a usar o Sublime há pouco tempo e já havia visto essa build, só que ela não funcionou aqui. Apresenta esse erro: "Error trying to parse build system: Trailing comma before closing bracket in ~/.config/sublime-text-2/Packages/User/test.sublime-build:9:1"
Comment: Eu também nunca usei mas tem várias pessoas usando esta configuração.
Comment: Consegui corrigir o erro que estava dando mas o resultado foi mostrado dentro do próprio Sublime, eu queria que imprimisse no terminal. Será que não tem como?
Comment: Até onde eu vi não tem como, mas não posso garantir. Na própria resposta que eu *linkei* fala sobre isto.
Comment: Pelo menos em Python é possível, segundo esse site: http://pythoneiro.blogspot.com.br/2014/10/sublime-text-3-executando-script-no.html


Atom how to run jasmine test?

Question: I try use Atom editor and angular2.
My tests are passed using terminal command
ng test

I have tried some install and start some plugins like: mocha-test-runner, atom-jasmine, jasmine-runner, rspec to run tests using atom editor... no one can run tests.
Recommend me please good atom plugin to run angular2 tests.
Comment: you should configure **autoWatch: true** in your karmaconfig file
Comment: @Aravind It was configured from very beginning
Answer: WallabyJs is a great tool for having your tests running in an automated way, it also works with many IDEs/text editors.
https://wallabyjs.com/
(This is a paid tool)
Comment: Daniel, it is not simply test runner, it's different test framework and I have to care to configure it... I have spend a lot of time for research of my current configuration (generated automatically), I'm not sure it is good idea spend same for `wallabyjs`, but i'll try it in next few days.
Comment: Hey Sergii, are you saying you want to use a different test framework or that using Wallaby means you are using a different framework?
Wallaby is just a runner that you can hook into different frameworks so it will work with your jasmine set-up. I may have misunderstood your comment though.
Comment: I'm new in js testing, and I need a time to configure `wallabyjs`, in default configuration `package.json` I have mapped instruction `ng test`, and this instruction calls all tests... I need good plugin & I need time to configure it, I've found some examples, but my level here needs working `hello world` example :), or detail instruction for beginner, so I'm working on it.
Comment: This is not exactly i need, but it is good to know and use. thanks.
Comment: Ok, no worries.

Atom how to run jasmine test?


Store unique substrings from many lines

Question: I have a file which looks like this.
a/b/X/Y/1
a/b/X/Y/2
a/b/X/Y/3
...
a/b/X/Z/1
a/b/X/Z/2
a/b/X/Z/3
...
a/c/M/N/1
a/c/M/N/2
a/c/M/N/3
...
a/d/F/G/123
a/d/F/G/124
a/d/F/G/125

Symbols are placeholders.
I'm interested in the unique substrings consisting of the first and second symbol separated by a "/". In other words, in the example I want to build a list consisting of ["a/b", "a/c", "a/d"].
How to do this in Python idiomatically?
Answer: I would split the lines according to / with a maxsplit of 2, discard the last item (which is the rest of the string that you don't need), join back the strings, then put that in a set comprehension to remove duplicates:
with open("file.txt") as lines:
    result = {"/".join(s.split("/",maxsplit=2)[:-1]) for s in lines}

result:
>>> result
{'a/b', 'a/c', 'a/d'}

Answer: I made an example without much magic.
The first we write to the file but you can remove that (it is just to easily show you how it works).
file_content = """a/b/X/Y/1
a/b/X/Y/2
a/b/X/Y/3
a/b/X/Z/1
a/b/X/Z/2
a/b/X/Z/3
a/c/M/N/1
a/c/M/N/2
a/c/M/N/3
a/d/F/G/123
a/d/F/G/124
a/d/F/G/125
"""

# This can be removed as it is just to show how it works
with open('file.txt', 'w') as f:
    f.write(file_content)

with open('file.txt', 'r') as f:
    lines = f.readlines()

result = set()
for line in lines:
    a, b, *rest = line.split('/')
    result.add(f'{a}/{b}')

print(result)

Comment: for your example you can just to `lines = file_content.splitlines()`
Answer: def find_unique(input_file):
    output = set()
    with open(input_file) as f:
        for line in f:
            output.add(line.strip()[0:3])

    return list(output)

print(find_unique("input_file"))

This gives:
['a/b', 'a/d', 'a/c']

for file containing:
a/b/X/Y/1
a/b/X/Y/2
a/b/X/Y/3
a/b/X/Z/1
a/b/X/Z/2
a/b/X/Z/3
a/c/M/N/1
a/c/M/N/2
a/c/M/N/3
a/d/F/G/123
a/d/F/G/124
a/d/F/G/125

Comment: `line.strip()[0:3]` is assuming that identifiers can only have 1 char long (and no need to strip either with that slicing)

Store unique substrings from many lines


Qual a diferença entre "mock" e "stub"?

Question:
Em quais situações devem ser usados?
Qual a diferença entre eles?
Answer: Qual a diferença entre mock e stub?
Enquanto um stub apenas provê respostas prontas para as chamadas que serão feitas durante o teste, o mock vai mais além e, além de prover as respostas, também valida as chamadas - ele conhece o comportamento esperado do sistema e testa este comportamento.
Assim, ao substituir um componente durante os testes, um stub teria a seguinte responsabilidade:

Se o teste invocar o método A, retorno B.
Se o teste invocar o método X, retorno Y.

Enquanto um mock teria a seguinte responsabilidade:

O teste deve invocar primeiro o método A, passando o valor 1 como parâmetro, daí retorno B.
O teste deve depois invocar o método X, passando o valor 2 como parâmetro, daí retorno Y.
Se o teste não seguir exatamente esta sequência, ele falha.

Então podemos colocar na lista de diferenças o fato de que um mock é mais complexo que um stub.
É comum desenvolvedores usarem frameworks de mocks (JMock, EasyMock, Mockito, ...) apenas como stubs (não validam a interação entre os testes e o componente "mockado"). Neste caso o framework é especializado em mocks, mas conceitualmente está sendo utilizado um stub e não um mock.
Também não é incomum desenvolvedores chamarem de "mock" todos os tipos de dublês. E não há nada de tão errado nisso desde que de coração você conheça as diferenças e a complexidade trazida por cada tipo.
Em quais situações usar um é mais vantajoso que usar o outro?
Estabelecido que ambos servem para substituir componentes reais (eles são "dublês" destes componentes) durante os testes, e entendida a diferença entre eles, fica nítido quando usar um e quando usar outro:

Use stub para testar se um código, dada uma determinada entrada (respostas prontas dos métodos do stub), produz determinada saída.

Use mockpara testar se um código se comporta da maneira esperada no que tange a interações com o componente que o mock está substituindo.

Um outro dublê na história:
Algumas vezes, além de prover respostas prontas, queremos saber se o teste de fato invocou um método do componente substiuído, ou mesmo quantas vezes ele invocou, mas não precisamos ser tão rígidos a ponto de verificar a sequência das chamadas ou o valor dos parâmetros, neste caso colocamos algum estado simples no stub (contador de chamadas do método, por exemplo) e obtemos assim um spy - um outro tipo de dublê que fica entre o stub e o mock, com alguns benefícios do segundo e quase toda a simplicidade do primeiro.
Concluindo:
Usamos os diferentes tipos de dublês conforme a necessidade dos nossos testes, e damos preferência aos tipos menos complexos pois são menos acoplados ao código de produção, mais claros para serem entendidos, mais fáceis de receber manutenção.
Uma lista de dublês por ordem crescente de complexidade seria:

1º Dummy

2º Fake - o Fake pode ser bastante complexo (como um banco de dados em memória ou um servidor de aplicativos embarcado) mas não é acoplado ao seu código de produção então por isso é tido como de baixa complexidade.

3º Stub

4º Spy

5º Mock

O caminho para conseguir usar dublês em pouca quantidade ou usar os de complexidade mais baixa é a constante atenção ao design do sistema.
Answer: Acho que a referência canônica sobre o assunto é o artigo do Martin Fowler. Ele mostra a diferença entre 4 tipos de substitutos:
Dummy
São objetos usados para preencher uma lista de parâmetros quando o que contém neles não é relevante. Estes objetos não serão usados de fato.
Fake
São objetos com implementações reais, mas que não fazem exatamente o que se espera em ambiente de produção.
Stubs
São objetos criados para facilitar os testes dando respostas pré-determinadas e fazendo operações que fornecem informações adicionais do uso do método em teste. É mais importante dar facilidades para o teste acontecer com tranquilidade do que executar o teste, tanto que ele não tem como função fazer o teste falhar. Usado para substituir estados.
Mocks
São objetos com implementações especificando como é esperado que um método seja usado em código real. É com eles que você substitui comportamentos.
Conclusão
As escolhas dependem um pouco do estilo como você faz seus testes. Eu sei que muita gente não vai gostar desta resposta, mas apesar de eu gostar das coisas corretas, acho tudo isto muito exagerado para a maioria dos projetos. Não estou desfazendo das ferramentas, mas isso muitas vezes parece NoSQL. Dá muito o que falar, mas poucos realmente precisam disto tudo. Só reforço que pensar bem na sua arquitetura e testar sua implementação é fundamental.
Coloquei no GitHub para referência futura.
Comment: Obrigado pela resposta, e eu também concordo com sua conclusão! As vezes nos prendemos demais a esses conceitos e deixamos tudo muito mais complexo! (Eu acredito que em alguns projetos, essa complexidade é invariavelmente necessária, mas também sei que é possível ter bom senso e investir na arquitetura, tornando testes mais simples)

Qual a diferença entre "mock" e "stub"?


Menu Token Broken

Question: I have been looking around for an answer and haven't found an answer, so if this is a duplicate, let me know and I'll delete it.
My menu token module no longer works, even with the same settings. I do have everything set up the way it should, meaning I've got "node from context" and the same field as I have had in the past.
If you go to this page: http://arvadahistory.org/hf/wadsworth-looking-north-arvada-clear-creek-bridge - and look at the right hand side, you'll see "Order Photograph" and "Photograph Order", when you click on the "Order Photograph" link, you'll be led to a form using this url: arvadahistory.org/historic-photograph-order-form?hin=[node:field_ahs_colleciton_number] (sorry it's not a link, I need more reputation points) - however, in the older working version, it would have ended up like this: http://arvadahistory.org/historic-photograph-order-form?hin=AHS.HF.0001
So I'm not sure what has changed, does anyone have an idea as to where I can start to work on?
Matt
Comment: Try downgrading it to the version that did work.
Comment: I shouldn't have a problem downgrading right? I had read of some issues with downgrading and just didn't want to push my luck.
Comment: try the dev version.
Comment: Look at the install file for beta6. All t does is alter the weight of the module, so no biggie (if it gives a problem after downgrading you can alter the weight yourself but I doubt it will give issues). Looking at the issue queue shows a lot of issues introduced by the uuid module, so beta6 is to be avoided.
Answer: Rolled back from Beta6 to Beta5, poor response on module projects issues site but as the comments state, there are several larger bugs not being addressed by the project. Have downgraded back to Beta5 and all issues have been resolved.

Menu Token Broken


Does $A^{p^2}=I$ lead to $(A-I)^{p^2}=0$?

Question: Suppose that $A$ is a $n\times n$ matrix with entries in a finite field $F_p$ where $p$ is prime, $1\leq n \leq p$. An exercise answer told me that $$A^{p^2}=I\Rightarrow (A-I)^{p^2}=0.$$
I don't know why this holds. I tried to expand the equation as
$$
0=A^{p^2}-I^{p^2}=\left( A-I \right) \left( A^{p^2-1}+A^{p^2-2}+\cdots +A+I \right),
$$
but still have no idea what info is given. Any help would be appreciated! Thank you in advance!
Comment: Expand $(A-I)^{p^2}$ instead, using the binomial theorem.
Comment: https://en.wikipedia.org/wiki/Freshman%27s_dream#Prime_characteristic
Comment: $\binom{9}{3}=84$ is not multiple of $9$. But it is a multiple of $3$.
Comment: The binomial theorem holds for any pair of commuting elements in a ring.
Comment: @player3236@PQH@runway44 Binomial theorem actually works: every coefficient (except the first and the last terms) is a multiple of $p^2$, i.e., $\bar{0} in F_p$! Thank you all!
Comment: @lhf You are right, I should have used $p$ instead of $p^2$.
Answer: The binomial coefficients $\binom{p}{k}$ are all multiples of $p$ for $0 <k < p$. Therefore
$$
(A-I)^{p^2}=((A-I)^p)^p=(A^p-I^p)^p=A^{p^2}-I=0
$$


What is the hitting time distribution for white noise?

Question: What is the distribution of the hitting time for a stochastic process
$(W_t)_{t\in [0,T]}$,
where $W_t$ are i.i.d. Gaussian random variables?
How about in cases, in which $W_t$ are i.i.d. with a common distribution other than Gaussian?
Comment: I'm skeptical that this is really the process you want to consider; an uncountable iid family has very bad measurability properties.  Are you sure you aren't thinking of Brownian motion instead?
Answer: If $H=\inf\{t\in\mathbb R_+\mid W_t\in B\}$ for some Borel set $B$ and if $(W_t)_{t\in\mathbb R_+}$ is i.i.d. with a distribution such that $P[W_1\in B]\ne0$, then $H=0$ almost surely.
This is because, before every time $t$, one already had infinitely many chances to hit $B$. Since $W_s\in B$ for each $s\leqslant t$ with positive probability, $H$ happens before $t$ almost surely, for every positive $t$. QED.


What is the use for the "tools" folder in an Nx monorepo?

Question: I'm looking for a little bit more insight and understanding about what the use of the "tools" folder in an Nx monorepo is for. What kind of information goes in there? I can see there is a "schematics" folder, would it be a place for my customer schematics?
Answer: Yes, this is where common tooling would go for your org. Common uses are for schematics and lint rules, but there can be other things in here to help with automation.


Set ASPNETCORE_ENVIRONMENT in Google Cloud Flex App

Question: I have an error in my asp.net core 2.1 app and would like to set the ASPNETCORE_ENVIRONMENT setting to Development so that I can better debug.
There are samples for the appsettings.json file, but the ones I find specify IIS Express and I do not know what the App Engine uses.
Is there a way I can tell the instance to run with ASPNETCORE_ENVIRONMENT = Development?
Answer: You can set the environment here:
WebHost.CreateDefaultBuilder(args)
    .UseEnvironment(EnvironmentName.Development)
Answer: Try adding the following section to your app.yaml
<code>env_variables:
  ASPNETCORE_ENVIRONMENT: Development
</code>
https://cloud.google.com/appengine/docs/flexible/dotnet/reference/app-yaml


Mediaplayer prepare showing Illegal State Exception

Question: I am trying to play an audio file from the internal storage.
The code I used is..
package com.abhi.firstapp.firstapp;
import android.content.Context;
import android.media.AudioManager;
import android.media.MediaPlayer;
import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.support.design.widget.FloatingActionButton;
import android.support.design.widget.Snackbar;
import android.support.v7.app.AppCompatActivity;
import android.support.v7.widget.Toolbar;
import android.util.Log;
import android.view.View;
import android.view.Menu;
import android.view.MenuItem;
import android.widget.Toast;

import java.io.File;
import java.io.IOException;
import java.net.URI;

public class MainActivity extends AppCompatActivity {

     MediaPlayer mp;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
         super.onCreate(savedInstanceState);
         setContentView(R.layout.activity_main);
         Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar);
         setSupportActionBar(toolbar);

    File f= new File("/sdcard/a.mp3");
    if(f.exists())
    {

        Toast toast= Toast.makeText(this, "file exists", Toast.LENGTH_LONG);
        toast.show();
        Log.d("uri","1");
        Uri uri= Uri.fromFile(f);
        Log.d("uri", "2");

        mp= new MediaPlayer();
        mp.setAudioStreamType(AudioManager.STREAM_MUSIC);
        Log.d("uri", "3");

        try {
            mp.setDataSource("/sdcard/a.mp3");
        } catch (IOException e) {
            e.printStackTrace();
        }
        //mp.setDataSource(getBaseContext(), uri);
            Log.d("uri", "4");

        try {
            mp.prepare();
        } catch (IOException e) {
            e.printStackTrace();
            Log.d("uri", "IOException");
        }

        mp.start();

    }
    else {
        Toast toast1 = Toast.makeText(this, "file does not exist", Toast.LENGTH_LONG);
        toast1.show();
    }

    //MediaPlayer mp= MediaPlayer.create(getBaseContext(), uri);
    //mp.start();

}

@Override
public boolean onCreateOptionsMenu(Menu menu) {
    // Inflate the menu; this adds items to the action bar if it is present.
    getMenuInflater().inflate(R.menu.menu_main, menu);
    return true;
}

@Override
public boolean onOptionsItemSelected(MenuItem item) {
    // Handle action bar item clicks here. The action bar will
    // automatically handle clicks on the Home/Up button, so long
    // as you specify a parent activity in AndroidManifest.xml.
    int id = item.getItemId();

    //noinspection SimplifiableIfStatement
    if (id == R.id.action_settings) {
        return true;
    }

    return super.onOptionsItemSelected(item);
}
}

By using the log, I can determine that this code is running till the mp.prepare(mediaplayer prepare). And on this step, it gives the error Illegal State Exception
 Caused by: java.lang.IllegalStateException
   at android.media.MediaPlayer.prepare(Native Method)

Please Help!
Answer: There are a couple of things you might want to change.
First: mp.prepare() will  block your main thread, which is forbidden and will result in an exception where Android will close your app. To prevent this, mp.prepareAsync was designed. Use that method instead and implement both an onPreparedListener and an onErrorListener.
Second: you should provide a datasource before you call prepare().
You could do this for example this way:
public class MainActivity extends AppCompatActivity implements MediaPlayer.OnPreparedListener, MediaPlayer.OnErrorListener {

    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        ...

        MediaPlayer mp = new MediaPlayer();
        mp.setAudioStreamType(AudioManager.STREAM_MUSIC);

        try {
            mediaPlayer.setDataSource(streamURL);
        } catch (IOException e) {
            // Error, do something
        }

        mp.prepareAsync();
        ...
    }
        @Override
    public void onPrepared(MediaPlayer player) {
        mediaPlayer.start();
    }

    ...

}

Comment: I see that he did call setDataSource before the prepare in the original code..

So why was the answer accepted? Was the IllegalState due to calling prepare() instead of prepareAsync()? I have a similar problem, and I'm still calling prepare() but I do it from a background thread, so it should work, right?
Comment: This answer might be helpful to folks looking here: https://stackoverflow.com/a/53390067/826946
Comment: What is the usage of"implements MediaPlayer.OnPreparedListener, MediaPlayer.OnErrorListener" to the activity if you don't use it. useless
Comment: @LasithaLakmal - `@override public void onPrepared(MediaPlayer player)` implements `OnPreparedListener`, which is the interface that is most relevant to this question.
Answer: I had the same problem. It can be raised because MediaPlayer is already prepared. When you create a new MediaPlayer by MediaPlayer mp = MediaPlayer.create(getContext(),someUri); mp prepares automatically so you should not prepare it by yourself.


Cannot run logwatch due to Date::Manip issue

Question: I tried to run logwatch at follows
[root@machine cron.daily]# ./0logwatch
ERROR: Date::Manip unable to determine TimeZone.

Execute the following command in a shell prompt:
        perldoc Date::Manip
The section titled TIMEZONES describes valid TimeZones
and where they can be defined.

My date is as follows
root@machine cron.daily]# date
Thu Aug 23 06:25:21 GMT 2012

Now based on details in various forums I tried to fix this by setting
/etc/timezone to "+0800"

but it didn't work
My /etc/localtime points to /usr/share/zoneinfo/GMT  and is managed by puppet
How do I go about fixing this? I still want all my machines to be in GMT timezone.
EDIT:
Sadly,
Both the changes are not working:
[root@machine cron.daily]# cat /etc/TIMEZONE
UTC

Quanta's
[root@machine cron.daily]# cat ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin
export TZ=GMT

export PATH
[root@machine cron.daily]# source ~/.bash_profile
[root@machine cron.daily]# ./0logwatch
ERROR: Date::Manip unable to determine TimeZone.

Execute the following command in a shell prompt:
        perldoc Date::Manip
The section titled TIMEZONES describes valid TimeZones
and where they can be defined.

Comment: Have you tried following instruction provided at http://www.redhat.com/advice/tips/timezone.html and manage symlink with puppet?
Comment: updated my answer.
Answer: Your current timezone is GMT, why did you set /etc/timezone to "+0800"?
perldoc Date::Manip
   The following time zone names are currently understood (and can be used in parsing dates).  These are zones
   defined in RFC 822.

       Universal:  GMT, UT
       US zones :  EST, EDT, CST, CDT, MST, MDT, PST, PDT
       Military :  A to Z (except J)
       Other    :  +HHMM or -HHMM
       ISO 8601 :  +HH:MM, +HH, -HH:MM, -HH

   In addition, the following time zone abbreviations are also accepted. These do not come from a standard, but were
   included in previous releases of Date::Manip 5.xx and are preserved here for backward compatibility:

      IDLW    -1200    International Date Line West
      NT      -1100    Nome
      SAT     -0400    Chile
      CLDT    -0300    Chile Daylight
      AT      -0200    Azores
      MEWT    +0100    Middle European Winter
      MEZ     +0100    Middle European
      FWT     +0100    French Winter
      GB      +0100    GMT with daylight savings
      SWT     +0100    Swedish Winter
      MESZ    +0200    Middle European Summer
      FST     +0200    French Summer
      METDST  +0200    An alias for MEST used by HP-UX
      EETDST  +0300    An alias for eest used by HP-UX
      EETEDT  +0300    Eastern Europe, USSR Zone 1
      BT      +0300    Baghdad, USSR Zone 2
      IT      +0330    Iran
      ZP4     +0400    USSR Zone 3
      ZP5     +0500    USSR Zone 4
      IST     +0530    Indian Standard
      ZP6     +0600    USSR Zone 5
      AWST    +0800    Australian Western Standard
      ROK     +0900    Republic of Korea
      AEST    +1000    Australian Eastern Standard
      ACDT    +1030    Australian Central Daylight
      CADT    +1030    Central Australian Daylight
      AEDT    +1100    Australian Eastern Daylight
      EADT    +1100    Eastern Australian Daylight
      NZT     +1200    New Zealand
      IDLE    +1200    International Date Line East

   All other time zone abbreviations come from the standards. In many cases, an abbreviation may be used for multiple
   timezones. For example, NST stands for Newfoundland Standard -0330 and North Sumatra +0630.  In these cases, only
   1 of the two is available. I have tried to use the most recent definition, and of those (if multiple timezones use
   the abbreviation), the most commonly used. I don't claim that I'm correct in all cases, but I've done the best I
   could.

   The list of abbreviations available is documented in the Date::Manip::DM5abbrevs document in this module, or more
   fully in the Date::Manip::Zones document of Date::Manip::6.xx.

   Date::Manip must be able to determine the time zone the user is in.  It does this by looking in the following
   places:

      $Date::Manip::TZ (set with Date_Init or in Manip.pm)
      $ENV{TZ}
      the Unix 'date' command (if available)
      $main::TZ
      /etc/TIMEZONE
      /etc/timezone

   At least one of these should contain a time zone in one of the supported forms.  If none do by default, the TZ
   variable must be set with Date_Init.

   The time zone may be in the STD#DST format (in which case both abbreviations must be in the table above) or any of
   the formats described above.  The STD#DST format is NOT available when parsing a date however.  The following
   forms are also available and are treated similar to the STD#DST forms:

         US/Pacific
         US/Mountain
         US/Central
         US/Eastern
         Canada/Pacific
         Canada/Mountain
         Canada/Central
         Canada/Eastern

Which Date::Manip version are you running?
If for some reasons, Perl doesn't read /etc/timezone or /etc/TIMEZONE, you can add this line into /root/.bash_profile:
export TZ=GMT

Make it takes effect immediately by running source /root/.bash_profile and try again.

UPDATE Thu Aug 30 10:55:54 ICT 2012
Try setting the timezone in your logwatch perl script:
use Date::Manip;

Date_Init("TZ=GMT");

Answer: Actually this worked
export DATE_MANIP=DM5

Answer: That's not a fix, it's a workaround. But it should get you going.
The reason it didn't work is that it's expecting TIMEZONE to be capitalized.
echo UTC > /etc/TIMEZONE


Determining if any of these three are an ideal of $\mathbb{R}[x]$

Question:
$\mathbb{R}[x]$ denotes the ring of polynomials in $x$ with real coefficients. Let $I \subset \mathbb{R}[x]$ be the subset of those polynomials with constant coefficient $0$, and let $J \subset \mathbb{R}[x]$ be the set of polynomials with linear coefficient $0$:
$$I = \{a_1x+a_2x^2+\dots+a_nx^n|n\geq 0,a_i\in\mathbb{R}\}$$
$$J = \{b_0+b_2x^2+\dots+b_mx^m|m\geq 0,b_i\in\mathbb{R}\}$$
i) Is $I$ an ideal of $\mathbb{R}[x]$?
ii) Is $ J$ an ideal of $\mathbb{R}[x]$?
iii) Is $I\cap J$ an ideal of $\mathbb{R}[x]$?

My answers:
i)$I = \mathbb{R}[x] / a_0$ so we need only compare $a_0 * I$ and $I * a_0$
$a_0 * I = {a_0(a_1x+a_2x^2+\dots+a_nx^n)\in I}$
$I * a_0= {(a_1x+a_2x^2+\dots+a_nx^n)a_0\in I}$
Hence $I$ is an ideal of $\mathbb{R}[x]$
ii) Proof $J$ is not an ideal by counterexample:
$xJ = {xb_0 + b_2x^3 + b_3x^4+\dots+b_mx^m+1}\not\in J$
iii) $I \cap J$ in an ideal of $\mathbb{R}[x]$
$I \cap J  = \{b_2x^2+\dots+b_mx^m|m\geq 0,b_i\in\mathbb{R}\}$
Laptop is lagging to a halt now, but it is an ideal, by same logic as i)
Are these correct?
Answer: You nailed the conclusions, but did the proofs wrong.

$I$ is the ideal generated by the polynomial $x$.
$J$ is not an ideal because $1\in J$ and $x\notin J$.
$I\cap J$ is the ideal generated by the polynomial $x^2$.
Comment: Can you explain $2$?
Comment: @Examin5days *You* should explain it! `;-)` It is the required counterexample, isn't it? Since $1\in J$, for every polynomial $f$ you should have $1\cdot f\in J$.


Getting "maximum redirect" when upgrading to nginx 3.11.1

Question: I have a Kubernetes cluster running in AWS, and I am working through upgrading various components.  Internally, we are using NGINX, and it is currently at v1.1.1 of the nginx-ingress chart (as served from old stable), with the following configuration:
controller:
  publishService:
    enabled: "true"
  replicaCount: 3
  service:
    annotations:
      external-dns.alpha.kubernetes.io/hostname: '*.MY.TOP.LEVEL.DOMAIN'
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
      service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: [SNIP]
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    targetPorts:
      http: http
      https: http

My service's ingress resource looks like...
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  [SNIP]
spec:
  rules:
  - host: MY-SERVICE.MY.TOP.LEVEL.DOMAIN
    http:
      paths:
      - backend:
          serviceName: MY-SERVICE
          servicePort: 80
        path: /
status:
  loadBalancer:
    ingress:
    - hostname: [SNIP]

This configuration works just fine, however, when I upgrade to v3.11.1 of the ingress-nginx chart (as served from the k8s museum).
With an unmodified config, curling to the HTTPS scheme redirects back to itself:
curl -v https://MY-SERVICE.MY.TOP.LEVEL.DOMAIN/INTERNAL/ROUTE
*   Trying W.X.Y.Z...
* TCP_NODELAY set
* Connected to MY-SERVICE.MY.TOP.LEVEL.DOMAIN (W.X.Y.Z) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.MY.TOP.LEVEL.DOMAIN
*  start date: Aug 21 00:00:00 2020 GMT
*  expire date: Sep 20 12:00:00 2021 GMT
*  subjectAltName: host "MY-SERVICE.MY.TOP.LEVEL.DOMAIN" matched cert's "*.MY.TOP.LEVEL.DOMAIN"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET INTERNAL/ROUTE HTTP/1.1
> Host: MY-SERVICE.MY.TOP.LEVEL.DOMAIN
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Content-Type: text/html
< Date: Wed, 28 Apr 2021 19:07:57 GMT
< Location: https://MY-SERVICE.MY.TOP.LEVEL.DOMAIN/INTERNAL/ROUTE
< Content-Length: 164
< Connection: keep-alive
<

<title>308 Permanent Redirect</title>

308 Permanent Redirect

nginx

* Connection #0 to host MY-SERVICE.MY.TOP.LEVEL.DOMAIN left intact
* Closing connection 0

(I wish I had captured more verbose output...)
I tried modifying the NGINX config to append the following:
config:
  use-forwarded-headers: "true"

and then...
config:
  compute-full-forwarded-for: "true"
  use-forwarded-headers: "true"

These did not seem to make a difference.  It was in the middle of the day, so I wasn't able to dive too far in before rolling back.
What should I look at, and how should I debug this?
Update:
I wish that I had posted a complete copy of the updated config, because I would have noticed that I did not correctly apply the change to add config.compute-full-forwarded-for: "true".  It need to be within the controller block, and I had placed it elsewhere.
Once the compute-full-forwarded-for: "true" config was added, everything started to work immediately.
Comment: What happens if you curl with -L option, following redirects? Does it keeps permanent redirecting you?
Answer: This is a community wiki answer posted for better visibility. Feel free to expand it.
As confirmed by @object88 the issue was with misplaced config.compute-full-forwarded-for: "true" configuration which was located in the wrong block. Adding it to the controller block solved the issue.


What does typedef void* key_type mean in C?

Question: I am a little confused with the following line of code in C:
typedef void* key_type;

does it mean that key_type is a void pointer? I guess it's a more intuitive naming than void* in C programming.  - Thank you
Comment: Yes, that's exactly what it means.
Answer: This is typically used in public header files. For the user of the API, key_type is an opaque pointer, or handle. All you can do with key_type is pass it to functions accepting a a key_type.
Those functions then internally cast that argument from key_type / void* to a pointer to a structure where their data is stored. This requires that the key_type object is previously allocated by another function from the same API.
Basically, it's a mechanism of hiding the inner workings of a library from its users, to improve modularity.
Example:
// Public api in a .h file
typedef void* xyz_key_type;
xyz_key_type xyz_init();
void xyz_print_data(xyz_key_type handle);
void xyz_close(xyz_key_type handle);

// Private implementation a the .c file
struct xyz_private_data {
   int x, y, z;
};
xyz_key_type xyz_init() {
    struct xyz_private_data *data = malloc(sizeof(xyz_private_data));
    memset(data, 0, sizeof(*data));
    return (xyz_key_type)(data);
}
void xyz_print_data(xyz_key_type handle) {
    struct xyz_private_data *data = (struct xyz_private_data*)handle;
    printf("x :%d, y: %d, z: %d\n", data->x, data->y, data->z);
}
void xyz_close(xyz_key_type handle) {
    free(data);
}


How do I design this multi-objective fitness function for use with genetic algorithm?

Question: I have a multi-objective optimization problem that I am applying genetic algorithm (GA) to solve. Currently, there are only 2 objectives:

minimize cost
maximize validity

The cost minimization is simple, it is simply $\sqrt{(c - t)^2}$, where $c$ is the cost and $t$ is an arbitrary, user-provided value (e.g. 1000, 2500, 900, etc.).
The validity maximization is also simple as it takes the chromosome (solution candidate) and maps it to the range $[0, 1]$, with 1 being better.
For a chromosome/solution, let's denote the associated cost as $p$ and the validity as $v$. For a chromosome $x$, the fitness function (for which I am trying to minimize) $f(x)$ could be defined as $f(x) = p + \frac{1}{v}$.

if $p$ is low, then $f(x)$ is low or if $p$ is high, then $f(x)$ is high
if $v$ is low, then $f(x)$ is high or if $v$ is high, then $f(x)$ is low

For the term $\frac{1}{v}$, the highest value $v$ can ever be is 1 if $v = 1$, and infinity $\infty$ if $v = 0$. However, when, $v - 0.0001 < \epsilon$, then we simply set $v = 0.001$, so $\frac{1}{v} \in [1, 1000]$. On the other hand, $p$ is in the range $[0, r]$ where certainly $r > 0$ and possibly $r \gg 1000$.
Both terms, $p$ and $\frac{1}{v}$ can be very large and dominate the evaluation of $f(x)$. We might get the following solutions.

low cost, low validity //bad solution
low cost, high validity //best solution
high cost, low validity //bad solution
high cost, high validity //ok solution

Since $p$ and $\frac{1}{v}$ are not in the same range, we could get a "good" fitness evaluation but is invalid. For example,

if $p = 1$ and $v = 0.5$, then $f(x) = 3$ //lower cost, lower validity
if $p = 2$ and $v = 0.8$, then $f(x) = 3.25$ //higher cost, but, higher validity

In this example, GA should favor the first solution since it has a lower score, but, it has a lower validity. So, I need to punish solutions with low validity. I could weight these terms as follows $f(x) = w_p p + w_v \frac{1}{v}$, where $w_p$ is the weight for $p$ and $w_v$ is the weight for $v$, but I do not think it's trivial to specify the weights or will work still (terms can still dominate and wash out the effects).
For now, the best I can do is specify a hard threshold $\theta$ for $v$ to punish all chromosomes where $v < \theta$.

$f(x) = p$ when $v > \theta$ //valid solutions should be evaluated according to how they deviate from $t$
$f(x) = 2t$ when $v < \theta$ //invalid solutions should cost twice as much as the target $t$

Is there a better way to approach formulating the objective function? I believe this is how constraint-based optimization works in some sense anyways.
Answer: You have a few options. First, you can often just scale the objectives to the same range. If that's viable for your domain, it can work fine. It can be difficult if you don't have a good known range of each function, and you might not know the appropriate way to weight them individually.
The other major approach is to just abandon the idea of trying to combine the objective functions into a single function and instead do true multiobjective optimization. Instead of finding a single solution that maximizes some function $f(p, v)$, you search for a set of pareto optimal solutions. So you'd get some number of solutions, some very good at $p$, some very good in terms of $v$, and others at various points on the trade-off surface between the two.
Multiobjective evolutionary algorithms are a very well-studied area. There are several approaches you can find in the literature (NSGA and SPEA were early approaches, each with several enhancements made over the years; decomposition-based methods like MOEA/D are more recent methods that often perform well).


Not able to execute delete command in Cypher Query in embedded neo4j database from java progrmming

Question: I want to delete Relationship between two node from neo4j database using Cypher Queries from Java
Programming below is my code :
ExecutionEngine engine = new ExecutionEngine( graphDb );
engine.execute( "Start n= node(*) delete n" );

But it gives error as following:
Exception in thread "main" expected return clause
"Start n= node(*) delete n"
                  ^
    at org.neo4j.cypher.internal.parser.v1_7.CypherParserImpl.parse(CypherParserImpl.scala:66)
    at org.neo4j.cypher.CypherParser.parse(CypherParser.scala:44)
    at org.neo4j.cypher.ExecutionEngine$$anonfun$prepare$1.apply(ExecutionEngine.scala:61)
    at org.neo4j.cypher.ExecutionEngine$$anonfun$prepare$1.apply(ExecutionEngine.scala:61)
    at org.neo4j.cypher.internal.LRUCache.getOrElseUpdate(LRUCache.scala:31)
    at org.neo4j.cypher.ExecutionEngine.prepare(ExecutionEngine.scala:61)
    at org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:55)
    at org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:52)
    at org.neo4j.cypher.javacompat.ExecutionEngine.execute(ExecutionEngine.java:63)

But when I try to return all node this query is execute:
ExecutionEngine engine = new ExecutionEngine( graphDb );
engine.execute( "Start n= node(*) return n" ); this query return all node.

Above problem occur only with delete command...
Can anyone give me suggestion how I delete node and relationship using Progrmming in java.
Comment: What version are you using? It seems that you use an old version (pre 1.8) that doesn't support updates yet. From java you can also get all relationships of a node, delete them and then delete the node.
Comment: hi Michael i am using neo4j 1.7.2, i want to delete Relationship between two node my graphical database as:
Abhi-(Memberof)->computer,Abhi-(Memberof)->English and Abhi-(Memberof)->Java like that, means one node have same relationship with different node now i want to delete specific relationship between node like Abhi-(Memberof)->computer from java Progrmming
Answer: Common issue : Neo4j won't delete the node if it still has a relation.
If you do this :
Start n= node(*) optional match (n)-[r]-() delete n,r

this will help for this case.


The ViewData item that has the key 'officeId' is of type 'System.String' but must be of type 'IEnumerable'

Question:
Controller:

 public ActionResult Cadastrar()
    {

        ViewBag.officelist = new SelectList(new OfficeREP().ListarTodos(),
           "id",
           "estado"
       );

[HttpPost]
    [ValidateAntiForgeryToken]
    public ActionResult Cadastrar(OpportunityMOD opportunidade)
    {

        if (ModelState.IsValid)
        {
            var opportunity01 = new OpportunityREP();
            opportunity01.Salvar(opportunidade);
            TempData["mensagem"] = "Cadastro realizado com Sucesso!";
            return RedirectToAction("Index");
        }

        return View(opportunidade);

    }

Model

    [DisplayName("Escritório")]
    public string officeId { get; set; }

View

                @Html.LabelFor(model => model.officeId)
                @Html.DropDownListFor(model => model.officeId,(SelectList)ViewBag.officeList, string.Empty, new { @class = "form-control" })
                @Html.ValidationMessageFor(model => model.officeId, "", new { @class = "text-danger" })

Comment: O engraçado que estava funcionando normal, agora parou e acusa este erro... não entendo o que aconteceu. Ele lista normal na hora de gravar ele dá este erro.
Comment: Tira essa linha  @Html.DropDownListFor(model => model.officeId,(SelectList)ViewBag.officeList, string.Empty, new { @class = "form-control" })
Comment: Tirei, mais ai não lista, ok.
Comment: Então, você está tentando gerar uma LISTA de uma STRING, esse é o problema
Comment: Ok, mais como eu vou listar os objetos e gravar como string no banco de dados. Porque antes ele listava e grava, sabe me dizer ?
Comment: Liste uma lista e não uma string
Comment: Ainda sem sucesso.
Comment: Como você listou?
Comment: Vamos [continuar esta discussão no chat](http://chat.stackexchange.com/rooms/35247/discussion-between-paulohdsousa-and-rodrigo-borghi).
Answer: Olá,
O problema é que você está tentando listar uma string.
Nesta linha.
@Html.DropDownListFor(model => model.officeId,(SelectList)ViewBag.officeList, string.Empty, new { @class = "form-control" })

Depois, faça assim
Model
[DisplayName("Escritório")]
    public IEnumerable<SelectListItem> officeId { get; set; }

Controller
ViewBag.officelist = new SelectList(new OfficeREP().ListarTodos(),
"id",
"estado"
);

View
@Html.DropDownListFor(model => model.officeId,(SelectList)ViewBag.officeList, string.Empty, new { @class = "form-control" })


missing value problem

Question: I have survival data. However,there are missing values in both categorical and numerical data list. That's, in each column, approximately more than two values are missing.
Now, I want to obtain Kaplan meier function and survival functions and run cox regression.
How can I deal with these missing values to estimate these two functions and to build this regression model?
Comment: How many instances (rows) do you have?
Comment: I have 30 rows @DavideVisentin
Comment: And, all missing values are nearly in different rows. So, row-wise deletion is not useful. I lost all data if I apply deletion method. @DavideVisentin Please help me in this issue.
Comment: The first thing that comes to my mind is to artificially create these missing values by averaging the values in the other rows.
Comment: But that's not easily feasible with categorical data. This paper Batista, G. E., & Monard, M. C. (2003). An analysis of four missing data treatment methods for supervised learning. Applied artificial intelligence, 17(5-6), 519-533 describes several approaches and suggest one based on k-NN. The idea is to replace the missing values with those coming from similar instances. I don't know if it work well in your case, but try taking a look.
Comment: Problems of missing data are complicated. You can look into multiple imputation, but you probably should consult with an expert.
Comment: I am not the consultant you want for this job. I know a bit about missing data, but not enough for this.
Answer: You could use the multiple imputation method to create multiple completed datasets based on which you could do your analysis. From your description, I presume that your outcome of interest is a time-to-event. In this case and because in the multiple imputation procedure it is important that you include the outcome, you should include the estimated log cumulative hazard function for the cases you have survival information. And then perform multiple imputation to fill in the covariates values you miss.
Comment: In fact, I do not understand how to deal with missing data in categorical data. Can you expand your answer for categorical data? Is there any software program and package that you suggest for this? Thanks a lot.

missing value problem


How to find all opened tags using regular expression

Question: <meals>
    <breakfast>
         Eggs and Toast
    </breakfast>
    <lunch>
         Salad and soup

this is my XML.I just wanted to get Opened Tag(Lunch,Meals) from this xml using regular expression.
Please help me.
Comment: What have you already attempted?
Comment: var pattern1 = "(?<(?[a-z]+)>)([^<]+?)";
Answer: This answer is a bit more to the point (it's about HTML, but concerns XML as well).
If this does not scare you, at least try to make the question more specific (is this just one kind of XML? what "meals"?)
Answer: To get the data between a right and left angle bracket (where the left bracket is not succeeded by a forward slash), you can use the following expression:
<([^<>/]+)>

...where the first (and only) capture group will contain the name of the tag; "group 0" will contain the whole match--name and brackets.


$group->addContent() in hook_node_insert() doesn't update the access permissions for the node

Question: I have a content type with a field that lets the users decide which group to add the content to.
The content is added to the group via hook_node_insert().
function tm_wiki_node_insert(NodeInterface $node) {
  $type = $node->getType();
  if ($type == 'mycontent') {
    $group = Group::load('1');
    $group->addContent($node, 'group_node:mycontent');
  }
}

It works: I can see the node is properly added to the group. However, users who aren't members of the group can still view the content, even though group permissions would not allow this.
When I manually edit and save the node once, the permissions are correctly set; users who aren't group members cannot view the node.
What code should I add to make the node only viewable from group members?
I don't understand the problem, since hook_node_insert() should be invoked after the entity is saved, and $group->addContent() should not require an additional save to set permissions properly.
Answer: I found the problem - it probably has to do with social_groups submodule, which adds a field for content visibility (field_content_visibility) - this field is not set properly when just adding the content via group->add content.


Controlling parents education that correlates with each other

Question: I want to include both parent's education variables as control variables in my estimation about the effects of maternal bargaining power to child's educational attainments. They are stated as categorical variables ranging 0 for "not receiving educ" and 4 for "tertiary-level educ". Many previous papers in my topic have also included these as controls, and most of them yield significant results in the main independent variable (maternal bargaining power). But when I did the same, my independent variable would lose significance greatly, and turns out they're highly correlated with each other. Is there any way to get around this and still include these variables in the model without sacrificing significance?
Comment: I guess my big question is why are you so convinced that strong correlation between the two education variables will kill the significance of your variable of interest?
Comment: Four considerations come to mind: 1) Does your variable of interest have a high variance inflation factor? (There are generalizations of variance inflation factor if you are not using OLS linear regression.) 2) What kind of effect do you observe in your data, and how does this compare to other studies? 3) What other control variables do other studies use that might be able to cut down on some of the error variance that is used for calculating standard errors? 4) What kind of sample size do you have compared to other studies. If they have billions compared to your hundreds, they have an edge.
Comment: Welcome to Cross Validated! What is correlated with what?
Comment: Mother's education correlates with father's education. Is this something to be concerned with or I can just ignore it? Thank you.
Comment: Do those education variables correlate with your variable of interest?
Comment: They do but not as much as the correlation between parent's education variables. The correlation of each parent's education variable to my dependent variable is 0.1 while the correlation between mother and father's education is 0.69
Comment: I assume that parent's education variables will always have that degree of correlation because educated males tend to marry educated females but I'm just wondering why I can't achieve the same significance results as similar, previous papers (even ones with the same data source). High correlation is just another suspicion, but it might not be the answer.
Answer: Four considerations come to mind.

Does your variable of interest ("maternal bargaining power") have a high variance inflation factor? (There are generalizations of variance inflation factor if you are not using OLS linear regression.) I will leave vague what I mean by "high", but people often take five or ten to be high and to indicate problems with the standard errors being inflated far too much. If you have a modest variance inflation factor on your "maternal bargaining power" variable, then the correlation between the two education features is not important for your study. Sure, you might get funky behavior with those, but those are not the variables of interest, so there is a sense in which you need not care. Further, the fact that the two education variables are correlated is not much of a consideration for this kind of variance inflation factor analysis. If they are independent yet the variance inflation factor on "maternal bargaining power" is high, then you might have an issue, and if the correlation is high yet the variance inflation factor on "maternal bargaining power" is low, then there probably is no problem.

What kind of effect do you observe in your data, and how does this compare to other studies? If you observe a smaller effect size than other studies have found (which might be a result of publication bias that leads to only the large magnitudes being published), then of course you are having a hard time rejecting a null hypothesis of zero effect.

What other control variables do other studies use that might be able to cut down on some of the error variance that is used for calculating standard errors? The point of including control variables is to account for variability in the outcome and to reduce the variance of the error term that is used for calculating coefficient standard errors. If you do not include controls that other studies include, you might not be shrinking the error variance as much as they are, so you wind up with larger standard errors and a study with less power.

What kind of sample size do you have compared to other studies? If they have billions compared to your hundreds, they have a considerable edge when it comes to power to reject a null hypothesis of zero effect, even if your observed effect magnitude is larger. The larger sample size, all else equal, leads to smaller standard errors and higher power.
Comment: Thank you for answering Dave, I realize my problem might be number 2 and 4 from your answer. I had a considerably smaller effect size and number of observations compared to the previous studies, as my data source has incomplete data here and there. If these were the case I don't think I should pursue level of significance anymore, and maybe put it in the limitations of my work.


How do you do change feed monitoring in a mongo API Cosmos DB now change feed processing has been disabled and Azure functions dont exist

Question: Hey we have been using the change feed processor library for 6 months with a Cosmos DB and Mongo API  but now when we try and connect to the change processor library we get a Document Exception and the mesage
<code>Sql api is not supported for this database account
</code>
The nuget we are using is
Microsoft.Azure.DocumentDB.ChangeFeedProcessor v2.2.7
The last time we created the new database was about 3 weeks ago and its still working but every new mongo api cosmos we create throws this error.
If this has been retrospectivly removed without documentation or warning is there any other way to listen to changes in the database without writing my own monitoring software.
The documentation has been changed and now says it doesnt support it but i have a working DB that says it does.
Answer: So the feature wasnt really supported and has been disabled for new databases but there is a new change feed processor lib for mongo api enabled cosmos due out in Spetember.
Marking as answered as not sure there is a workaround available until the lib is out


Problema com formatação de hora ao usar o Moment.js e o Dialogflow

Question: Estou com um problema ao digitar algum horário no Dialogflow para meu chatbot. Sei que o sistema trabalha com AM e PM, mas tentei formatar a data com o Moment.js (segue o código abaixo), mas não está funcionando. Ao colocar um input de 9h, 10h, 11h ou 12h ele reconhece respectivamente como 21h, 22h, 23h e 00h. Alguma sugestão?
function consulta(agent) {
    let data = moment(agent.parameters.date).format('L');
    let tempo = moment(agent.parameters.time).tz('America/Recife').format('HH:mm');
    agent.add(`${data} às ${tempo}`);
}

Comment: Sugiro [edit] a pergunta e colocar o valor de `agent.parameters.time`, para os casos que funciona e que não funciona. Assim podemos saber melhor o que pode estar acontecendo (já que agora não dá pra saber se é uma string, um Date, em qual formato vem, etc)
Answer: De acordo com a documentação sobre formatação do Moment.js, os tokens HH e mm formatam, respectivamente, para a hora em 24 horas (entre 00 e 23) e o minuto (entre 00 e 59).
Desse modo, se você quiser formatar, por exemplo, o horário 3 horas e 8 minutos PM, utilizando o padrão acima, você terá:
"HH:mm" → 15:08

Se você não quiser a formatação no modelo de 24 horas, precisará usar os tokens hh e mm, além do A – para indicar AM ou PM.
Assim, se você quiser formatar o mesmo horário de antes – 3 horas e 8 minutos PM – utilizando esse novo padrão, você terá:
"hh:mm A" → 03:08 PM

Então, basicamente:
┌─────────┬─────────┬────────────────────────────┐
│  Tipo   │ Formato │          Exemplo           │
├─────────┼─────────┼────────────────────────────┤
│ Período │ A       │ AM (ou) PM                 │
│ Período │ a       │ am (ou) pm                 │
│ Horas   │ HH      │ 01, 02, ... 12, 13, ... 23 │
│ Horas   │ hh      │ 01, 02, ... 12, 01, ... 11 │
└─────────┴─────────┴────────────────────────────┘

Novamente, para saber mais sobre esses formatos, consulte a documentação de formatação.


zerorpc in flask app is not working when launched in apache server but working fine when launched by pycharm

Question: I am using flask application to display the data read using zero rpc subscriber.
While developing it was working fine as I was launching the app using Pycharm But once deployed in Apache server the subscriber is not able to fetch any data. Basically the method in subscriber is not getting called even though the publisher is publishing the data.
Thanks
Edit: (adding more info):
In app_start.py
from gevent import monkey
monkey.patch_all()
from flask import Flask
app = Flask(__name__)

import stratgy_subscriber
startegy_state_info_provider = stratgy_subscriber.StartegyStateInfoProvider()
@app.route('/')
def strategy_info():
    return startegy_state_info_provider.ip_port_to_subscriber_map[('0.0.0.0', '4249')]

if __name__ == '__main__':
    startegy_state_info_provider.start_subscriber('0.0.0.0', '4249')
    app.run()

I am connecting to subscriber using following code.
In stratgy_subscriber.py
from gevent import monkey
monkey.patch_all()

class StartegyStateInfoProvider(object):

  def __init__(self):
      self.ip_port_to_subscriber_map = {}

  def start_subscriber(machine_ip, port):
      strategy_state_subscriber = StrategyStateSubscriber()
      ip_port_to_subscriber_map[(machine_ip, port)] = strategy_state_subscriber
      end_point = 'tcp://' + machine_ip + ':' + port
      strategy_state_subscriber = zerorpc.Subscriber(strategy_state_subscriber)
      strategy_state_subscriber.connect(end_point)
      gevent.spawn(strategy_state_subscriber.run)

class StrategyStateSubscriber():
  def __init__(self):
    self.strategy_id_to_info_map = {}
  def update_strategy_state(self, strategy_id, updated_strategy_state):
    from datetime import datetime
    current_time = datetime.strftime(datetime.now(), '%H:%M:%S')
    updated_strategy_state.append(current_time)
    print updated_strategy_state
    strategy_id_to_info_map[strategy_id] = updated_strategy_state

This is the Publisher`
endpoint = "tcp://0.0.0.0:4249"
publisher = zerorpc.Publisher()
publisher.bind(endpoint)

for a in xrange(0, 1000):
    info = [False, a * a, 3 * a, a % 20, a % 50]
    publisher.update_strategy_state(a, info)
    if a < 50:
        gevent.sleep(1)
    elif a >= 50:
        gevent.sleep(2)

After subscription i.e. gevent.spawn(strategy_state_subscriber.run), update_strategy_state should get called each time the publisher publishes the data which is not happening.
I am getting following exception:
[Wed May 28 11:24:03 2014] [error] Traceback (most recent call last):
[Wed May 28 11:24:03 2014] [error]   File "/home/sricharan/git_ceres_viewer/src/strategy_state_subscription.py", line 68, in subscribe
[Wed May 28 11:24:03 2014] [error]     subscriber = zerorpc.Subscriber(service)
[Wed May 28 11:24:03 2014] [error]   File "/usr/local/lib/python2.7/dist-packages/zerorpc/core.py", line 371, in __init__
[Wed May 28 11:24:03 2014] [error]     zmq_socket=zmq.SUB)
[Wed May 28 11:24:03 2014] [error]   File "/usr/local/lib/python2.7/dist-packages/zerorpc/core.py", line 312, in __init__
[Wed May 28 11:24:03 2014] [error]     super(Puller, self).__init__(zmq_socket, context=context)
[Wed May 28 11:24:03 2014] [error]   File "/usr/local/lib/python2.7/dist-packages/zerorpc/socket.py", line 34, in __init__
[Wed May 28 11:24:03 2014] [error]     self._events = Events(zmq_socket_type, context)
[Wed May 28 11:24:03 2014] [error]   File "/usr/local/lib/python2.7/dist-packages/zerorpc/events.py", line 177, in __init__
[Wed May 28 11:24:03 2014] [error]     self._socket = zmq.Socket(self._context, zmq_socket_type)
[Wed May 28 11:24:03 2014] [error]   File "/usr/local/lib/python2.7/dist-packages/zerorpc/gevent_zmq.py", line 61, in __init__
[Wed May 28 11:24:03 2014] [error]     self.__dict__["_state_event"] = gevent.hub.get_hub().loop.io(
[Wed May 28 11:24:03 2014] [error]   File "/usr/lib/python2.7/dist-packages/gevent/hub.py", line 135, in get_hub
[Wed May 28 11:24:03 2014] [error]     raise NotImplementedError('gevent is only usable from a single thread')
[Wed May 28 11:24:03 2014] [error] NotImplementedError: gevent is only usable from a single thread

After some research I found out that :-
gevent isn't compatible with Apache's multithreaded worker model
Comment: Your system consist of couple of components and you will have to investigate one by one, if all works as expected. This is not possible to do from the description you have provided.
Comment: Thanks for the reply Jan.
But my concern is why its happening only in apache server not when using Pycharm IDE?
And which component is affected when deployed in apache server.
Struggling for the solution for couple of days.
Comment: @JanVlcinsky
I am connecting to subscriber using following code.
`
strategy_state_subscriber = StrategyStateSubscriber()
end_point = 'tcp://' + machine_ip + ':' + port
strategy_state_subscriber = zerorpc.Subscriber(strategy_state_subscriber)
strategy_state_subscriber.connect(end_point)
gevent.spawn(strategy_state_subscriber.run)
`
Comment: @JanVlcinsky

This is the StrategyStateSubscriber class.

`
class StrategyStateSubscriber():

    def update_strategy_state(self, strategy_id, updated_strategy_state):
        from datetime import datetime
 current_time = datetime.strftime(datetime.now(), '%H:%M:%S')
 updated_strategy_state.append(current_time)
 self.__strategy_id_to_state_map[strategy_id] = updated_strategy_state

`
Comment: It is better to put additional info for the question into the question - easier to read (I did it for you). Add more info yourself, it is hard to read in comments. Anyway, I still have no idea, what is going on. Either add more info (do not focus on what you think is wrong, but on what allow others to reproduce the problem). Or better - split your problem to pieces and test one by one.
Comment: I am not sure you have enough credit to edit your post so I did it again. Not sure, I formatted it properly.
Comment: @JanVlcinsky Thanks for the tips

I have added some more info to reproduce the error
Comment: Please, add all required imports into your question. I found `zerorpc` very promising and would like to test it, but in some places I miss something, e.g. at "Publisher" I have no idea, where the `.update_strategy_state` comes as method of `publisher` being instance of `zerorpc.Publisher()`.
Comment: I managed the publishing code to publish (those calls of `publisher.update_strategy_state`).
Comment: @JanVlcinsky
    update_strategy_state is a user defined function, on which publisher publishes the data at an end_point (ip, port).
update_strategy_state method of the subscriber will be invoked if subscriber is subscribing to this end point
Comment: I managed consuming published calls, but had to add one line after `gevent.spawn(strategy_state_subscriber.run)` which actually gave the code a chance to run. I added `gevent.sleep(5)` and immediately got the data printed. Without that, the code of the subscriber simply runs out without any chance to receive any call from publisher. There could be other calls of this kind.
Answer: Finally I got it working.
import zerorpc
import gevent
from subscrclass import StrategyStateSubscriber
strategy_state_subscriber = StrategyStateSubscriber()
machine_ip = "127.0.0.1"
port = "5555"
end_point = "tcp://" + machine_ip + ":" + port

strategy_state_subscriber = zerorpc.Subscriber(strategy_state_subscriber)

strategy_state_subscriber.connect(end_point)
gevent.spawn(strategy_state_subscriber.run)
gevent.sleep(5) # This line was added

Original code with subsriber call ended with spawning it, but did not actually give any chance for the code to do anything.
If I added gevent.sleep(5), coroutines got a chance to run for 5 seconds and the prints were appearing.
Be sure to use some gevent call, simple time.sleep(5) does not work as it is blocking one and coroutines have no chance to say a word or run a line of code.
With your PyCharm IDE I can imagine, you either had similar line present, or you were performing the actions "slowly", and it had a chance to run. But this is pure speculation - important is to give in your code a chance to your coroutines to run before your code exits.
Complete example (without Flask part)
Here is complete runnable sample
subscrclass.py
This class will actually receive calls from publisher. It is important:

having name update_strategy_state
having 2 parameters

Here it goes:
from datetime import datetime

class StrategyStateSubscriber():
    def update_strategy_state(self, strategy_id, state):
        print datetime.strftime(datetime.now(), "%H:%M:%S"), state

pub.py
Here is the publisher.
It does not import anything to get the StrategyStateSubscriber, it only creates a call to some remote subscriber, which might have implemented a method named update_strategy_state with two parameters.
import gevent
import zerorpc

publisher = zerorpc.Publisher()
publisher.bind("tcp://0.0.0.0:5555")

for a in xrange(0, 1000):
    publisher.update_strategy_state(a, [False, a * a, 3 * a, a % 20, a % 50])
    gevent.sleep(1 if a < 50 else 2)

client.py
Finally we come to client, who subscribes to published calls and passes them to relevant method (decided by publisher) implemented on class StrategySateSubscriber
import zerorpc
import gevent
from subscrclass import StrategyStateSubscriber

strategy_state_subscriber = zerorpc.Subscriber(StrategyStateSubscriber())
strategy_state_subscriber.connect("tcp://127.0.0.1:5555")
gevent.spawn(strategy_state_subscriber.run)
gevent.sleep(5)

It is important to allow spawned process to run, here we call gevent.sleep(5), other calls or non-blocking I/0 operations could help too.
Running the code
In first console start the publisher:
$ python pub.py

In second console, start the client:
$ python client.py
22:30:06 [False, 16, 12, 4, 4]
22:30:07 [False, 25, 15, 5, 5]
22:30:08 [False, 36, 18, 6, 6]
22:30:09 [False, 49, 21, 7, 7]
22:30:10 [False, 64, 24, 8, 8]

Conclusions

zerorpc turns to be really powerful tool
with spawning processes, programmers sometime forget to give the code a chance to run. This was the cause here.
In case, the web page in Flask shall return quickly, it would be faster using request/response (remote procedure call) instead of sleeping to get some data.
In case, the strategy status is to be fed to client in long term (e.g. by WebSockets), publish/subscribe messaging patterns is best fitting one.
Comment: Thanks for the reply. I try to implement this and let you know whether it worked or not.
But I think gevent.spawn() will run forever and I used ` from gevent import monkey; monkey.patch_all() `. My flask app also runs for long time enough to capture the published data.
 P.S. the same code without any sleep runs fine when launched by Pycharm
Comment: @SricharanMadasi Without some waiting instruction it will be like pouring water into basin with open sink. `spawn` starts new thread, but does not wait - it is actually the purpose of it - start new thread and go on having free hands. And in your Flask app, your code generating some output exits as soon as you have your request processing finished. This request processing is mostly very quick (and you want have your response generated quickly), so the request processing code does not run for very long.
Comment: @SricharanMadasi Btw, where did you learn using `zerorpc` this way? I did not find any tutorial and documentation is rather short on this. I like this and would like to learn something more.
Comment: Actually the code gevent.spawn() is called on start of the application and will run foreever.
updated_strategy_state for each strategy_id is stored in server side in a dict.
On request from web I fetch the data stored in server and display on browser.
Main problem is update_strategy_state function it self is not called. ( I have added logs and checked)
Comment: Below is th link for som examples of pub/sub and push/pull....https://github.com/dotcloud/zerorpc-python/blob/master/tests/test_pubpush.py
Comment: @SricharanMadasi Could you share the Flask code? (and thanks for link to the test suite).
Comment: I have added the flask code in the question.
Comment: @SricharanMadasi The code you have provided is incomplete and cannot be run. It does not even show, how you share updated status with your request and this probably hides root of the problem. Also imports are missing. If you attempt to create minimal working example, you either find the problem, or help me or others to answer.
Comment: As I don't have the code in this system, I will update the actual code in few hours once I log in to office machine.
Thanks
Comment: I have updated the question with the code similar to my flask app.
I will update the exact code tomorrow. Apology for the delay.
Comment: gevent isn't compatible with Apache's multithreaded worker model
Comment: @SricharanMadasi That is sad. But one becomes expert by failing in all possible ways. Did you try `nginx`? I am not sure, it supports gevent, but it might.


Collatz Conjecture: does an integer $4x+1$ resolve in $n$ steps if and only if an odd $x$ resolves in $n$ steps

Question: When analyzing the collatz conjecture in reverse. I came across this argument that attempts to show that an integer $4x+1$ resolves in $n$ steps if $x$ is odd and resolves in $n$ steps and likewise if $x$ is odd an resolves in $n$ steps, then $4x+1$ likewise resolves in $n$ steps.
Is my reasoning correct?
Let:

$\nu_2(x)$ be the 2-adic valuation of $x$
$x_1, x_2, \dots, x_n$ be $n$ distinct odd integers such that:

$x_{i+1} = \dfrac{3x_i + 1}{2^{\nu_2(3x_i+1)}}$

$x_1$ resolves in $n$ steps if $\dfrac{3x_n+1}{2^{\nu_2(3x_n + 1)}} = 1$

Observation 1:  if an odd integer $x$ resolves to $1$ in $n$ steps, then there exist $n$ positive integers $a_1, a_2, \dots, a_n$ where $x = f(a_1, a_2, \dots, a_n)$
(1)  If $n=1$, then $\dfrac{3x+1}{2^{a_1}} = 1$ and $x = f(a_1) = \dfrac{2^{a_1}-1}{3}$
(2)  Assume that for any $x$ that resolves in $n-1$ steps, there exist $n-1$ integers $a_1, \dots, a_{n-1}$ with $f_{n-1}(a_1, a_2, \dots, a_{n-1})= x$
(3)  If $x$ resolves in $n$ steps, there exists a positive integer $a_n$ such that $\dfrac{3x+1}{2^{a_n}}$ is an integer that resolves in $n-1$ steps and $x = f(a_1, a_2, \dots, a_n) = \dfrac{3f_{n-1}(a_1, a_2, \dots, a_{n-1}) + 1}{2^{a_n}}$
Observation 2: if an odd integer $x$ resolves in $n$ steps, then it follows that $4x+1$ also resolves in $n$ steps
(1) From Observation 1, there exist $a_1, a_2, \dots, a_n$ such that $\dfrac{3x+1}{2^{a_n}} = f_{n-1}(a_1, \dots, a_{n-1})$
(2) $\dfrac{3(4x+1)+1}{2^{a_n+2}} = \dfrac{12x+4}{2^{a_n+2}} = \dfrac{4(3x+1)}{2^{a_n+2}} = f_{n-1}(a_1, \dots, a_{n-1})$
Corollary: If $x$ is a positive odd integer and $4x+1$ is a solution in $n$ steps, then $x$ is also a solution in $n$ steps:
By assumption, there exist $a_1, a_2, \dots, a_n$ such that:
$$\frac{3(4x+1)+1}{2^{a_n}} = \frac{4(3x+1)}{2^{a_n}} = \frac{3x+1}{2^{a_n-2}}= f_{n-1}(a_1, a_2, \dots, a_{n-1})$$

Update 1: corrected an ambiguity -- my definitions only apply to odd numbers.
Update 2: I wrote a simple java app to validate.  I have been able to validate this result up to $90\times10^6$.
My code iterates through odd numbers $x$ and validates that the count of steps for $x$ matches the count of steps for $4x+1$.
Comment: Without weighing in on the details of the argument (I'm so out of touch on details) I would wonder if this conjecture has been attempted to be verified empirically for at least some range of $x$, and if so, how large that range would be.
Comment: @leslietownes I confirmed it up to $90\times10^6$.  Thanks very much for the suggestion.
Comment: Cool!  I just love anything empirical before I worry about formulas. Even if being empirical is dumber than formulas and goes nowhere, I feel like there's no substitute for the "real thing."
Answer: If $x$ is odd then after one (×3+1) step we have $3x+1$. We then complete the op's step by applying $k$ (/2) steps. Where $k$ is the largest integer such that $2^k|3x+1$. $4x+1$ is also odd so after one (×3+1) step we have $12x+4$ then if we apply two (/2) steps we get $3x+1$ then we divide by $2^k$ and get to the same number. So after one of the op's steps $x$ and $4x+1$ arrive at the same number $\frac{3x+1}{2^k}$.


Ranking batters by average when number of observations (innings) vary

Question: This is my first post here and I'm new to this area so please forgive me if I'm asking a naive question.
I want to rank a number of batsmen (e.g., in cricket) by their skill. I'm planning to use batting averages as a measure of skill.
But there are some batsmen who have only been to bat a few times and therefore their average is not a good estimate of performance (I.e., I don't have a very good estimate of their "true" average).
Assuming that, for each batsman, I only have scores of each time he came to bat (e.g., 89 4, 92, 45, ... ).
How could I best sort them?
Thanks in advance.
Answer: "Best" is impossible to answer since you don't give a criterion you want to optimize (that is, until you define pretty precisely what 'better' is for your purpose, 'best' isn't really possible to judge).
Trying to rank things with very different numbers of (/ precision of) observations is a common problem (think online ratings for movies for example), so you might search around for solutions to quite different problems with the same issue.
I'll present one approach I've seen used:
Take the observed value as as estimate of a population quantity (the batter's true ability*) and so calculate a one-sided interval for the quantity (a lower bound), and then order by that. You could call it an "experience-adjusted average" (or more accurately, inexperience-adjusted), which downrates the average more for less experience.
* This assumes we're not dealing with a shifting target -- if we are (and you'd expect with batting in any sport that it would shift over time), then you might weight by recency as well, such as an exponentially weighted moving average. This will still work with the above suggested approach.
Whether this suits your purpose or not, I can't say, but with a reasonably chosen interval  it often doesn't perform too badly (e.g. perhaps an 75% or an 80% interval might work okay; you can try other values).
Comment: I was thinking about asking a similar question at some point. This is what I currently do in tables as well, but I was wondering if there is a quick and dirty way to do empirical Bayes shrinkage with rates. (I will have to re-read those Efron and Morris papers on the James-Stein estimator more closely.)
Comment: @AndyW It sounds like it might make for a better answer than this one. Actually I think a Bayesian approach to this problem would probably make more intuitive sense, so perhaps it might also be better in terms of explaining it to other people.
Comment: Glen_b and Andy W, thanks for your answers.
In my problem we can assume a static target per batsmen (i.e., the skill I'm ranking them on does not vary over time).

@Glen_b, do you have any pointers to one of the movie ranking methods you mentioned in your answer?
Comment: @Akanes - the first time I saw this referenced was by [Evan Miller](http://www.evanmiller.org/how-not-to-sort-by-average-rating.html) for online product reviews with +/- votes. It is such a simple concept though I'm sure it has been suggested multiple times independently. (The idea extends to really any numeric value, not just rates.)


How to resolve "Internal server error" using mod_wsgi with web.py

Question: I'm trying to configure mod_wsgi to work with a trivial web.py python script. I followed this cookbook: http://webpy.org/cookbook/mod_wsgi-apache
Every time i go to https://<server>/appname/ chrome says Internal Server Error
Here are my config details:
in httpd.conf I have
LoadModule wsgi_module modules/mod_wsgi.so

WSGIScriptAlias /appname /var/www/webpy-app/cody.py/
Alias /appname /var/www/webpy-app/static/
AddType text/html .py

<Directory /var/www/webpy-app/>
  Order deny,allow
  Allow from all
</Directory>

<Location /appname>
AuthType Basic
AuthName "Authenication Required"
AuthUserFile "/etc/httpd/conf/some_sample_users"
</Location>

Here is the code:
  import web

  urls = (
      '/.*', 'hello',
      )

  class hello:
      def GET(self):
          return "Hello, world."

  application = web.application(urls, globals()).wsgifunc()

I checked the error logs but didnt find much in there:
[Wed Oct 09 02:24:50 2013] [notice] caught SIGTERM, shutting down
[Wed Oct 09 02:24:55 2013] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Wed Oct 09 02:24:55 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Oct 09 02:24:55 2013] [warn] module wsgi_module is already loaded, skipping
[Wed Oct 09 02:24:55 2013] [notice] Digest: generating secret for digest authentication ...
[Wed Oct 09 02:24:55 2013] [notice] Digest: done
[Wed Oct 09 02:24:55 2013] [notice] Apache/2.2.15 (Unix) mod_wsgi/3.2 Python/2.6.6 DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations

What modifications would I need to get this working?
Answer: This is wrong for a start:
WSGIScriptAlias /appname /var/www/webpy-app/cody.py/
Alias /appname /var/www/webpy-app/static/

The Alias directive takes precedence over WSGIScriptAlias. You have them for the same mount point, so Alias wins. I would suggest commenting out Alias for a start.
As well as that, having a trailing slash on second argument and not the first when mounting at sub URL can cause issues. Use:
WSGIScriptAlias /appname /var/www/webpy-app/cody.py

Next problem is:
[Wed Oct 09 02:24:55 2013] [warn] module wsgi_module is already loaded, skipping

This indicates you have:
LoadModule wsgi_module modules/mod_wsgi.so

listed twice in different places in Apache configuration, or you have managed to include a configuration file twice that has it listed.
Fix those issues first and then edit you question with updated information.
Comment: fixing those brought more relevant answers in play. I installed mod_wsgi using yum, however, it used the default python2.6. Now I need to remove that and install mod_wsgi from source so that it can use python2.7
Answer: I follow this tutorial which is complete step to run web.py without .htaccess,
but remember when you copy paste the python code you must watch the indentation, because it will show Internal server error as well. this is how httpd.conf looks like
<VirtualHost _default_ *:80>
ServerAdmin firstname.lastname@example.com
DocumentRoot /var/www/project.com/public_html/
ErrorLog /var/www/project.com/logs/error.log
CustomLog /var/www/project.com/logs/access.log combined
WSGIScriptAlias / /var/www/project.com/production/main.py
Alias /static /var/www/project.com/public_html
AddType text/html .py
WSGIDaemonProcess www-data threads=15
WSGIProcessGroup www-data
    <Directory /var/www/project.com/production/>
        Order deny,allow
        Allow from all
        Options +FollowSymLinks
        Options -Indexes
    </Directory>
</VirtualHost>

with main.py as index file


Função de erro complementar em java?

Question: Como calcular função de erro complementar em java ?
Comment: Veja aí: https://commons.apache.org/proper/commons-math/jacoco/org.apache.commons.math3.special/Erf.java.html
Comment: Já agora: erfc(x)=1-erf(x).
Comment: @JoseVieiraNeto  Porque a função erfc(x) é definida a partir da função erf(x). A primeira é apenas uma conveniência para não se estar sempre a escrever 1-erf(x). A função erro (erf) permite calcular a probabilidade segundo a distribuição normal. A erfc é o complementar dessa probabilidade.
Comment: Amigo, tem algum exemplo do que já fez? Ou de como é esta função? Ficaria mais fácil de ajudar!
Comment: Editei a pergunta dá uma olhada pra ver se ajuda.
Comment: @ThiagoLuizDomacoski é essa função do excel "erfc" quero saber de como fazer ela com java. o nome dela é função de erro complementar. não estou encontrando nada
Comment: Alexandre funcionou cara . mas to curioso com uma coisa por que sempre temos que subtrair por menos 1 ?
Answer: Adicione essa dependência ao projeto.
<code>   <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-math3</artifactId>
        <version>3.2</version>
    </dependency>
</code>
Basta chamar esta função.
<code>double vlr = Erf.erf(x);
System.out.println("valor" + vlr);
</code>

Função de erro complementar em java?


Expectation of the squared difference of two Gaussian random variables?

Question: Consider two independent standard Gaussian random variables $X \sim \mathcal{N}(0, 1)$ and $Y \sim \mathcal{N}(0, 1)$ and their sum $Z = X +Y$. We know that $Z \sim \mathcal{N}(0, 2)$. Now consider the problem of deriving $\mathbb{E}[(Z - aY)^2]$ for some scalar $a \in \mathbb{R}$.
To begin, I attempted to expand as follows:
$$\mathbb{E}[(Z - aY)^2] = \mathbb{E}[Z^2] - 2a\mathbb{E}[ZY] + \mathbb{E}[Y^2].$$
The first and final terms are easily computed using the definition of variance; however, I cannot seem to be able to find out what $\mathbb{E}[ZY]$ is. Clearly $Z$ and $Y$ are not independent (right? since knowing $Y$ would shift the distribution on $Z$).
Not sure where to go from here, do I want to resort to the definition of expectation or is there something easier?
Comment: $Z-aY=X+(1-a)Y$ has a Normal distribution with mean $0$ and variance $1+(1-a)^2,$ as you can readily calculate.
Comment: @whuber Doh! Thanks!
Comment: @whuber Aren't you jumping the gun a little in deducing that $Z$ is normal? The OP doesn't state that $X$ and $Y$ are _jointly_ normal (unless you think that the Wikipedia page that the OP links to is a sufficient indication that he has joint normality in mind).
Comment: I had independence of $X$ and $Y$ in mind, which would imply that they are jointly normal. I will add that to the question.
Comment: @Dilip I considered that, but noticed that the question also stipulates that $Z$ has a Normal distribution.  I took that as the "sufficient indication" you seek.
Answer: Now that the OP has confessed that $X$ and $Y$ have been independent $\mathcal N(0,1)$ random variables all along, which of course implies that $Z=X+Y \sim \mathcal N(0,2)$, consider that
$E[Z-aY] = E[Z]-aE[Y] = 0$ and so
\begin{align}E[(Z-aY)^2 &= \operatorname{var}(Z-aY)\\
&= \operatorname{var}(X+Y-aY)\\
&= \operatorname{var}(X+(1-a)Y)\\
&= \operatorname{var}(X) + (1-a)^2\operatorname{var}(Y) &{\scriptstyle{\text{since}~X~\text{and}~aY~\text{are independent random variables}}}\\
& &{\scriptstyle{\text{and}~\operatorname{var}((1-a)Y)~ = ~(1-a)^2\operatorname{var}(Y).}}
\end{align}
And of course, $Z-aY$ is a normal random variable.
Comment: It was never necessary to assume independence: from the original statement that $Z$ has a distribution with variance $2$ you can obtain the answer, because it implies $X$ and $Y$ are uncorrelated and that suffices for your calculation.