Rsync triggered Linux OOM killer on a single 50 GB file

Question: I have a single 50 GB file on server_A, and I'm copying it to server_B. I run
server_A$ rsync --partial --progress --inplace --append-verify 50GB_file root@server_B:50GB_file

Server_B has 32 GB of RAM with 2 GB swap. It is mostly idle and should have had lots of free RAM. It has plenty of disk space. At about 32 GB, the transfer aborts because the remote side closed the connection.
Server_B has now dropped off the network. We ask the data center to reboot it. When I look at the kernel log from before it crashed, I see that it was using 0 bytes of swap, and the process list was using very little memory (the rsync process was listed as using 600 KB of RAM), but the oom_killer was going wild, and the last thing in the log is where it kills metalog's kernel reader process.
This is kernel 3.2.59, 32-bit (so no process can map more than 4 GB anyway).
It's almost as if Linux gave more priority to caching than to long-lived running daemons. What gives?? And how can I stop it from happening again?
Here is the output of the oom_killer:
Sep 23 02:04:16 [kernel] [1772321.850644] clamd invoked oom-killer: gfp_mask=0x84d0, order=0, oom_adj=0, oom_score_adj=0
Sep 23 02:04:16 [kernel] [1772321.850649] Pid: 21832, comm: clamd Tainted: G         C   3.2.59 #21
Sep 23 02:04:16 [kernel] [1772321.850651] Call Trace:
Sep 23 02:04:16 [kernel] [1772321.850659]  [<c01739ac>] ? dump_header+0x4d/0x160
Sep 23 02:04:16 [kernel] [1772321.850662]  [<c0173bf3>] ? oom_kill_process+0x2e/0x20e
Sep 23 02:04:16 [kernel] [1772321.850665]  [<c0173ff8>] ? out_of_memory+0x225/0x283
Sep 23 02:04:16 [kernel] [1772321.850668]  [<c0176438>] ? __alloc_pages_nodemask+0x446/0x4f4
Sep 23 02:04:16 [kernel] [1772321.850672]  [<c0126525>] ? pte_alloc_one+0x14/0x2f
Sep 23 02:04:16 [kernel] [1772321.850675]  [<c0185578>] ? __pte_alloc+0x16/0xc0
Sep 23 02:04:16 [kernel] [1772321.850678]  [<c0189e74>] ? vma_merge+0x18d/0x1cc
Sep 23 02:04:16 [kernel] [1772321.850681]  [<c01856fa>] ? handle_mm_fault+0xd8/0x15d
Sep 23 02:04:16 [kernel] [1772321.850685]  [<c012305a>] ? do_page_fault+0x20e/0x361
Sep 23 02:04:16 [kernel] [1772321.850688]  [<c018a9c4>] ? sys_mmap_pgoff+0xa2/0xc9
Sep 23 02:04:16 [kernel] [1772321.850690]  [<c0122e4c>] ? vmalloc_fault+0x237/0x237
Sep 23 02:04:16 [kernel] [1772321.850694]  [<c08ba7e6>] ? error_code+0x5a/0x60
Sep 23 02:04:16 [kernel] [1772321.850697]  [<c08b0000>] ? cpuid4_cache_lookup_regs+0x372/0x3b2
Sep 23 02:04:16 [kernel] [1772321.850700]  [<c0122e4c>] ? vmalloc_fault+0x237/0x237
Sep 23 02:04:16 [kernel] [1772321.850701] Mem-Info:
Sep 23 02:04:16 [kernel] [1772321.850703] DMA per-cpu:
Sep 23 02:04:16 [kernel] [1772321.850704] CPU    0: hi:    0, btch:   1 usd:   0
Sep 23 02:04:16 [kernel] [1772321.850706] CPU    1: hi:    0, btch:   1 usd:   0
Sep 23 02:04:16 [kernel] [1772321.850707] CPU    2: hi:    0, btch:   1 usd:   0
Sep 23 02:04:16 [kernel] [1772321.850709] CPU    3: hi:    0, btch:   1 usd:   0
Sep 23 02:04:16 [kernel] [1772321.850711] CPU    4: hi:    0, btch:   1 usd:   0
Sep 23 02:04:16 [kernel] [1772321.850713] CPU    5: hi:    0, btch:   1 usd:   0
Sep 23 02:04:16 [kernel] [1772321.850714] CPU    6: hi:    0, btch:   1 usd:   0
Sep 23 02:04:16 [kernel] [1772321.850716] CPU    7: hi:    0, btch:   1 usd:   0
Sep 23 02:04:16 [kernel] [1772321.850718] Normal per-cpu:
Sep 23 02:04:16 [kernel] [1772321.850719] CPU    0: hi:  186, btch:  31 usd:  70
Sep 23 02:04:16 [kernel] [1772321.850721] CPU    1: hi:  186, btch:  31 usd: 116
Sep 23 02:04:16 [kernel] [1772321.850723] CPU    2: hi:  186, btch:  31 usd: 131
Sep 23 02:04:16 [kernel] [1772321.850724] CPU    3: hi:  186, btch:  31 usd:  76
Sep 23 02:04:16 [kernel] [1772321.850726] CPU    4: hi:  186, btch:  31 usd:  29
Sep 23 02:04:16 [kernel] [1772321.850728] CPU    5: hi:  186, btch:  31 usd:  61
Sep 23 02:04:16 [kernel] [1772321.850731] CPU    7: hi:  186, btch:  31 usd:  17
Sep 23 02:04:16 [kernel] [1772321.850733] HighMem per-cpu:
Sep 23 02:04:16 [kernel] [1772321.850734] CPU    0: hi:  186, btch:  31 usd:   2
Sep 23 02:04:16 [kernel] [1772321.850736] CPU    1: hi:  186, btch:  31 usd:  69
Sep 23 02:04:16 [kernel] [1772321.850738] CPU    2: hi:  186, btch:  31 usd:  25
Sep 23 02:04:16 [kernel] [1772321.850739] CPU    3: hi:  186, btch:  31 usd:  27
Sep 23 02:04:16 [kernel] [1772321.850741] CPU    4: hi:  186, btch:  31 usd:   7
Sep 23 02:04:16 [kernel] [1772321.850743] CPU    5: hi:  186, btch:  31 usd: 188
Sep 23 02:04:16 [kernel] [1772321.850744] CPU    6: hi:  186, btch:  31 usd:  25
Sep 23 02:04:16 [kernel] [1772321.850746] CPU    7: hi:  186, btch:  31 usd: 158
Sep 23 02:04:16 [kernel] [1772321.850750] active_anon:117913 inactive_anon:9942 isolated_anon:0
Sep 23 02:04:16 [kernel] [1772321.850751]  active_file:106466 inactive_file:7784521 isolated_file:0
Sep 23 02:04:16 [kernel] [1772321.850752]  unevictable:40 dirty:0 writeback:61 unstable:0
Sep 23 02:04:16 [kernel] [1772321.850753]  free:143494 slab_reclaimable:128312 slab_unreclaimable:4089
Sep 23 02:04:16 [kernel] [1772321.850754]  mapped:6706 shmem:308 pagetables:915 bounce:0
Sep 23 02:04:16 [kernel] [1772321.850759] DMA free:3624kB min:140kB low:172kB high:208kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolate
d(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:240kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tm
p:0kB pages_scanned:0 all_unreclaimable? yes
Sep 23 02:04:16 [kernel] [1772321.850763] lowmem_reserve[]: 0 869 32487 32487
Sep 23 02:04:16 [kernel] [1772321.850770] Normal free:8056kB min:8048kB low:10060kB high:12072kB active_anon:0kB inactive_anon:0kB active_file:248kB inactive_file:388kB unevictable:0kB isolated(anon)
:0kB isolated(file):0kB present:890008kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:513008kB slab_unreclaimable:16356kB kernel_stack:1888kB pagetables:3660kB unstable:0
kB bounce:0kB writeback_tmp:0kB pages_scanned:1015 all_unreclaimable? yes
Sep 23 02:04:16 [kernel] [1772321.850774] lowmem_reserve[]: 0 0 252949 252949
Sep 23 02:04:16 [kernel] [1772321.850785] lowmem_reserve[]: 0 0 0 0
Sep 23 02:04:16 [kernel] [1772321.850788] DMA: 0*4kB 7*8kB 3*16kB 6*32kB 4*64kB 6*128kB 5*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 3624kB
Sep 23 02:04:16 [kernel] [1772321.850795] Normal: 830*4kB 80*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8056kB
Sep 23 02:04:16 [kernel] [1772321.850802] HighMem: 13*4kB 14*8kB 2*16kB 2*32kB 0*64kB 0*128kB 2*256kB 2*512kB 3*1024kB 0*2048kB 136*4096kB = 561924kB
Sep 23 02:04:16 [kernel] [1772321.850809] 7891360 total pagecache pages
Sep 23 02:04:16 [kernel] [1772321.850811] 0 pages in swap cache
Sep 23 02:04:16 [kernel] [1772321.850812] Swap cache stats: add 0, delete 0, find 0/0
Sep 23 02:04:16 [kernel] [1772321.850814] Free swap  = 1959892kB
Sep 23 02:04:16 [kernel] [1772321.850815] Total swap = 1959892kB
Sep 23 02:04:16 [kernel] [1772321.949081] 8650736 pages RAM
Sep 23 02:04:16 [kernel] [1772321.949084] 8422402 pages HighMem
Sep 23 02:04:16 [kernel] [1772321.949085] 349626 pages reserved
Sep 23 02:04:16 [kernel] [1772321.949086] 7885006 pages shared
Sep 23 02:04:16 [kernel] [1772321.949087] 316864 pages non-shared
Sep 23 02:04:16 [kernel] [1772321.949089] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
            (rest of process list omitted)
Sep 23 02:04:16 [kernel] [1772321.949656] [14579]     0 14579      579      171   5       0             0 rsync
Sep 23 02:04:16 [kernel] [1772321.949662] [14580]     0 14580      677      215   5       0             0 rsync
Sep 23 02:04:16 [kernel] [1772321.949669] [21832]   113 21832    42469    37403   0       0             0 clamd
Sep 23 02:04:16 [kernel] [1772321.949674] Out of memory: Kill process 21832 (clamd) score 4 or sacrifice child
Sep 23 02:04:16 [kernel] [1772321.949679] Killed process 21832 (clamd) total-vm:169876kB, anon-rss:146900kB, file-rss:2712kB

Here is the 'top' output after repeating my rsync command as a non-root user:
top - 03:05:55 up  8:43,  2 users,  load average: 0.04, 0.08, 0.09
Tasks: 224 total,   1 running, 223 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0% us,  0.0% sy,  0.0% ni, 99.9% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:  33204440k total, 32688600k used,   515840k free,   108124k buffers
Swap:  1959892k total,        0k used,  1959892k free, 31648080k cached

Here are the sysctl vm parameters:
# sysctl -a | grep '^vm'
vm.overcommit_memory = 0
vm.panic_on_oom = 0
vm.oom_kill_allocating_task = 0
vm.oom_dump_tasks = 1
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.dirty_background_ratio = 1
vm.dirty_background_bytes = 0
vm.dirty_ratio = 0
vm.dirty_bytes = 15728640
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 3000
vm.nr_pdflush_threads = 0
vm.swappiness = 60
vm.lowmem_reserve_ratio = 256   32      32
vm.drop_caches = 0
vm.min_free_kbytes = 8192
vm.percpu_pagelist_fraction = 0
vm.max_map_count = 65530
vm.laptop_mode = 0
vm.block_dump = 0
vm.vfs_cache_pressure = 100
vm.legacy_va_layout = 0
vm.stat_interval = 1
vm.mmap_min_addr = 4096
vm.vdso_enabled = 2
vm.highmem_is_dirtyable = 0
vm.scan_unevictable_pages = 0

Comment: I'm no expert at reading kernel crash messages, but I can't see any indication in there that B was using 32GB of core.  Before we proceed on the assumption that it was, can you confirm that it currently is?  Because there's a big difference between memory-exhausting a box with 32GB of core, and one with only 4GB.
Comment: @dataless I think Olivier_Dulac is refering to [man swapon](http://linux.die.net/man/8/swapon)
Comment: Show us your swapiness parameter.
Comment: Updated with Top output.  This is after running that same rsync command as a non-root user.  Pretty much all but 1GB is used for cache right now.
Comment: Thanks.  As I said, I'm no expert - but it seemed worth checking.
Comment: you should also verify that your kernel does allow swapping (ie, swapping is not turned off) (and you should dedicated a bigger chunk of disk space, let's say 16Gb or even 32Gb). Some weird persons on the net recommend to turn swapping off, which is very wrong.
Comment: @OlivierDulac what setting are you referring to?  swap support is compiled in or we wouldn't be able to mount swap, and the 'swappiness' is set to 60.  As for swap size, wouldn't that just make the problem worse, on a 32-bit kernel?  The answer appears that kernel data structures were what killed us.  We aren't running 32GB of user processes, we just want that much ram for disk cache, for performance.
Answer: So let us read the oom-killer output and see what can be learned from there.
When analyzing OOM killer logs, it is important to look at what triggered it. The first line of your log gives us some clues:

[kernel] [1772321.850644] clamd invoked oom-killer: gfp_mask=0x84d0, order=0

order=0 is telling us how much memory is being requested. The kernel's memory management is only able to manage page numbers in the powers of 2, so clamd has requested 20 pages of memory or 4KB.
The lowest two bits of the GFP_MASK (get free page mask) constitute the so-called zone mask telling the allocator which zone to get the memory from:
Flag            value      Description
                0x00u      0 implicitly means allocate from ZONE_NORMAL
__GFP_DMA       0x01u      Allocate from ZONE_DMA if possible
__GFP_HIGHMEM   0x02u      Allocate from ZONE_HIGHMEM if possible

Memory zones is a concept created mainly for compatibility reasons. In a simplified view, there are three zones for an x86 Kernel:
Memory range   Zone       Purpose

0-16 MB        DMA        Hardware compatibility (devices)
16 - 896 MB    NORMAL     space directly addressable by the Kernel, userland
> 896 MB       HIGHMEM    userland, space addressable by the Kernel via kmap() calls

In your case, the zonemask is 0, meaning clamd is requesting memory from ZONE_NORMAL.
The other flags are resolving to
/*
 * Action modifiers - doesn't change the zoning
 *
 * __GFP_REPEAT: Try hard to allocate the memory, but the allocation attempt
 * _might_ fail.  This depends upon the particular VM implementation.
 *
 * __GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
 * cannot handle allocation failures.
 *
 * __GFP_NORETRY: The VM implementation must not retry indefinitely.
 */
#define __GFP_WAIT      0x10u   /* Can wait and reschedule? */
#define __GFP_HIGH      0x20u   /* Should access emergency pools? */
#define __GFP_IO        0x40u   /* Can start physical IO? */
#define __GFP_FS        0x80u   /* Can call down to low-level FS? */
#define __GFP_COLD      0x100u  /* Cache-cold page required */
#define __GFP_NOWARN    0x200u  /* Suppress page allocation failure warning */
#define __GFP_REPEAT    0x400u  /* Retry the allocation.  Might fail */
#define __GFP_NOFAIL    0x800u  /* Retry for ever.  Cannot fail */
#define __GFP_NORETRY   0x1000u /* Do not retry.  Might fail */
#define __GFP_NO_GROW   0x2000u /* Slab internal usage */
#define __GFP_COMP      0x4000u /* Add compound page metadata */
#define __GFP_ZERO      0x8000u /* Return zeroed page on success */
#define __GFP_NOMEMALLOC 0x10000u /* Don't use emergency reserves */
#define __GFP_NORECLAIM  0x20000u /* No realy zone reclaim during allocation */

according to the Linux MM documentation, so your requst has the flags for GFP_ZERO, GFP_REPEAT, GFP_FS, GFP_IO and GFP_WAIT, thus being not particularly picky.
So what's up with ZONE_NORMAL? Some generic stats can be found further on in the OOM output:

[kernel] [1772321.850770] Normal free:8056kB min:8048kB low:10060kB high:12072kB active_anon:0kB inactive_anon:0kB active_file:248kB inactive_file:388kB unevictable:0kB isolated(anon)
  :0kB isolated(file):0kB present:890008kB

Noticeable here is that free is just 8K from min and way under low. This means your host's memory manager is somewhat in distress and kswapd should be swapping out pages already as it is in the yellow phase of the graph below:

Some more information on the memory fragmentation of the zone is given here:

[kernel] [1772321.850795] Normal: 830*4kB 80*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8056kB

basically stating that you have a single contiguous page of 4MB with the rest heavily fragmented into mainly 4KB pages.
So let's recapitulate:

you have a userland process (clamd) getting memory from ZONE_NORMAL whereas non-privileged memory allocation usually would be performed from ZONE_HIMEM
the memory manager should at this stage have been able to serve the requested 4K page, although you seem to have significant memory pressure in ZONE_NORMAL
the system, by kswapd's rules, should have seen some paging activity beforehand, but nothing is being swapped out, even under memory pressure in ZONE_NORMAL, without apparent cause
None of the above gives a definite reason as for why oom-killer has been invoked

All of this seems rather odd, but is at least to be related to what is described in section 2.5 of John O'Gorman's excellent "Understanding the Linux Virtual Memory Manager" book:

As the addresses space usable by the kernel (ZONE_NORMAL) is limited in size, the kernel has support for the concept of High Memory. [...] To access memory between the range of 1GiB and 4GiB, the kernel temporarily maps pages from high memory into ZONE_NORMAL with kmap(). [...]
That means that to describe 1GiB of memory, approximately 11MiB of kernel memory is required. Thus, with 16GiB, 176MiB of memory is consumed, putting significant pressure on ZONE_NORMAL. This does not sound too bad until other structures are taken into account which use ZONE_NORMAL. Even very small structures such as Page Table Entries (PTEs) require about 16MiB in the worst case. This makes 16GiB about the practical limit for available physical memory Linux on an x86.

(emphasis is mine)
Since 3.2 has numerous advancements in memory management over 2.6, this is not a definite answer, but a really strong hint I would pursue first. Reduce the host's usable memory to at most 16G by either using the mem= kernel parameter or by ripping half of the DIMMs out of the server.
Ultimately, use a 64-bit Kernel.
Dude, it's 2015.
Comment: That was beautiful. There still is hope for SF.
Comment: When I said above "*I'm no expert*", **this** is what I was hoping to read. +1000, if I but could; +1 for sure.  What a great answer!
Comment: @dataless Yes. I suspect that all of your ZONE_NORMAL is filled with metadata about the upper memory regions. When a userland process is requesting memory pages, it most likely is going to request ZONE_HIGHMEM (which might be upgraded by the MM to ZONE_NORMAL if HIGHMEM has no more free pages to serve the request but NORMAL has), so unless ZONE_HIGHMEM is under memory pressure (yours isn't), ZONE_NORMAL will have no user-space processes' pages.
Comment: @the-wabbit Hot network questions.
Comment: [slams fists on keyboard] Give this wabbit the bounty
Comment: just wow, nice work.
Comment: @the-wabbit Might be that common knowledge is "Linux x86 with PAE can go up to 64GB".  But my problem and your answer just burst everyone's bubble!  Also you quoted half a dozen sources and used a graph :-)
Comment: +401 for giving me a whole slew of material I need to learn and great references for all.
Comment: OMG, what's happening with this post? I never suspected ServerFault even to *have* as much users, let alone get them to read *and* vote on this narrow topic. Have I unintentionally posted to [SO] instead? Or is there a funny kitten meme somewhere in there I am unable to see?
Comment: Upvoting because of the conclusion
Comment: Awesome!  I was hoping someone here would be able to make sense of the lower-level bits of that data.  I guess that explains why it wasn't swapping?  because in order to swap the kernel would need to allocate even more data structures to describe the swap that had been written out, and the data structures were what was filling RAM already?
Comment: really great explanation @the-wabbit,
can you share the source from where you got the above pic, would like to read this in detail . Thanks
Comment: @DarkKnight just klick the image, it hyperlinks the source.
Answer: A few things ...
My rule of thumb for swap space has been to have at least 2x the amount of physical ram.  This allows the page/swap daemon the ability to reorg memory efficiently.
Server_B has 32GB of ram, so try configuring it for 64GB of swap.  IMO, the 2GB of swap space your server has is way too low, especially for a server.
If you don't have an extra partition that you can make into a swap partition, you can test this by creating a file and mounting it as a swap partition [it will be slow].  See https://www.maketecheasier.com/swap-partitions-on-linux/
Since server_B has plenty of disk space, --inplace is not needed, and may be undesired as it may be what is causing rsync to use 32GB.  --inplace is only really useful if you're short on filesystem space [which you aren't] or have some special performance requirement.
My guess is that rsync will want to use 50GB of ram [the file size] with your current options.  Normally, rsync doesn't need that much memory to do its job, so one or more of your options may be the problem.  I routinely transfer 200GB files without a problem.
Do some test runs using no options.  Do this with smaller files, say 10GB--this should prevent the kernel panic, but still allow you to monitor the behavior that is causing the problem.  Monitor rsync's memory usage.
Gradually, add back options, one at a time, to see which option [or combination of options] causes rsync to start pigging out on RAM (e.g. while the transfer is happening, rsync's ram usage grows proportionately to the amount of file data transferred, etc.).
If you truly need the options that cause rsync to keep some in-ram file image, you'll need the extra swap space and your maximum file size will be limited accordingly.
A few more things [UPDATED]:
(1) The kernel stack traceback shows that rsync was page faulting on an mmap area.  It is probably mmaping the file.  mmap offers no guarantee that it will flush to disk until the file is closed [unlike read/write] which goes to the FS block cache right away [where it will be flushed]
(2) The kernel crash/panic is occurring when the transfer size hits the size of RAM.  Clearly rsync is grabbing that much non-fscache memory via malloc or mmap.  Once again, with the options you've specified, rsync will allocate 50GB of memory to transfer a 50GB file.
(3) Transfer a 24GB file.  That will probably work. Then, boot the kernel with mem=16G and do the 24GB file test again.  It will blow out at 16GB rather than 32GB.  This will confirm that rsync really needs the memory.
(4) Before you say that adding swap is ridiculous, try adding some [via the swap-to-file method].  This is far easier to do and test than all the academic arguments about how swap isn't required.  Even if it's not the solution, you may learn something from it.  I'll bet that mem=16G test will succeed without a panic/crash.
(5) Chances are that rsync is hitting swap, but it happens too fast to see with top before OOM kicks in and kills rsync.  By the time rsync gets to 32GB, other processes have already been forced out to swap, particularly if they're idle.  Perhaps, a combination of "free" and "top" will give you a better picture.
(6) After rsync gets killed, it takes time to flush mmap to the FS.  Not fast enough for OOM and it starts killing other things [some are obviously mission critical].  That is, the mmap flush and OOM are racing.  Or, OOM has a bug.  Otherwise, there wouldn't be a crash.
(7) In my experience, once a system "hits the memory wall", Linux takes a long time to fully recover.  And, sometimes it never really recovers properly and the only way to clear it is a reboot.  For example, I have 12GB of RAM.  When I run a job that uses 40GB of memory [I have 120GB of swap to accommodate large jobs] and then kill it, it takes about 10 minutes for the system to return to normal responsiveness [with the disk light on solid all the while].
(8) Run rsync without options.  This will work.  Get a baseline example to work from.  Then add back --inplace and retest.  Then do --append-verify instead. Then, try both.  Find out which option gets rsync doing the huge mmap.  Then decide if you can live without it.  If --inplace is the culprit, that's a no-brainer, since you've got plenty of disk space.  If you must have the option, you'll have to get the swap space to accommodate the malloc/mmap that rsync will do.
SECOND UPDATE:
Please do the mem= and smaller file tests from the above.
The central questions:  Why does rsync get killed by OOM?  Who/What is chewing memory?
I read [but forgot] about the system being 32 bit.  So, I agree, rsync may not be directly responsible (via malloc/mmap--glibc implements large mallocs via anonymous/private mmaps), and rsync's mmap page fault just triggers OOM by coincidence.  Then, OOM calculates total memory consumed by rsync directly and indirectly [FS cache, socket buffers, etc.] and decides it's the prime candidate. So, monitoring total memory usage may be helpful.  I suspect it creeps up at the same rate as the file transfer.  Obviously, it shouldn't.
Some things you can monitor in /proc or /proc/rsync_pid via a perl or python script in a fast loop [a bash script probably won't be fast enough for the end-of-the-world event] that can monitor all of the following several hundred times/sec.  You can run this at a higher priority than rsync so it will keep itself in RAM and running so you can monitor things just before the crash and hopefully during OOM so you can see why OOM goes crazy:
/proc/meminfo -- to get more fine grain on the swap usage at "point of impact".  Actually, getting the final number on how much RAM is being used total may be more useful.  While top provides this, it may not be fast enough to show the state of the universe just prior to the "big bang" (e.g. the last 10 milliseconds)
/proc/rsync_pid/fd directory.  Reading the symlinks will allow you to identify which fd is opened on the target file (e.g. readlink of /proc/rsync_pid/fd/5 --> target_file).  This probably only needs to be done once to get the fd number [it should stay fixed]
Knowing the fd number, look at /proc/rsync_pid/fdinfo/fd.  It's a text file that looks like:

pos: <file_position>
flags: blah_blah
mnt_id: blah_blah
Monitoring the "pos" value can be helpful as the "last file position" may be useful.  If you do several tests with varying sizes and mem= options, does the last file position track any of these [and how]?  The usual suspect:  file position == available RAM
But, the simplest way is to start with "rsync local_file server:remote_file" and verify that works. You may be able to get similar [but faster] results by doing "ssh server rsync file_a file_b" [you'd need to create a 50GB file_a first].  A simple way to create file_a is scp local_system:original_file server:file_a and this might be interesting unto itself (e.g. does this work when the rsync crashes? If scp works, but rsync fails, this points to rsync.  If scp fails, this points to something else like the NIC driver).  Doing the ssh rsync also takes the NIC out of the equation, which may be helpful.  If that hoses the system, then something is really wrong.  If it succeeds, [as I've mentioned] start adding back the options one-by-one.
I hate to belabor the point, but adding some swap via swap-to-file may change/delay the crash behavior and may be useful as a diagnostic tool. If adding, say 16GB, of swap delays the crash [as measured by memory usage or target file position] from 32GB to 46GB, then that will say something.
It may not be any particular process, but an errant kernel driver that is chewing memory.  The kernel's internal vmalloc allocates stuff and it can be swapped out. IIRC, it's not bound by addressibility under all circumstances.
Clearly, the OOM is getting confused/panicked.  That is, it kills rsync, but doesn't see the memory freed up in a timely manner and goes looking for other victims.  Some of them are probably critical to system operation.
malloc/mmap aside, this could be caused by unflushed FS cache that takes a long time (e.g. with 30GB of unflushed data, assuming a disk rate of 300 MB/sec, it might take 100 seconds to flush it).  Even at that rate, OOM may be too impatient.  Or, OOM killing rsync doesn't start the FS flush fast enough [or at all].  Or FS flush happens fast enough, but it has a "lazy" release of the pages back to the free pool. There are some /proc options you can set to control FS cache behavior [I can't remember what they are].
Try booting with mem=4G or some other small number.  This might cut back on the FS cache and shorten its flush time to keep OOM from looking for other things to kill (e.g. flush time is reduced from 100 sec to < 1 sec).  It might also unmask an OOM bug that can't handle physical ram > 4GB on a 32 bit system or some such.
Also, an important point:  Run as non-root.  Root users are never expected to chew resources, so they are given more forgiving limits (e.g. 99% of memory vs 95% for non-root users).  This may explain why OOM is in such a state.  Also, this gives OOM et. al. more headroom to do its job of reclaiming memory.
Comment: swap > RAM is only really useful if you run without VM overcommit, so the kernel needs to reserve swap space for allocated pages until they are dirtied and need real physical pages backing them.  Current practice is to allow overcommit, and run with a small amount of swap space to page out pages that were only touched at startup and aren't needed in normal operation.  overcommit=0 with small swap is fine if you have a lot of RAM, for most systems.
Comment: @CraigEstey 64 GB of swap is completely ridiculous since as I stated earlier we do not have large user processes, we only want the disk cache, and as my log showed, we were using ZERO swap at the time of the crash.  ZERO.  Also rsync uses 600KB of ram even on a 50GB file.  My initial confusion was that maybe linux was agressively holding onto disk cache.  And finally, I want to see some numbers or documentation about how much memory the kernel uses to track its swap space before I would add any more to this box.
Comment: @CraigEstey I believe you've got  a misconception here: *"mmap offers no guarantee that it will flush to disk until the file is closed"* does not mean that a mmapped file's dirty pages *cannot* be flushed, it just means that they *may not* be flushed. Of course, they are flushed frequently and even force-flushed after the limit defined by either `vm.dirty_bytes` or `vm.dirty_ratio` has been exceeded or the time period defined by `vm.dirty_expire_centisecs` has passed.
Comment: @the-wabbit I said "no guarantee" but I never said "never".  But, the problem may be with the mmaps done in the last N seconds before the cataclysm.  I've updated my answer again to add some additional tests to try and it may not be mmaps delayed to FS cache, but FS cache flushing times/delays [which happens too slowly to satisfy OOM].  As I mention in the update, because of the 32 bit, rsync itself can't get malloc-crazy and cause the problem.  It may be FS cache, socket buffers, etc. that are lingering/leaking to cause memory to get chewed
Comment: @dataless I should have had a table of contents on my original post.  You would have found the rsync stuff midway down and realized that I did read what you wrote :-)  I've done some more thinking and I've done a second update with more triage tests you can try and some other things to monitor.  If you notice anything unusual, post the updated info.
Comment: See [How much SWAP space on a high memory system?](http://serverfault.com/q/5841) - and you don't want to see your system use 63GB of swap. It will not be usable.
Comment: It appears that rsync really _is_ trying to use >32GB, so the swap is needed for that.  That is, rsync is going to use 50GB for this file.The 2x has been a tried and true metric for 30 years.  Since a 6TB disk is ~$300, there's no reason not to.  What else might be running on this server that collectively will push over the RAM limit?
Comment: @dataless I've added additional information that fully explains what is happening and why.  rsync _is_ grabbing the memory via malloc/mmap and it will go for 50GB before it's done.  See the updated section.  It has tests that can prove/disprove what I'm saying and you can skip the added swap initially to test.  BTW, I've been programming kernels/drivers for 40+ years--I might just know something that you don't, so please moderate the tone--I _am_ trying to help.
Comment: @CraigEstey Sorry about the tone.  It just felt like you didn't really read my full question.  Rsync can't be mapping 50GB of memory because it only has 3GB of address space  (32-bit)  so clearly it must un-map at some point.  Also, when you mmap a file and write to it, it acts as its own "swap". (i.e. the memory blocks are already backed by a file, so they won't ever end up in the swap partition)  Also, the 'top' numbers I posted were after running the same rsync again as a non-root user, and it finished the 50GB file without the swap file ever being touched.
Comment: Some additional information I discovered after the-wabbit's post were that /proc/buddyinfo does show a lot of fragmentation after the transfer, and if I start writing a new 50GB file it gets worse.  When I ran the initial rsync, causing the crash (and by crash I mean oom-kill of entire userspace, not actually a kernel panic) the box had a few weeks of uptime and I suspect there was existing fragmentation that was made worse by all the new cached pages.
Answer: clamd? It sounds like you're using ClamAV and have on-access scanning enabled where the anti-virus engine attempts to scan opened files for viruses, by loading into memory, the entire contents of every file opened by any other process.
Depending on your security posture and the necessity of this transfer, you should evaluate disabling ClamAV on-access scanning while you perform the transfer.
Comment: The Linux kernel is reporting that clamd is the process whose memory allocation requests are invoking the OOM killer -- ClamAV is almost certainly the secondary cause (the primary cause being not enough memory). Whether it's on-access scanning or some other configuration, all signs point to ClamAV.
Comment: I wasn't aware this was even a thing clamav could do... but no we only scan specific files piped to it from clamc.  Also, 32-bit, so no danger of clamav hogging all the system memory.  (you see why we thought 32-bit was still a good idea?)
Comment: Next time you start the rsync, run top and monitor the resident size of the clamd process.
Comment: clamd was the first process to invoke the oom killer, but also the first to die to it since it weighs in at almost 42MB (one of the larger processes on the server).  The oom_killer runs repeatedly after this until even metalog gets killed.


Storyboard popover dismissed, delegate methods not called

Question: I have a view controller that's presented in a popover using a storyboard segue.

In the presenting view controller, I had the following code:
override func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject?) {
    if let svc = segue.destinationViewController as? SettingsViewController {
        svc.popoverPresentationController?.delegate = self
    }
}

However, it turns out that the presented view controller, even though it appears as a popover, has a modalPresentationStyle of '.Modal, and hence a nil popoverPresentationController. Weird!
So, I updated the code as follows:
override func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject?) {
    if let svc = segue.destinationViewController as? SettingsViewController {
        svc.modalPresentationStyle = .Popover
        svc.popoverPresentationController?.delegate = self
    }
}

The svc.popoverPresentationController delegate is now set OK, but if the popover is dismissed by the user tapping outside, none of the UIPopoverPresentationControllerDelegate delegate methods (e.g. popoverPresentationControllerShouldDismissPopover are called. What am I missing?
Comment: Probably a stupid question, are you sure the segue is set as "Present as Popover" in the storyboard?
Comment: Yes, definitely - and it presents correctly as a popover (and did even before setting `svc.modalPresentationStyle = .Popover`). Worth checking though!
Comment: One thing is weird, it should say "Present as Popover", as just "Popover" should be deprecated, but i don't think that's the issue...
Comment: "Present as Popover" is an option when size classes are enabled. Without that, it's just "Popover"
Comment: is `svc.popoverPresentationController` non-nil at that time? Else, it would silently not set the delegate.
Comment: It is non-nil in the second code snippet
Comment: @AshleyMills: Just wanting to check in to see if you have had a chance to test out the below answer, or if you are looking for something different.
Answer: No need for delegation in this case. If the presentingViewController (whatever vc is containing the popover) just overrides:
Swift 4
override func dismiss(animated flag: Bool, completion: (() -> Void)? = nil) {
    print("Dismiss: \(String(describing: self.presentedViewController))")
    super.dismiss(animated: flag, completion: completion)
}

Swift 3
override func dismissViewControllerAnimated(flag: Bool, completion: (() -> Void)?) {
    // Before calling super get a handle on which controller is being dismissed
    print("Dismiss: \(self.presentedViewController)")
    super.dismissViewControllerAnimated(flag, completion: completion)
}

You will get notified no matter how it is dismissed. You also do not need to set any additional variables/settings in the prepareForSegue: (at least to handle this interaction).
Comment: It's the only answer, so get the reward. Doesn't seem right though, somehow
Comment: @AshleyMills curious why you don't think so? With the move towards popovers, alerts, and action sheets being actual containers (starting in iOS 8), as opposed to their own independent view elements (e.g. windows), it actually really does make sense. A popover now functions the same as a normal VC.
Comment: Sorry, not in the sense that your was a bad answer. Just that I was hoping someone would know why the popoverPresentationControllerDelegate methods weren't getting fired. And in my case, the popover is presented from a UIBarButtonItem in the nav bar, so the navController needs subclassing to implement your solution
Answer: Ran into the same issue and after reading through the documentation, I realized that you need to call:
[self presentViewController:myPopoverViewController animated: YES completion: nil];
in order for the delegate methods to get called.
The full snippet is as follows and is run within my -(void)prepareForSegue:sender method:
// Present the view controller using the popover style.
myPopoverViewController.modalPresentationStyle = UIModalPresentationPopover;
[self presentViewController:myPopoverViewController animated: YES completion: nil];

// Get the popover presentation controller and configure it.
UIPopoverPresentationController *presentationController =
         [myPopoverViewController popoverPresentationController];
presentationController.permittedArrowDirections =
         UIPopoverArrowDirectionLeft | UIPopoverArrowDirectionRight;
presentationController.sourceView = myView;
presentationController.sourceRect = sourceRect;

https://developer.apple.com/documentation/uikit/uipopoverpresentationcontroller


JavaScript: Regex to remove string from one tag to another

Question: I have a string

<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1">
#followforfollow #likeforlike yo!

in this I want to remove everything inside the style tags. Can anyone please tell me the optimal solution.The expected output is:

<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1">
#followforfollow #likeforlike yo!

Comment: please add what have you tried so far to remove the mentioned string.
Answer: There is no need for regex since there are DOM methods available to deal with it.
Create a dummy div element and use querySelector to remove the child style element.
var div = document.createElement( "div" );
div.innerHTML = str; //your input string
div.removeChild( div.querySelector( "style" ) );
console.log( div.innerHTML );

Demo

var str = `<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1">
#followforfollow #likeforlike yo!

`;

var div = document.createElement( "div" );
div.innerHTML = str;
div.removeChild( div.querySelector( "style" ) );
console.log( div.innerHTML );
Comment: No need to lengthen the code when you can achieve that using `regex`
Comment: Yes it is as compared to one.
Comment: And also the OP asks for regex solution
Comment: I meant it is not lengthy enough to become unreadable and is anyways more readable than a regex solution.
Answer: If you can use jquery then you don't need to use regex.
$(document).ready(function(){
 $('style').remove();
});

Answer: Here is the regex you are looking for. Short and simple.

var str = `<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1">
#followforfollow #likeforlike yo!

`;

//replace everything between
str = str.replace(/


How to track IP addresses

Question: I want to track IP address of visitors in my site using statistics module.
A log of which is attached here( 3 day log). I can't figure out if an IP address is used by a bot or human being. For example when I checked the 2nd and 3rd IP address, they turned out to be that of Google and Microsoft( Bing). But I've no idea about the first ip address with 900 visits in 3 days.
I want to track malicious user ip address for Adsense invalid clicks.
Answer: What module are you using to get the list of IPs ? You need a Reverse DNS lookup to integrate with your module, I wasn't able to find such module on drupal.org but there's a lot of sites that provides such service like Domain tools: lookup for 22.214.171.124 .
Comment: Is there a reason not to use Google Analytics (or piwik if you want more privacy)?
Comment: Can you explain in your question what StatCounter provides that you want to replicate in Drupal ?
Comment: No... I just need to monitor if someone is visiting frequently and causing invalid Adsense clicks for fraud activity. In short something like Statcounter monitoring functionality built in Drupal
Comment: Have you tried the [StatCounter](http://drupal.org/project/statcounter) modules ?
Comment: Currently I'm using it's free plan. It is costly. I've put up this question to see if I can save money using Drupal modules like that of Statcounter.

How to track IP addresses


Testing for equality using floor function in R does not work

Question: I am doing research on quantiles and I have following problem.
The sample size and the quantile levels are specified as follows.
n <- 200
probs <- seq(0.01, 0.99, 0.01)

The product of n and probs returns integers.
np <- (n * probs)

Thus, the floor function of the product should return the same integers.
floor_np <- floor((n * probs))

However, not all values are TRUE.
np == floor_np

To give two examples:
x1 <- np[6]
x1
[1] 12
y1 <- floor_np[6]
y1
[1] 12
x1 == y1
[1] FALSE

x2 <- np[7]
x2
[1] 14
y2 <- floor_np[7]
y2
[1] 13
x2 == y2
[1] FALSE

Does anybody has an idea why this happens and how to solve it?
Comment: You know from decimal arithmetic that your expression implies integer results, but R like any similar program uses binary arithmetic underneath what you see. Only of a few of 0.01(0.01)0.99 have exact binary equivalents, so very occasionally what should be true isn't quite true.
Comment: `np` is not declared as integer, hence the discrepancy, see eg that `range(as.integer(np)-floor(np))` returns 0...
Answer: The values in probs are numeric type. Just coerce them to integers and this works as expected.
> class(np)
[1] "numeric"

> class(as.integer(np))
[1] "integer"

as.integer(np)[6] == 12
[1] TRUE

Comment: Thank you for your answer. This works indeed. However, I think that the problem exists within the quantile function of the package stats.
Comment: If my answer has been helpful, please consider up-voting and/or accepting it.


Force deletion of Windows service

Question: I have installed a Windows service using the sc create method, then I later used the sc delete method to get rid of it, it then marked the service for deletion.
I made changes to the service and recompiled it, then I tried to recreate it using sc create but it said: The service has been marked for deletion So I thought I'd give it some time.
Now 1 day later and it's still marked for deletion. Can't do anything with the service.
I then tried the same thing using installutil with a different service and the first couple of times it added and removed it successfully, then it eventually also got stuck. Now I've got 2 services marked for deletion, installed using different methods and they were also marked for deletion using different methods.
If installutil and sc can't immediately remove my services, what other options do I have? (Excluding 3rd party components) Or is there a way I could forcibly remove these services?
Comment: I'm pretty sure that requires a reboot.
Comment: Close the services window and delete the service by using the command sc delete service_name
Comment: I used that command without the services window even being open and it's  still there.
Comment: But most of the time it removes it, why would it all-of-a-sudden require a reboot?
Comment: @Anomaly In my case it was because the Visual Studio host was still locking some dependencies that stopped the service from being removed. Closing Visual Studio completely and using [@dileep-kk](http://serverfault.com/users/251431/dileep-kk)'s [method](http://serverfault.com/a/641826/199051) worked for me.
Comment: Close the **Services** window. Then open it again and the service might be gone.
Answer: The only way to remove such services is registry editing.
go to regedit and find
"HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services"

open the key which has your service name. That means the service name
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\AdobeFlashPlayerUpdateSvc

is for Adobe Flash player update service. You can see it in the corresponding displayname on the right panel of regedit.
Delete the key in the left panel and you are done. Also delete the files corresponding to it.
Make sure you are not deleting any wrong system services or else you will end up in trouble starting your machine.
Comment: Worked great. Remember that the service will not vanish from the Services.mvc application until the Service.mvc application has been closed and opened again. Refresh doesn't stop the service showing even though the registry key as gone probably due to caching.
Answer: Using sc.exe delete srvname will postpone the service deletion if it's: 1) running or 2) the "services" management console is open. Making sure the service is stopped and closing the "services" console deletes the service immediately (Windows 10 Pro 10.0.18363).
Comment: Can confirm, this was my issue. I wish this easy and good practice solution would be more known than those requiring heavy registry editing or similar measures.
Comment: Thank you; I was looking at having to reboot after every time I tried to define my service, when the actual problem was only that I had the services panel open to verify the service was created and with the correct parameters. Saved me hours of time.
Answer: "marked for deletion" means it will be deleted at next reboot. Just reboot your server, it will automatically disappear.
Comment: Prefer [@dileep-kk](http://serverfault.com/users/251431/dileep-kk)'s [approach](http://serverfault.com/a/641826/199051).
Comment: "Just reboot" -- And after 100 attempts to define a service for the first time that'll be 400 minutes of time I'll never get back.

Force deletion of Windows service


Find out the kernel of $\phi $?

Question:
Given $\phi : \mathbb{Z}[x] \rightarrow \mathbb{R}$  given by $\phi(f(x)) := f( \sqrt 2)$ is a ring homomorphism.
  Find out the kernel of $\phi $

My  attempt : kernel of $\phi $  is  $x^2-2$
is  its  true  ??
Comment: @jasmine Why are you duplicating questions?
Comment: Cf. [this question](https://math.stackexchange.com/questions/3407657/prove-phi-mathbbzx-rightarrow-mathbbr-given-by-phifx-f-s)
Answer: Hint: The kernel of a ring homomorphism is always an ideal.
Comment: @jasmine but $x^2 - 2$ is not an ideal
Comment: @jasmine note for example that $x(x^2 - 2) = x^3 - 2x$ will also be an element of the kernel.
Comment: That mean @Omnomnomnom .kernel of $\phi $  is  $x^2-2$
Comment: thanks u , now got its   @Omnomnomnom .....ur name  is  diificult  to type

Find out the kernel of $\phi $?


How to add a Mac to Windows Active Directory

Question: I've gone into the Directory Utility on the mac, clicked on the services tab, entered the Active Directory Domain and Computer ID, clicked the Bind button and it showed a little status label which flickered through 5 of 5 steps and then said "An unknown error occurred".
If I ignore the error and just click the OK button further down the screen and make sure the Active Directory service is ticked in the list of services, then go to the Directory Servers tab and try to add an Active Directory by entering my ad domain, computer id, admin username and password I get an error "Unable to add the domain. An unexpected error of type -14090 (eDSAuthFailed) occurred."
Any ideas?
Comment: Turns out I had followed all the steps correctly except that I was entering my machine local admin username and password when trying to do the bind. You eeeediot stimpy!
Answer: Either the credentials you provided were incorrect to join the computer to the domain or the AD server is not set up to allow that machine to join it.
Assuming your credentials are correct, and you see nothing else preventing that computer from joining the domain, try to follow this guide over again from scratch:
Expired link removed
Comment: I had a similar issue and by following the guide realised that the time was out ( it was set to the old time server) and over 3 minutes out is too much for kerberos.


Android volley JsonObjectRequest takes too long

Question: I'm searching now for a few days to figure out why my POST request takes so long to load. I'm using the volley library instead of a HTTPRequest.
This is how I'm doing the request:
RequestQueue requestQueue = Volley.newRequestQueue(getApplicationContext());

...

JSONObject jsonObject = new JSONObject();
jsonObject.put("geoLong", longitude);
jsonObject.put("geoLat", latitude);
jsonObject.put("radius", 5);
JsonObjectRequest jsonRequest = new   JsonObjectRequest(Request.Method.POST, URL,
    jsonObject, createResponseListener(), createErrorListener());
jsonRequest.setShouldCache(false);
jsonRequest.setRetryPolicy(new DefaultRetryPolicy(15000, 2, 1));
jsonRequest.setTag(requestTag);
requestQueue.add(jsonRequest);

The request takes about 10-15 seconds to load until I receive the response which is just ridiculous because the response size is about 1886 bytes. I tested it with a good WLAN connection and also with 3G.
If I'm doing the request with Postman on my Laptop over WLAN it only takes around 400ms so it's not a server-side problem. I'm confused why does it take so long to make this volley request, am I doing something wrong?
Comment: Then how do you know it takes that long?  How do you know the problem isn't in your display code?  Or in a dozen other places?  Or that there even is a problem?
Comment: How are you measuring the time?  I see nothing here that does that.
Comment: I don't measure the time on the phone it's just my assessment.
Comment: Is this in the debugger or without?  I've noticed that some Gson serialization takes forever when running on the debugger, and very fast without.  Maybe you should try to return a string so you can see if it's the request or json converter taking the time.  @GabeSechan has the right idea, you need to instrument your code a bit better to identify the issue.
Answer: Normally volley not taken more time.
you can check by your self.
private long mRequestStartTime;

public void performRequest()
{
    mRequestStartTime = System.currentTimeMillis(); // set the request start time just before you send the request.

    JsonObjectRequest request = new JsonObjectRequest(URL, PARAMS,
        new Response.Listener<JSONObject>()
        {
            @Override
            public void onResponse(JSONObject response)
            {
                // calculate the duration in milliseconds
                long totalRequestTime = System.currentTimeMillis() - mRequestStartTime;
            }
        },
        new Response.ErrorListener()
        {
            @Override
            public void onErrorResponse(VolleyError error)
            {
                long totalRequestTime = System.currentTimeMillis() - mRequestStartTime;
            }
        });

    requestQueue.add(request);
}

If still you will face issue then you can go with Retrofit:http://square.github.io/retrofit/
Comment: I measured the time as you said and the problem was the json parsing. Thank you for your help!


Measuring average from a scale

Question: I am not even sure my question makes sense but hoping one of you might see what I'm trying to achieve.
I have a survey which has the following responses: Unhappy, neutral, happy
I am not sure what values to assign these responses to draw out an average % conclusion of the survey responses I have got. Any ideas would be greatly appreciated!
Comment: What's an "average % conclusion"? What substantive question are you trying to address? What kinds of things do you want to be able to say?
Comment: Arithmetic averages are typically not used in situations like yours. People usually use the median or mode instead.
Answer: As @gung said in his comment, "average" (i.e. arithmetic mean) does not apply to categorical or ordinal data such as yours.  As @Glen_b said, you need to be clear about what question you are trying to answer.
For example, if you want to answer "What is the most common answer?", then you would use the mode, i.e. the answer with the most number of responses.  If you want to answer "What answer is the most likely, if I draw one survey at random?", then you probably want to use the mode if it is above some threshold.  (Unhappy = 51, Neutral = 50, Happy = 50 is not very strong evidence in favor of calling "Unhappy" the most likely answer.)  If you want to assess the collective emotional state of all survey respondents as a group, then you might use the median answer, but this has some problems, because "U = 50, N = 2, H = 50" has the same median as "U = 2, N = 50, H = 2", i.e. both have a median answer of "Neutral".
If, instead, you convert these ordinal categories to ratio-scale numbers, e.g. "Unhappy" = 1, "Neutral" = 2, and "Happy" = 3, you might think that you can then use arithmetic mean or weighted mean, but you can run into problems very quickly.  For example, all three of these cases would have the same mean score of "2":  A) U = 20, N = 20, H = 20; B) U = 0, N = 60, H = 0; C) U = 30, N = 0, H = 30.  In fact, a very large number of cases would result in a score that rounds to "2", even if one of either "Unhappy" or "Happy" were most frequent by a significant margin.
What ever method you choose, just be aware of it's limitations.

Measuring average from a scale


Finding the Laplace transform of $f(x)=|\cos(x)|$

Question: I have function $f(x)=|\cos(x)|, x≥0$ and like to derive its Laplace transform. I am told that $f(x+\pi)=f(x)$. Help me please.
Comment: If you just need the result without derivation: http://www.wolframalpha.com/input/?i=laplace+transform+abs%28cos%28x%29%29
Comment: I see there is a formula for your function.
Comment: @in_wolfram_we_trust: Make me a detailed hint.
Comment: Sorry, I don't have one.
Comment: But You can look at @BabakSorouh answer below.
Answer: Your function is periodic ($T=\pi$) so you can easily use the formula:

$L(f(x))=\frac{1}{1-e^{-sT}}\int_{0}^{T}e^{-sT}f(x)dx$

Note that your function is a piecewise function broken at $\frac{\pi}{2}$.
Comment: Nice!! :+) $\quad +1 \quad$


Egulias Email validator error when upgrading from Drupal 9.5 to 10.1.8 using composer

Question: I am trying to upgrade from Drupal 9.5 to 10.1.8 using composer. I have used upgrade status module to ensure that all modules are compatible with Drupal 10.
I have used the composer command to begin the upgrade.
<code>composer require drupal/core-recommended:10.1.8 drupal/core-composer-scaffold:10.1.8 drupal/core-project-message:10.1.8 --update-with-all-dependencies
</code>
However I keep running into the following error.
<code>  - Root composer.json requires drupal/core-recommended 10.1.8 -> satisfiable by drupal/core-recommended[10.1.8].
- drupal/core-recommended 10.1.8 requires egulias/email-validator ~4.0.1 -> found egulias/email-validator[4.0.1, 4.0.2] but these were not loaded, likely because it conflicts with another require
</code>
Trying to debug the error results in
<code> composer why-not egulias/email-validator 4.0.1
drupal/core             9.5.11      requires         egulias/email-validator (^2.1.22|^3.2)
drupal/core-recommended 9.5.11      requires         egulias/email-validator (~3.2.1)
swiftmailer/swiftmailer v6.3.0      requires         egulias/email-validator (^2.0|^3.1)
egulias/email-validator 4.0.1       requires         doctrine/lexer (^2.0 || ^3.0)
cti/lfhw                dev-develop does not require doctrine/lexer (but 1.2.3 is installed)
Not finding what you were looking for? Try calling `composer update "egulias/email-validator:
</code>
I am not able to understand what should be done to resolve the issue and what is locking the validator to this specific version.
Answer: Swift Mailer module has been deprecated, see Swift Mailer - Moderately critical - Access bypass - SA-CONTRIB-2024-006, so you should consider switching to Drupal Symfony Mailer which will probably fix this dependency.


Inverse of completely multiplicative function with respect to dirichlet convolution

Question: Is the inverse of a completely multiplicative function $f(n)$ with respect to Dirichlet convolution again completely multiplicative? I know that for multiplicative functions its true(Apostol's Analytic number theory page 36) but I think it doesn't hold for completely multiplicative functions. So i was looking for a counterexample(if it isn't true)...
Comment: They are again multiplicative, but not necessarily completely multiplicative.
Comment: I anticipated that. I was looking for a counterexample for a completely multiplicative function whose inverse isn't completely multiplicative.
Comment: Your first sentence can be misunderstood then. Anyway, take the constant $1$ function. Its inverse is the möbius function.
Comment: The inverse of $f(n)$ is $\mu(n)f(n)$.
Answer: The Dirichlet inverse of the completely multiplicative constant $1$ function, i.e. $f(n) = 1$ for all $n$ is given by the Möbius function $\mu$ which is only multiplicative but not completely multiplicative.
Comment: Thanks. Got it. In general $f^{-1}=\mu f$.


How can $r$ be negative when dealing with polar coordinates?

Question: If by definition $r=\sqrt{x^2 + y^2}$, then why do we allow $r$ to be negative? Relatedly, I do not understand the last section of this conversation discussing points being represented by multiple $\theta$:

Student: So a single point could have many different values?
Mentor: Correct! The values for $r$ can be given as positive and negative values and $\theta$ can be given not only in positive and negative values, but also as any value $\theta$ + any multiple of $2\pi$. So, unlike the Cartesian system where each point has a unique set of coordinates, in the polar system any point can have an infinite number of coordinates!
Student: That means that the point given as $(2,\pi/4)$ could also be given as $(2,-7\pi/4)$ or $(2, 9\pi/4)$ or $(-2,5\pi/4)$!
Mentor: Exactly. Try plotting those points using the Polar Coordinates activity and verify they are the same point!

This part in particular is confusing to me:

$\theta$ can be given not only in positive and negative values, but also as
any value $\theta$ + any multiple of 2π.

So if $\theta=5\pi/3$ then it would also equal $5\pi/3 + \pi$? I'm pretty confused on polar coordinates; they seem so inferior to traditional Cartesian when not dealing with imaginary numbers. Could someone clarify this redundancy in polar coordinates?
Comment: Polar coordinates in imaginary numbers are similer to those of trigonometry functions. As $\sin(x) = \sin(x+2k\pi)$. Which means that a point can have infinte number of values. I dont exactly understand your confusion.
Answer: By definition, a point with polar coordinates $r,\theta$ has position:
$$x = r\cos \theta$$
$$y = r\sin \theta$$
Inverting this equation gives you many solutions, as you mentioned. Specifically for $r$, you can square both equations and add them up to get:
$$x^2 = r^2 \cos^2 \theta$$
$$y^2 = r^2 \sin^2 \theta$$
$$x^2+y^2 = r^2(\cos^2 \theta + \sin^2 \theta) = r^2$$
From this, you can determine that $r$ can be either of the following:
$$\sqrt{x^2+y^2},\quad -\sqrt{x^2+y^2}$$
Which one is true? If we know that $\cos \theta <0$, but $x>0$, then we must take $r=-\sqrt{x^2+y^2}$ for the equation to "work out". If the converse is true, we must take the positive value.
In actual usage though, most people prefer to stick with positive $r$, and just change the angle accordingly. Thus instead of using $r=-1, \theta =-\pi$, people commonly just use $r=1, \theta =0$. This means that people just use $r=+\sqrt{x^2+y^2}$,and then choose $\theta$ accordingly.
Comment: I think "actual usage" is context-dependent. If we're given Cartesian coordinates of a point in the plane and are asked for its polar coordinates, we'll generally pick a positive $r$ and a value of $\theta$ from a preferred interval of values such as $(-\pi,\pi].$ But if we are asked about the graph of $r = 2 \cos\theta,$ we will likely consider all real values of $\theta$, some of which result in negative values of $r.$
Comment: So if I had the point (2.5,5pi/3)  another answer could be (2.5,2pi/3)?
Comment: or would it be (-2.5,2pi/3)?
Comment: Option 2. Plot it out and see!


Pagina mostra apenas os arquivos no IIS 7 Aplicação MVC 5

Question: possuo uma aplicação Mvc 5, para desenvolvimento utilizei o IIS Express, tudo funciona normalmente nele, porém estou um problema para subir no IIS 7. Ainda na minha máquina local, posteriormente levarei ao Windows 2008. O problema é que ao rodar http://localhost/Aprovacoes/ minha página não abre, abre apenas os arquivos como na imagem abaixo:

Configuração de acesso a pasta inetpub onde se encontra o publish do projeto está com acesso, ja foi testado no iis.
Já testei com todos os pools de aplicativos, atualmente está setado o 4.0 Integrated.
Meu route config não fiz nenhuma alteração, está da seguinte maneira:
public static class RouteConfig
    {
        public static void RegisterRoutes(RouteCollection routes)
        {
            routes.IgnoreRoute("{resource}.axd/{*pathInfo}");
            routes.LowercaseUrls = true;
            routes.MapRoute("Default", "{controller}/{action}/{id}", new
            {
                controller = "Home",
                action = "Index",
                id = UrlParameter.Optional
            }).RouteHandler = new DashRouteHandler();
        }
    }

Já tentei adicionar documento padrão Index.cshtml no iis mas também não deu certo.
Ao acessar o Home/Index, a action vai identificar que não há usuário logado pelo Forms Authentication, e irá redirecionar para a Account/Login, todo este processo funciona no IIS Express.
Poderiam me ajudar a resolver este problema?
Answer: Para solucionar isto, você precisa acessar a pasta do Windows onde está instalada a versão do seu Asp.net. Faça os seguintes passos:
Abra o prompt de comando como administrador
Acesse:
C:\Windows\Microsoft.Net\Framework\v4.30319

após isto execute o comando aspnet_regiis.exe -i
irá lhe apresentar da seguinte maneira no prompt
C:\Windows\Microsoft.NET\Framework\v4.0.30319>
Microsoft (R) ASP.NET RegIIS version 4.0.30319
Administration utility to install and uninstal
Copyright (C) Microsoft Corporation.  All righ
Start installing ASP.NET (4.0.30319.36392).
.........

Após isto, receberá a mensagem de que foi finalizado, então apenas de um refresh na aplicação dentro do IIS e rode, revise se está setado o pool correto.
Comment: obg, funcionou!
Comment: Sem problemas, eu que agradeço a pergunta, ajudará outros com o mesmo problema.
Comment: Valeu cara, funcionou aqui...


How to fully disable Dynamic DNS on Windows Server?

Question: so we're a University campus, and in DNS Manager, all the students phones, tables, etc, are registering their A records in DNS. We don't want them to. Ideally, we'd just have domain joined devices registering/updating their DNS records.
We've done the following:

DNS: Right click on domain in fwd lookup zone > Properties > Changed "Dynamic Updates" to "Secure only." (it was previously "Nonsecure and secure")
DHCP: Right click the respective DHCP Scope > Properties > DNS Tab > UNCHECK "Enable DNS dynamic updates according to the settings below"
DNS: Delete all records from the student subnets (from every DNS server)

It seemed like it was working for a little while, but then every hour, on the hour, I start to see old records repopulate Windows DNS Manager.
What are we missing? The main push for this was, we had a DNS Squatting issue, where a student's phone's hostname was the same as one of our servers, which prevented end users from accessing that server via the DNS name. Temporary mitigation was to create a CNAME for it, as it appears Dynamic Updates don't update CNAMEs, but we'd ideally like to prevent all non-domain joined devices from registering in DNS.
Thanks in advance.
Comment: Is the DNS zone in question AD integrated?
Comment: @joeqwerty yes it is. And about 15 minutes ago, there weren't any student A records. Now there a a gajillion. And it's summer, so I can't imagine these are all valid, especially when the number of leases in DHCP that recently renewed within this 15 minute time frame is less than the number of DNS records that just appeared.
Comment: Are these records coming from a replicated forward lookup zone on another DNS server?
Comment: @Citizen hopefully not? as part of the process, I went to each DNS server and deleted all records from the student subnet. So if DNS Manager isn't lying to me when it deletes the records, then there shouldn't be any replication. I would delete the records, refresh, go to the next server, delete, refresh, repeat, until I hit all of them, then I would cycle through and again and refresh each.
Comment: we may have figured it out, though we're not sure. need to try something. we're seeing that some of these DNS records that are repopulating, the Owner of them is the same as the account configured in DHCP > Server > IPv4 (right click > Properties) > Advanced > Credentials...

but, those leases in DHCP are old. I just checked and any new leases are actually not showing up in DNS, just, the old leases are the ones are getting repopulated...

need to do some more testing to confirm...
Comment: so we killed all the dhcp leases in these student dhcp scopes, then went to all the dns servers and killed all the dns records for the student subnets.

going to let it sit overnight and check it in the morning since it's time to go home, but as of right now, there aren't any new DNS records repopulating DNS Manager...yet...*knock knock knock*
Answer: Something else you should do/need to do is to configure scavenging on one of the DC\DNS servers and on the DNS zone(s). This will clean up old, stale A records automatically. Note that you only need to enable scavenging on one of your DC/DNS servers, as the AD zone is integrated the DNS zone will be replicated to all other DC\DNS servers so any changes you make will be replicated as part of AD replication to all other DC\DNS servers. There's no need to enable scavenging on more than one DC\DNS server and doing so isn't helpful.
Also note that because the DNS zone is AD integrated, any manual clean up you perform only needs to be done on one DC\DNS server. Again, because the zone is AD integrated any DNS records you cleanup on one DC\DNS server will be replicated to all other DC\DNS servers.
Comment: thanks @joeqwerty. that is something we've looked into in the past, but when we implemented it, the settings i enabled were slightly overly aggressive somehow, because what ended up happening was that some valid records that had old timestamps, that were getting purged. why the records weren't getting updated, i have no idea, but when we did turn on scavenging, a few services dropped off until we could figure out what happened. obviously the solution to avoid that is to create static records for critical services, but still.
Comment: and as far as the AD Integration, yes, I knew that, BUT, because we were having that abnormal situation where even after purging the records, more records than what could have been valid were repopulating, I then took the approach of thinking "maybe the other servers are replicating old records somehow." Hence, hitting each one.
Answer: After doing what I mentioned in the comments, checking this morning, there are only DNS updates from the student subnets from workstations that are joined to the domain. No DNS updates from phones, tablets, personal devices, etc.
So to recap, we did the following:

DNS: Right click on domain in fwd lookup zone > Properties > Changed "Dynamic Updates" to "Secure only." (it was previously "Nonsecure and secure")
DHCP: Right click the respective DHCP Scope > Properties > DNS Tab > UNCHECK "Enable DNS dynamic updates according to the settings below"
DNS: Delete all records from the student subnets (from every DNS server)

But, in addition to all that, we also did the following:

DHCP: Delete all leases from each respective scope.

After doing the first three items, we sorted the DHCP leases by Lease Expiration. This showed new leases. We then checked DNS and saw that any new leases that were clearly a non-domain devices (users-iphone.domain.com, bigdicksipad.domain.com [yup...college students...]), were NOT Dynamically Updating DNS.
So the first two of the first three actions we took actually worked. The issue we were having was that, after 15 or so minutes after the hour, every hour, 1400 new DNS records from the student subnets were getting repopulated into DNS and THAT'S what we couldn't figure out.
On a whim, we noticed that NEW, valid lease records' Owners were the devices themselves (domainworkstation01$), but RENEWED or OLD lease records were owned by either the DHCP server itself, or the domain account handling dynamic dns updates mentioned in a comment above (DHCP > Server > IPv4 (right click > Properties) > Advanced > Credentials...), or SYSTEM.
While those leases are fine, and will eventually drop off after expiration, what we found was that, while they were in DHCP, DHCP would create records based off those. What we did then was delete all the DHCP lease records in DHCP on the student subnet scopes, and then deleted the DNS records.
After checking this morning, I can see that only domain joined devices have DNS records on those student subnets (staff connecting to them).
Whew.
Thanks everyone for your input.


Music notation software for JianPu and erhu notation

Question: My wife's band, which features the erhu, uses music notation like the page shown below.  We want to take hand-written scanned/faxed from stained napkins with bad handwriting and two-side bleedthrough and turn it into neatly typed pages like this.  (I've done some work with making it legible in photoshop but some are just beyond that and would be easier to just retype)
She used to have a "word processor" for this notation, but it got lost at some point and perhaps it would not run on a modern Win7-x64 machine anyway.
What is this number notation called in English (anyone know?), and is there software for editing it?
Comment: NoteAbility Pro is able to create and edit JianPu scores as well as convert between JianPu and western notation.
Answer: This is "JianPu" or "numbered musical notation". Wikipedia on Jianpu mentions some software tools, but most of them seem to require knowledge of chinese (since that notation system is mostly used in china). A (quite old but free) tool is "S-Music". Simp-Erhu is another free tool, but  it requires MS Word.
There's also a comparison of four free Jianpu notation tools online. It compares S-Music, Simp-Erhu, "MUPA" and "MeiDeLi". Since i don't know mandarin, i can't provide a link for the latter two. Some more information is available in this musescore form thread.
Answer: I'd just look for Jianpu.
Maybe this page showing a utility for use with LilyPond helps?
Answer: as others have pointed out, JianPu choices are indeed limited. Most of them are lacking in features, or look really terrible (which can be distracting when playing). There was promise in 2011 when someone wanted to bring about JianPu support for MuseScore, but sadly it never came to reality. Hopefully in the future there'll be more widespread support for JianPu, but personally I highly doubt it - many professional Chinese emsembles have taken to Staff notation so there is a lack of demand for proprietary WYSIWYG editors (IMHO).
Since you're scribing for erhu, Simp-Erhu would be ideal because the formatting is acceptable and the symbols representing instrument techniques are available. Of course, as aforementioned, this comes with the downside of needing MS Word.
Personally, I'd scribe JianPu by hand myself after printing out templates. This method is much much faster than any editor to my knowledge (since it requires no fiddling and is WYSIWYG), and is much easier to make legible than Staff notation.
For templates, it varies from person to person; I've seen a myriad ranging form foolscap to grids made using tables in MS Word (my personal favourite; attached) to dotted lines.
I do speak (and write) Mandarin Chinese, so I could help you with editors mentioned by others if you'd like. Personally I'd rather choose to scribe them by hand simply because this is a case where it's easier to do so (as compared to with Staff notation where WYSIWYG editors are aplenty and easy to use AND yeilds great results).
Answer: This question was asked 8 years ago, but is still relevant. There are a few other tools available:

Notability Pro (mentioned in a comment above) - available for Mac (free) and iPad (small fee). It's quite an advanced notation editor, and can do Jianpu (numbered) notation quite well. It can also convert between western and Jianpu. You can play back scores in Jianpu also. There was a small bug on the iPad version which caused a crash on playback of numbered notation. I reported it, and it was fixed within a few weeks. So that's a good sign.

EOP NMN Master (Everyone Piano - Numbered Music Notation) - available for Windows 7 to 11 (free). A fairly simple program, but seems to have most of what is needed for Jianpu. Can open it's own native format, and also Midi. Can play back numbered notation, and export as Midi.


What is the technical term for a value which can be either a scalar or an array?

Question: I'm working with the JSON API format, which has the notion of a data property which can hold either a scalar (single) or array (multiple) value. I'm writing code for encoding and decoding into the format, and when naming my types, was trying to come up with a good name for such types of values. In TypeScript, it would be
type Poly<T> = T | T[];

For your information, here is the relevant part of the JSON API doc (my emphasis):

Primary data MUST be either:
      a single resource object, a single resource identifier object, or null, for requests that target single resources
      an array of resource objects, an array of resource identifier objects, or an empty array ([]), for requests that target resource collections

As an example, here is a mapping function for such mutant values:
function polymap<T, U>(data: Poly<T>, fn: (input: T) => U, thisArg?: any): Poly {
  if (data instanceof Array) return (data as T[]).map(fn, thisArg);
  return fn.call(thisArg, data as T);
}

Anyway, as you can see, I'm going with "poly", but is there any established terminology for this, or other good suggestions?
Comment: Can't think of a name but more importantly, it's a bad practice that introduces complexity and can trick api users. Just send an Array of 1 element instead.
Comment: @AlexG I agree in principle, but this notion of either-scalar-or-array is baked into the API I am dealing with.
Comment: I believe the TypeScript terminology is the one from computer science - union type. Of course union types does refers to much more than T | T[]
Comment: Yes, I'm aware of union types, I was looking for something for this particular case.
Comment: It is interesting that this type of question will not get a "no established terminology" answer even if it is the correct answer. If the term exists someone will answer but if it does not how can anyone be sure that it does not exist as opposed to him not knowing it? :)
Comment: you may say categorical or Functor
Comment: If you know whether you are targeting a collection of resources or a single resource then you can discard the whole notion and have different return types for each case. That should simplify your code by removing the need to deal with your Poly data type.
Comment: Given that "rank" is a term used to refer to dimensionality as I understand it, we could try "multi-rank" or "poly-rank" or something along those lines.
Comment: You can perhaps look at `R` for inspiration
Answer: First, the difference between scalars and arrays isn't the number of elements, it's the dimensionality.
Scalars are arrays. Specifically, they're 0-dimensional arrays. So you'd just call all of them arrays.
But note that usually the focus isn't on what values the variable can hold, but what operations are allowed on the variable's potential values.
Some operations can generalize from 1 element to N elements, which seems to be what you want.
The CS-y term for this kind of operation is a "vectorizing operation".
The math term for this kind of operation is a "lifting operation".
Comment: @zerkms: I agree. I didn't mean it to be a formal/theoretical statement, since the question is quite informal/practical. (So e.g. I wouldn't have written this on CS.SE.)
Comment: "Scalars are arrays. Specifically, they're 0-dimensional arrays." --- out of curiosity, where does it come from?
Comment: @zerkms: Where does what come from?
Comment: The statement that scalar values are in fact 0d arrays. Not a type system/CT expert here, but it looks weird to me.
Comment: Thanks for your insights. As a matter of terminology, although in theory scalars are indeed 0-dimensional arrays, it seems odd and potentially misleading to therefore call scalars arrays. Could we call these maybe-scalars-maybe-non-zero-dimensional-arrays "polydimensional" values?
Comment: "although in theory" --- in what *theory*? What section of math/cs defines those as so?
Comment: @zerkms: Well this isn't a formal definition -- if you want to compare them you *have* to be willing to play a little loose with the terminology since one is a term from linear algebra and one is a term from programming. So we want a practical definition here. Now if you look at e.g. [NumPy](http://docs.scipy.org/doc/numpy-1.10.0/reference/arrays.scalars.html#indexing), you see they treat scalars like 0-dimensional arrays -- you can even index them like you can index arrays. They're not really different.
Comment: @Mehrdad from type systems perspective this generalisation looks awkward to me. But it's just a personal opinion.
Comment: @torazaburo: I've never heard of that term.
Comment: @Mehrdad I know, I just made it up.
Answer: I've never heard of anything like this and judging from the fact that the available JSON data types shown here
http://www.tutorialspoint.com/json/json_data_types.htm do not mention anything related to such a data type, I'm betting that this is just smoke and mirrors implemented behind the scenes.
Scalar values and arrays are structured very differently.  Combining the two into a true data type would be self contradicting.  I'm betting that when the variable is instantiated a method is called somewhere to check the value for an array or scalar value, at which point one of two things takes place. 1- The data type is automatically set as an array and if a scalar is given, it is converted to an array of length 1 and whenever it is called behind the scenes an array is being accessed at index 0.  Or 2- The behind the scenes method checks what the data type that was passed in is and sets the same data type for the variable given before instantiation actually takes place.
If a MonoPoly Mohn-O-Pohl-E (if you will because I can't find anything else to call it and it looks like Monopoly) was a scalar and is set to an array data type at some point, the old value can be destroyed and a new one assigned with the same name but as an array.  This can happen vise versa and all this can be done behind the scenes as well, making this data type appear to house the description of a scalar along with an array.
Comment: The term MonoPoly is in reference to the fact that this data type is either one (Mono) or more than one (Poly)
Answer:
is there any established terminology for this, or other good
  suggestions?

I'm going to ignore "established terminology", and likely "good" as well, and make the following suggestions on a linguistic basis.  My personal choice, due to its sense of one or more rungs and its being the latin origin of scalar, would be:
ladder
Another possible noun:
assemblage
Like constant, I think the following adjectives are ripe to become nouns:
inconstant (similar to incontinent, "insufficient voluntary control" which seem to be what you described you have over this API issue.)
transferable
indiscrete (nicely homophonic to indiscreet)
dual-purpose
multipurpose (the adjective our schools love)
This might even be an oportunity to coin a new word:
polyunary
versutility  (my second favorite)
Comment: You left out "unconstant".
Answer: Such functionality is likely replacing a scalar with an array when another element is "added" to a slot. AFAIK there's no term for this, so the question seems more like an English language question.
Try these:

elastic scalar
expandable scalar
scarray (scalar/array)
scalarray (another portmanteau)
arrayable
tardis (holds more than it appears to)
Comment: +1 for scarray which sounds like scarry... just like the whole scalar-array-mix idea is kind of scarry... ^^
Comment: @poby see last in new list :)


Using an ISAPI Filter in Visual Studio Development Server

Question: I'm trying to get a current web application running on my local machine, for dev purposes... the previous developer left abruptly and didn't document ANYTHING.
Basically, it uses an ISAPI filter, installed into IIS for some url redirection...
this is fine, but, i want to set it up on my local dev machine...
How can i go about doing that? so that my project runs when i hit f5....
i will probably end up using an httphandler or something in the very near future.
Answer: You need to add the ISAPI filter to IIS using the IIS Management Console. You add ISAPI filters globally at the server level, or at the Web Site level (and the options to look at them) only exist if you have selected the right node.
(Details of using the tool depend on IIS6 vs IIS7.)
If the ISAPI filter is installed, then it should be working, but without documentation you are stuck looking at source or debugging.


Proof of Schlömilch's Generalization

Question: Schlömilch's Generalization theorem states:

Let $u(n)$ be a strictly increasing sequence of positive integers such that the ratio of successive differences is bounded;there is a positive real number $N$, for which:
$$\frac{\Delta u(n) }{\Delta u(n-1) }=\frac{u\left(n+1\right)-u\left(n\right)}{u\left(n\right)-u\left(n-1\right)}<N$$
Then provided that  $f(n)$meets the same preconditions as in Cauchy's test, the convergence of the series $\sum_{n=1}^{\infty}f\left(n\right)$ is equivalent to the convergence of:
$$\sum_{n=1}^{\infty}\Delta u(n)f\left(u\left(n\right)\right)=\sum_{n=1}^{\infty}\left(u\left(n+1\right)-u\left(n\right)\right)f\left(u\left(n\right)\right)$$
the Cauchy condensation test emerges as a special case.

I searched for a proof of this theorem, but I could not find and it seems that the theorem is not famous that much, can anybody prove this and if it's possible then gives some examples?
Updated

Given a real non-negative decreasing sequence $\left\{a_{i}\right\}_{i\in \mathbb N^{+}}$,
Define:
$$S_{n}:=\sum_{i=1}^{n}a_{i}\;\;\;\;\;,\;\;\;\;\;\;u_i:=b^i\;\;\;\;\;,\;\;\;\;\;u_{-1}:=b^{-1}$$ Where $b \in \mathbb N_{\ge2}$ and $i\in \mathbb N$.
Clearly $u_i$ forms a sequence of strictly positive integers.
Since the terms $a_i$'s are non-negative hence:
$$S_{n+1}-S_{n}=\sum_{i=1}^{n+1}a_{i}-\sum_{i=1}^{n}a_i=a_{n+1}\ge0$$
Which follows the sequence $\left\{S_{n}\right\}_{n \in \mathbb N^{+}}$ is monotone increasing,using this fact we see that
If $n<u_{k}$ then:
$$S_{n}=\sum_{i=1}^{n}a_{i}\le\sum_{i=1}^{u_{k}}a_{i}\le a_{1}+a_{2}+a_{3}+a_{4}+...+a_{u_{k}}+...+a_{u_{k+1}-1}$$$$=\left(u_{1}-u_{0}\right)a_{u_{0}}+\left(u_{2}-u_{1}\right)a_{u_{1}}+...+\left(u_{k+1}-u_{k}\right)a_{u_{k}}=\sum_{i=0}^{k}\left(u_{i+1}-u_{i}\right)a_{u_{i}}=\zeta_k$$
$$\iff$$
$$\bbox[5px,border:2px solid #C0A000]{S_{n}\le\zeta_k}\tag{I}$$
If $n>u_{k}$ then:
$$S_{n}=\sum_{i=1}^{n}a_{i}\ge\sum_{i=1}^{u_{k}}a_{i}=a_{1}+a_{2}+a_{3}+a_{4}+...+a_{u_{k-1}+1}+...+a_{u_{k}}$$$$\ge\left(u_{0}-u_{-1}\right)a_{u_{0}}+\left(u_{1}-u_{0}\right)a_{u_{1}}+...+\left(u_{k}-u_{k-1}\right)a_{u_{k}}=\sum_{i=0}^{k}\left(u_{i}-u_{i-1}\right)a_{u_{i}}=\frac{\zeta_k}{u_1}$$
$$\iff$$
$$\bbox[5px,border:2px solid #C0A000]{u_1S_{n}\ge\zeta_k}\tag{II}$$
(to be more precise $(\text{II})$ holds for any $M\ge u_{1}\ge 2$)
Combining $(\text{I})$ and $(\text{II})$ follows:
$S_{n}$ converges if and only if $\zeta_k$ converges and diverges if and only if $\zeta_k$ diverges.
Setting $u_i\mapsto2^i$ gives Cauchy condensation test .

The problem is that I have not used the facts that $u_i>u_{i-1}$ (I know this is true) and for a positive $M$ : $$\frac{u_{k+1}-u_{k}}{u_{k}-u_{k-1}}\le M$$
This is true if and only if $M\ge u_{1}\ge2$
(I again know that this is true).
So what is wrong with my proof? However I know that these two are true, but Since I've proved the theorem without using these two facts I'm not sure if the proof is right.
I think as long as $S_{n}\le\zeta_k\le MS_{n}$ holds for $M\ge u_{1}$ we are able to use the test (moreover since $u_i$ forms a strictly increasing sequence then if $M=u_{i}$ for natural $i\ge1$ then again we can use the test, that's what I think Wikipedia says).

Updated:
From page 122 of Theory And Application Of Infinite Series by Knopp,Konrad, I know that my definiton of $u_i$ is a special case, however I still don't know why do we need
Comment: The proof is found on page 121 of "Theory and Application of Infinite Series" by K. Knopp. I can write out the proof later this week if this question doesn't get a reply.
Comment: For a biography (in German) of Schlömilch that includes a list (on pp. 263-281) of about 360 items by Schlömilch from 1841 through 1900, see Moritz Benedikt Cantor (1829-1920), [*Nachruf an Oskar Schlömilch*](http://archiv.ub.uni-heidelberg.de/volltextserver/13393/1/schloemilch.pdf) [Obituary of Oskar Schlömilch], **Bibliotheca Mathematica** (3) 2 (1901), 260-281. From this you might be able to identify the actual original publication, and if so, you'll almost certainly be able to find it freely available on the internet. **(continued)**
Comment: Curiously, this very useful reference for Schlömilch's publications (published in probably the best known math history journal in the 1800s and early 1900s) is not mentioned in [The MacTutor History of Mathematics archive's biography of Schlömilch or in their list of references for Schlömilch](http://mathshistory.st-andrews.ac.uk/Biographies/Schlomilch.html). Similarly, it is not mentioned in the [Wikipedia page for Schlömilch](https://en.wikipedia.org/wiki/Oscar_Schl%C3%B6milch).
Comment: @DaveL.Renfro, thanks for the information.
Comment: @TheoDiamantakis, I tried to prove the theorem (and using your reference) , can you please look at that and explain my mistakes?
Comment: Bump: did anyone manage to track down the original paper in which this result by Schlömilch is proved?
Answer: Here's the complete proof if my memory serves me correctly, I always found the proof in Knopp a bit unpleasant to read so I tried to make it clearer.
We have by assumption that $f(n)$ is decreasing and $u_n$ is non-negative strictly increasing satisfying $\Delta u_n \leq C \Delta u_{n-1}$ for a $C > 0$, call the partial sums of the series $s_m = \sum_{n=0}^{m} f(n)$ and the partial sums of the condensed series by $t_m = \sum_{n=0}^{m} \Delta u_{n} f\left(u_{n}\right)$
Define $\displaystyle A=\sum_{n \geq 0: f(n)<f\left(u_{0}\right)} f(n) \quad$ when the index $m$ is such that $m < u_k$ we have the inequality
$$\begin{aligned}
s_{m} \leq s_{u_{k}} & =  A+\Big(f(u_{0})+f(u_{0}+1)+ \dots f\left(u_{1}-1\right)\Big)\\ & +\ldots+\mkern-1mu\Big(f\left(u_{k}\right)+f\left(u_{k}+1\right)+\ldots+f\left(u_{k+1}-1\right)\Big) \\
& \leq A+\mkern2mu\left(u_{1}-u_{0}\right) f\left(u_{0}\right)+\ldots+\left(u_{k+1}-u_{k}\right) f\left(u_{k}\right) \\[0.25cm] \text {since } f(n) \text { decreasing } \\{}\\
&=A+t_{k}
\end{aligned}$$
The first line is breaking up the sum of the first $u_k$ many terms by $A$ and whatever is left over, then we group up the terms in the large brackets and estimate each $f$ with the largest choice in the large bracket.
When the index $m$ is such that $m>u_{k}$
$$
\begin{aligned}
s_{m} \geq s_{u_{k}} & \geq\Big(f\left(u_{0}+1\right)+f\left(u_{0}+2\right)+\ldots+f\left(u_{1}\right)\Big)+\ldots+\Big(f\left(u_{k-1}+1\right)+\ldots+f\left(u_{k}\right)\Big) \\
& \geq\left(u_{1}-u_{0}\right) f\left(u_{1}\right)+\ldots+\left(u_{k}-u_{k-1}\right) f\left(u_{k}\right)\\
& = \sum_{j=1}^k\Delta u_{j-1}f(u_j)\\
& \geq \sum_{j=1}^k\frac{1}{C}\Delta u_{j}f(u_j)
\end{aligned}
$$
The first and second line use the same trick as before, but this time we estimate each $f$ with the lowest choice in the large brackets, and the final inequality uses the assumption on $u_n$.
The last inequality leads to
$$
C s_{m} \geq t_{k}-t_{0}
$$
So we have shown that $t_k$ and $s_k$ always bound one another above and below (we don't worry about the conditions $m > u_k$ and so on since we can make either one as big as we need in the infinite sum), thus the convergence and divergence of one are determined by the other.
The special case $u_n = 2^n = \Delta u_n\;, C=1$ is the Cauchy condensation test.
Comment: If anyone knows how to stop Latex from shunting the entire first inequality for no reason whatsoever I would be very appreciative.
Comment: First thank you for the proof, but did you read my question?
Comment: I think your $(I)$ is correct, the line starting with $(u_1 - u_0 )$ uses the fact that $u_i$ is increasing and would not be true otherwise. I think the mistake is in $(II)$, for a start $u_{-1}$ makes no sense and I am unsure if the first inequality is correct.
Comment: Good point, but let me explain why it's true, the mentioned inequality is true if $$a_{1}\ge\left(u_{0}-u_{-1}\right)a_{u_{0}}$$  is true, and notice that :
$$a_{1}\ge\left(u_{0}-u_{-1}\right)a_{u_{0}}$$

 $$\iff$$

$$a_{1}\ge\left(b^{0}-b^{-1}\right)a_{b^{0}}\tag{definiton of $u_i$}$$$$\iff$$$$a_{1}\ge\left(1-\frac{1}{b}\right)a_{1}$$$$\iff$$

Non-negatively of $a_i$ follows:

$$1\ge1-\frac{1}{b}$$

Which is true, since $b$ is strictly positive based on the definition of $u_i$.
Comment: Besides if you think a negative index is not regular, then you are right, that's why I defined that explicitly (moreover  "Theory and Application of Infinite Series" by K. Knopp also has used such negative index).
Comment: I see now, then I think your proof works fine thanks to the nice manipulations you can perform when $u_i$ is of the form $b^i$. This satisfies the assumption of the general proof with $C = b-1$ so the conclusion doesn't lead to any contradictions which is a good sign.
Comment: Now I understand that my $u_i$ is indeed a special case, but why in your proof do we need $u_i$ to be non-negative?what will happen if it's negative?also why this sequence should be a sequence of integers?maybe because it's an index?
Comment: Every time something like $s_{u_k}$ or $f(u_k)$ is written I am implicitly assuming that $u_k$ is a positive integer so it can be used as an index for the sequence $s_m$ or in the domain of $f(n)$.

I probably should have included this as another assumption that $u_i$ is $\mathbb{N}$ valued, but I suppose the proof could be extended if we considered something like the integer part of $u_k$, so positive, increasing would then indeed suffice. Negative values would not work however, since a "$-1$th partial sum" would not make sense.
Comment: You are right, the only thing that I still don't get is that why $u_n$ should be strictly increasing?, I know that :$$C\left(u\left(n\right)-u\left(n-1\right)\right)\ge u\left(n+1\right)-u\left(n\right)$$$$\iff$$$$C\ge\frac{u\left(n+1\right)-u\left(n\right)}{\left(u\left(n\right)-u\left(n-1\right)\right)}$$ is true, if and only if $u_n$ is strictly increasing
Comment: But why do we even need to to convert $$C\left(u\left(n\right)-u\left(n-1\right)\right)\ge u\left(n+1\right)-u\left(n\right)$$ to $$C\ge\frac{u\left(n+1\right)-u\left(n\right)}{\left(u\left(n\right)-u\left(n-1\right)\right)}$$?this division just creates some limitation.
Comment: The strictly increasing assumption is again required for $u_k$ to be well defined as an index, and has nothing to do with the bound relating $\Delta u_k$, that is just used to bound things at in the final line in terms of $t_k$. Formally we are considering $(s_{u_m})_{m \geq 0}$ as a sub-sequence of  $(s_m)_{m \geq 0}$. Something like $a_1, a_2, a_4, a_8 ....$ or $a_p : p$ prime is a valid subsequence, but we can't have the index "go down" or repeat in a way like $a_0, a_1, a_1, a_0, a_2, a_3, a_4 ...$. So we impose strictly increasing to make $u_k$ a "legal" choice for a sub-sequence.

Proof of Schlömilch's Generalization


is there a way to take input from case 1 to case 2 in java?

Question: case 1 the user needs to validate the XML file.
case 2 I need to check if the user validated the XML file and if validation = true I need to display the info of the XML file
my problem is how can I check if the user validated the XML
even if I validate the XML when I inset 2 it is always false
boolean valid = false;

  switch (number)
      {
      case 1 :
          validation();

         break;

      case 2 :
         if (validtrue(valid) == true)
             System.out.println("true");// checking

         else

        System.out.println("please validate the XML file first  ");
         break;

static void validation()
{

    Scanner sc= new Scanner(System.in);
    System.out.println("Enter the name of te XML file wiht .xml");
    String xmlFName = sc.nextLine();
    try {

    File file = new File(xmlFName); // validate the xml
    DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
    DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
    Document doc = dBuilder.parse(file);
    NodeList lst = doc.getElementsByTagName("book");
    }catch(Exception e)

    {
        System.out.print(e.getMessage());
    }

    System.out.println("the "+ xmlFName +" file has been validated ");
    boolean valid = true;
    validtrue(valid);

    }

static boolean validtrue(boolean valid)
{

    if valid == true

    return true;

    else return false;
}

Comment: The point of your `validtrue` method is unclear.
Comment: That all depends on how you're planning to do it. You could validate and display in one go, you could display the menu again or you could use multiple runs. I assume you want to display the menu again so use a loop here.
Comment: if you remove the break in case 1 it will fall through to case 2. but the way i understand it you shouldn't really use a switch/case for this. you should first run `if (!valid(foo)) return;` for example as a guard clause. and if that is ok run whatever other logic you need
Comment: @Thomas yeh I did " do while" that's not my full code I just posted the important part of my question the problem if the user started with case 2 i need to check if he validated in case 1
Comment: @Arnaud the validtrue method I wrote it down just to try some logic and it didn't work
Answer: If I understand correctly you have two menu options: 1 where user validates something and 2 where user does something if validation has been made. And it is required that user should be able to go to option 2 without doing option 1. You could try something like this
public static void main(String[] args) {
    Scanner myInput = new Scanner( System.in );
    boolean validated = false;
    int option;
    while(true){
        option = myInput.nextInt();

        switch (option) {
            case 1:
                validated = validation();
                break;
            case 2:
                if (validated){
                    System.out.println("true");// checking
                }
                else{
                    System.out.println("please validate the XML file first  ");
                }
                break;
        }
    }

}

private static boolean validation(){
    return true; //or false
}


Monotone convergence implies $\mathbb{E}\sum X_n = \sum \mathbb{E}X_n$?

Question: My professor stated the following implication made from the Monotone Convergence Theorem:
\begin{align}\mathbb{E}[\sum X_n] = \sum \mathbb{E}[X_n] \end{align}
Up till now I have been assuming the following operation holds by the linearity of the expectation:
\begin{align}
\mathbb{E}[X_1+X_2 + \cdots + X_n] = \mathbb{E}[X_1] + \cdots + \mathbb{E}[X_n]\end{align}
Aren't these two equalities identical? However, I am confused why I was allowed to assume the second equality, thus far, by the linearity of the expectations when in fact it is given by the Monotone Convergence Theorem...
Basically, is there a difference?
Thank you.
Answer: $ \mathbb{E}[X_1+X_2 + \cdots + X_n] = \mathbb{E}[X_1] + \cdots + \mathbb{E}[X_n]$ holds for any finite $n$ where $\mathbb{E}[\sum X_n] = \sum \mathbb{E}[X_n]$ is an infinite summation.
In order to exchange the expectation and sum, you need some form of monotone convergence, dominated convergence, Fubini's, etc.
Comment: Ahhh, got you. Thanks for the great, and quick, answer!


NSData from CGImageRef in Swift

Question: I am having difficulty figuring out how to get an NSData representation of an image from a CGImageRef. All of the answers I've found make use of UIImage or NSImage, but my application is cross-platform, so I want to use only Core Graphics. Objective-C answers state simply that CFData is toll-free bridged to NSData and simply cast it, but Swift will not allow this. The closest I've got is:
var image: CGImageRef? = nil

//...
if let dataProvider: CGDataProviderRef = CGDataProviderCreateWithURL(url) {
    image = CGImageCreateWithPNGDataProvider(dataProvider, nil, false, CGColorRenderingIntent.RenderingIntentDefault)
    // works fine
    //...
    if let data = CGDataProviderCopyData(CGImageGetDataProvider(image)) as? NSData {
        // Do something with data, if only it ever got here!
    }
}

but the cast doesn't ever succeed...
Answer: CGDataProviderCopyData() returns the optional CFData?, and that cannot be cast to the non-optional NSData. But you can convert/bridge
it to NSData? and use that in the optional binding:
if let data = CGDataProviderCopyData(CGImageGetDataProvider(image)) as NSData? {
    // Do something with data ...

}

Here is a simpler example demonstrating the same issue with
CFString and NSString:
let cfstr : CFString? = "Hello world"
if let nsstr = cfstr as? NSString {
    print("foo") // not printed
}
if let nsstr = cfstr as NSString? {
    print("bar") // printed
}

But I admit that my explanation is not fully satisfying, because
a similar optional cast works in other cases:
class MyClass { }
class MySubclass : MyClass { }

let mc : MyClass? = MySubclass()
if let msc = mc as? MySubclass {
    print("yes") // printed
}

So this must be related to the toll-free bridging between CoreFoundation
and Foundation types.

NSData from CGImageRef in Swift