Filter DataFrame according to different columns combinations to be used for filter

Question: Example of my CSV:
open_local_data,country,competition,match_id,match_name,market_id,market_name,runner_id,runner_name,status,total_matched,odds,market_matched,percentage,above_odds,result,back,lay
2022-08-24 15:00:00,UY,Uruguayan Segunda Division,31684262,Uruguay Montevideo FC v Progreso,1.202440748,Match Odds,11076801,Uruguay Montevideo FC,OPEN,197.2,2.88,448.52,43.96682422188531,9.24460199966309,WINNER,1.7578,-1
2022-08-24 15:00:00,AT,Austrian Matches,31685733,SV Gerasdorf Stammersdorf v Dinamo Helfort,1.202453470,Match Odds,10299781,SV Gerasdorf Stammersdorf,OPEN,15.99,3.05,27.12,58.96017699115043,26.17329174524879,LOSER,-1,0.45609756097560983
2022-08-24 15:00:00,UY,Uruguayan Segunda Division,31684267,Villa Espanola v Sud America,1.202440560,Match Odds,58805,The Draw,OPEN,458.35,3.5,651.11,70.39517132281796,41.82374275138939,LOSER,-1,0.37400000000000005
2022-08-24 15:00:00,UY,Uruguayan Segunda Division,31684266,Miramar Misiones v Central Espanol,1.202440654,Match Odds,5300627,Miramar Misiones,OPEN,642.05,2.1,1075.66,59.68893516538684,12.069887546339224,LOSER,-1,0.85
2022-08-24 15:00:00,UY,Uruguayan Segunda Division,31684266,Miramar Misiones v Central Espanol,1.202440654,Match Odds,5330852,Central Espanol,OPEN,356.65,4.5,1075.66,33.1563877061525,10.93416548393028,LOSER,-1,0.2671428571428571
2022-08-24 15:00:00,BR,Brazilian U20,31688169,Santa Cruz FC U20 v Retro U20,1.202492350,Match Odds,12819986,Santa Cruz FC U20,OPEN,74.82,2.18,78.72999999999999,95.03365934205512,49.16209970902761,LOSER,-1,0.7923728813559321
2022-08-24 15:00:00,CH,Swiss 1 Liga Promotion,31684279,Etoile Carouge v SC Bruhl St Gallen,1.202455897,Match Odds,544529,Etoile Carouge,OPEN,108.24,1.98,169.58999999999995,63.82451795506812,13.319467450017612,WINNER,0.9163,-1
2022-08-24 15:00:00,CH,Swiss 1 Liga Promotion,31684279,Etoile Carouge v SC Bruhl St Gallen,1.202455897,Match Odds,21005121,SC Bruhl St Gallen,OPEN,57.21,3.75,169.58999999999995,33.734300371484174,7.067633704817506,LOSER,-1,0.34

When I want to filter the lines that have the same data as the first line and I know which columns to use to filter the DataFrame, I can find the final result this way:
import pandas as pd

df = pd.read_csv('history_p&l.csv')

def results(a,b):
    filter = df[
        (df['percentage'] == a) &
        (df['odds'] == b)
    ]
    back_sum = filter['back'].sum()
    print(back_sum)

results(
    df.iloc[0]['percentage'],
    df.iloc[0]['odds']
)

But currently I want to discover the result for all possible combinations of columns with values in the DataFrame, in this case these are the options and this is how I create the list of possible combinations:
def combs(a):
    if len(a) == 0:
        return [[]]
    cs = []
    for c in combs(a[1:]):
        cs += [c, c+[a[0]]]
    return cs

options = [
    'country',
    'competition',
    'match_id',
    'match_name',
    'market_id',
    'market_name',
    'runner_id',
    'runner_name',
    'total_matched',
    'odds',
    'market_matched',
    'percentage',
    'above_odds'
]

combinations = combs(options)

How can I manage to make my function (def results()) malleable to analyze the result for each of the combinations?
Also wondering how I will be able to call the function according to the different amount of arguments.
Note: I thought about using eval() and creating the code from strings, but I know that this is not indicated and is not smart to do.
Answer: To get filter you can dynamically build a predicate according to a single combination in combinations along the following lines
filter = df[reduce(lambda x, y : x & (df[y[0]] == y[1]), zip(combination, map(lambda x: df.iloc[0][x], combination)), True)]

Not sure what the best argument(s) for the results() would be but it depends on use cases. For example, results() could be called with just combination rather than a, b, or with two sequences: combination and the corresponding values.
Comment: Hello radof, I tried to do some tests, but all the options used in combination such as ```['competition']```, but it always generates a empty DataFrame, for example ```Series([], Name: competition, dtype: bool)```
Comment: @DigitalFarmer Oops, I have forgotten to filter the dataFrame with the predicate. The predicate is in fact a bool series with the same index as the dataFrame. Corrected.
Comment: Hi @radof ```UserWarning: Boolean Series key will be reindexed to match DataFrame index.``` when using this method, I end up receiving this alert and when trying to adjust it so that it does not happen, I ended up not being successful, if possible to keep the answer updated, help me to create an option that does not end into this message, please. Tks
Comment: @DigitalFarmer I don't get this warning with the above filtering. Can you post a complete reproducible example for your problem? The warning means that the index of the dataframe does not match the index of the Boolean Series. For example, applying the filtering for the second time might result in this Warning in which case the behaviour might be non-deterministic.


Porque o meu ciclo de forEach não está imprimindo nada?

Question: public static Iterable<String> select(Iterable<String> it, Predicate<String> pred) {
    ((Collection<String>) it).removeIf(pred);
    return it;
}

public static void main(String[] args) {
    List<String> list = new ArrayList<>();
    list.add("a");
    list.add("");
    list.add("b");
    list.add("");
    list.add("c");
    Iterable<String> nonEmpty = select(list, s -> !s.isEmpty());
    for(String s : nonEmpty) {
        System.out.println(s);
    }
}

Eu quero através da função select filtrar o conteúdo (em que o filtro é dado através do Predicate) de um objeto que seja iterável. Tentei mexer em várias coisas, uma delas foi fazer o cast para utilizar o removeIf porém quando testo, não sai nenhuma print.
Answer: O problema está neste lambda:
s -> !s.isEmpty()

Ele retorna true quando a string não é vazia. E o método removeIf remove os elementos para os quais o predicado é true.
Ou seja, o código está removendo as strings não-vazias, e no fim só sobram as strings vazias. Você pode conferir isso se imprimir alguma coisa antes de cada elemento:
Iterable<String> nonEmpty = select(list, s -> !s.isEmpty());
for (String s : nonEmpty) {
    System.out.println("- " + s);
}

A saída é:
-
-

Ou seja, no fim sobraram as duas strings vazias.

Se quer remover as strings vazias, basta retirar o ! do lambda:
Iterable<String> nonEmpty = select(list, s -> s.isEmpty());
for (String s : nonEmpty) {
    System.out.println("- " + s);
}

Assim ele retorna true quando a string for vazia, e removeIf remove as strings vazias. A saída é:
- a
- b
- c

Mas precisa usar um Predicate? Você pode usar um loop simples se quiser:
for (Iterator<String> it = list.iterator(); it.hasNext();) {
    if (it.next().isEmpty()) {
        it.remove();
    }
}

Ou, se quiser mesmo usar um Predicate, pode transformar a lista em um stream e filtrar as strings não vazias:
List<String> naoVazios = list.stream()
                             .filter(s -> !s.isEmpty())
                             .collect(Collectors.toList());

Nesse caso eu usei !s.isEmpty(), pois eu quero filtrar as strings que não são vazias.
O código acima cria outra lista, contendo somente as strings não vazias. Mas se quiser só imprimir os elementos, por exemplo, pode fazer:
list.stream()
    .filter(s -> !s.isEmpty())
    .forEach(System.out::println);

Ou, de maneira mais geral:
list.stream()
    .filter(s -> !s.isEmpty())
    .forEach(s -> {
    // fazer o que quiser com a String s
    });

Answer: Pelo que entendi seu código está removendo o que é diferente de vazio e deve estar imprimindo coisas em branco.
Troque para select(list, String::isEmpty); que irá funcionar


Printing fields in a View template file (Devel / Krumo)

Question: My "Article" node type has a field image. I can print this field in a node template with
<?php print render ($content['field_image']); ?>

I have a simple View which lists Articles here. The default Views template /modules/views/theme/views-view.tpl.php prints the content with <?php print $rows; ?> (line 58 here).
I want to use my own Views template which will specify the various elements of the node ie print title, print image, print body etc instead of <?php print $rows; ?>
I don't think...
<?php print render ($content['field_image']); ?>

...will work in a Views template so what syntax do I need?
In case it helps I have the devel / krumo output on the page.
I am aiming to use the Views template to control the layout of the Articles in the View, for example I might want the View to look like this:

So I think the default Views template's <?php print $rows; ?> is no good for this and I need to isolate the various elements of the View's output (image field etc)
Thanks...
Comment: Give a bit more context - why you don't want to just output all those fields through Views?
Comment: @Topsitemakers I am trying to control the layout of the View in the template file because trying to do it in the Views UI has not worked. My question here is meant to be about a simple example View, which I think would be more likely to get replies here than if I ask a question about doing this with my actual View in the Views UI which would be a more difficult / niche question. Is that what you mean by context?
Answer: You not using the correct Views template, views-view.tpl.php is the base template. You need to use views-view-field.tpl.php but you need to use template overrides in your theme.
Firstly, copy views-view-field.tpl.php from the Views module into your theme, make a copy of views-view-field.tpl.php calling it views-view-field--field_image.tpl.php and make your adjustments, remember to clear your template cache so Drupal picks up these files.
Comment: @ Colin thanks - I have added a final sentence above to clarify that I want to control the layout of the fields not their content - does that affect your answer?
Comment: A little, instead of `views-view-field.tpl.php` use `views-view-fields.tpl.php` instead. The file will contain the `$fields` variable: an array of $field objects so you should be able to access `$fields['field_image']`, see @Ravi's link for more details.
Answer: you should read this for more info about views templates.


Deducing $\sin x + \cos x \ge 1$ from $(\sin x + \cos x)^2 = 1 + 2 \sin x \cos x$

Question: So in trig, say I have an acute angle $X$. And one can intuitively conclude that $\sin x + \cos x \ge 1$, but how does the fact that
$$(\sin x + \cos x)^2 = 1 + 2 \sin x \cos x$$
tell me that it is true that $\sin x + \cos x \ge 1$? I don't quite see the connection.
Thanks
Answer: If $x$ is acute $\sin x$ and $\cos x$ are both positive, and therefor
$$2\sin x\cos x\ge 0\\
1+2\sin x\cos x\ge 1\\
\sin^2x+\cos^2x+2\sin x\cos x\ge 1\\
\left(\sin x+\cos x\right)^2\ge 1$$
Where both $\sin x$ and $\cos x$ are positive, so is their addition:
$$\sin x+\cos x\ge 1$$
Comment: But just because sinX and cosX are greater than 0 doesn't guarantee that their addition is greater than 0?
Comment: @Kat Order is preserved in addition so if $a>0$ and $b>0$ then $a+b>0+0=0$.  Now if $c=a+b$ is positive, and $c^2\ge1^2=1$, then $c\ge1$.  (If $c$ is negative, then $c\le-1$.)
Answer: For an acute angle $X$, $\sin X \ge 0$ and $\cos X \ge 0$. So $1 + 2 \sin X \cos X \ge 1$...
Answer:
Let's use $\theta$ instead of $X$.
We are assuming that $0 < \theta < 90^\circ$. If we don't assume this, many of the things I say are not true.
Let $y = \sin \theta$ and $x = \cos \theta$
Now draw right triangle $ABC$ with

$m\angle B = \theta$

$x = BC$

$y = AC$

$1 = AB$

There is a theorem that says that the sum of the lengths of any two sides of a triangle exceeds the length of the third side. So you will always have $x+y> 1$. That is to say,
$\sin \theta + \cos \theta > 1$ for all $\theta$ in the first quadrant.
So you don't care if $(\sin \theta + \cos \theta)^2 = 1 + 2 \sin \theta \cos \theta$ because, if $\theta$ is an acute angle, then $\sin \theta + \cos \theta > 1$


Flash shared object (cookie) - Safe to store password?

Question: I'ld like to use the Flash AS3 shared object ("Flash cookies") to store a password in.
Is it safe by default or do I need a technique to encrypt it?
I couldn't find informations about anybody can lookup what shared objects are set by Flash and what content is inside.
Any information is pretty welcome.
Uli
Answer: A general rule is that anything stored on the client can be lifted from the client unless some form of shared encryption is used (HTTPS, for example). With Flash, this is doubly true -- Flash is very simple to decompile and SharedObjects work while offline, which means that if you can spoof the site of origin, you can read the object. It would take me about 30 minutes, but I suspect I could render every shared object on my machine as a human-readable JSON.
If you must store a password or username on the client, then store it in some form of one-way encryption like SHA or (if you're in a bind) MD5. The as3crypto library is here. But, remember, someone client-side can still grab that value and copy it to another machine.
A couple of asides: SharedObjects are not new in AS3. I remember using them in legacy AS2 libraries. Second: there is little to be gained by use of SharedObjects I've found them inconsistent between IE and the rest of the world. Of course, with Flash Projector, they are your only option. Another benefit of cookies is that cookies are far easier to secure as they can be validated server-side with two-way encryption. Definite bonus.
Answer: Don't save passwords on the client! Though it's also unsafe you could use a checksum:
How to calculate CRC checksum in Actionscript 3?
Flash cookies are Local Shared Objects and they are saved as binary <code>.sol</code> files.
Comment: @Mire Caron: the mentioned article ["How to calculate CRC checksum in Actionscript 3?"](http://stackoverflow.com/q/3459953/256544) refers to [as3corelib](http://github.com/mikechambers/as3corelib/) which supports MD5, SHA1 and more.
Comment: Better yet, use an actual hash function: http://gsolofp.blogspot.com/2006/01/actionscript-3-md5-and-sha1.html


Can I practice playing or singing more emotional?

Question: A few years ago I started playing the guitar, but after two years my guitar teacher told me something which made me aware of a hidden plateu that eventually made me quit. He said that my play was technically correct, but that it was also void of emotion.
This came out of nowhere and I had no clue how to fix it. For me playing the guitar was just about getting the notes and the timing right, nothing more.
Recently I picked up singing, and I've run in to the very same problem. Whenever I look at myself sing I get the impression of someone who's reciting a memorized text, but doesn't know the meaning of what he says.
Are there any practices that could help me overcome this issue?
Answer: I think it's rather inept & cruel of your teacher to tell you your playing is void of emotion. That's subjective, and is intangible so very hard to know how to fix, or know when you've fixed it. Even if you did, you'd only have been satisfying your teacher. I'm really sorry to hear it made you quit.
Example:
I find Bon Jovi utterly devoid of emotion or excitement, but a lot of people really love their music so they must find something enjoyable in it. If I was Jon Bon Jovi's tutor, I'd be tempted to comment on his emotionless performances but probably would hold back so as to let him be what he is. Excuse my arrogance there, but that's kind of my point: We're all allowed an opinion, especially with the arts, and we may not all agree :-)
There are other examples I could list but it'll start to sound like me bitching lol
A thing not to do (at first):
The current trend is to sing a zillion notes around the tune when one or two would have done. Even that can sound mechanical (and sometimes awesome too). If you'd like to sing like that, it's probably best to work on the basic stuff first and then embelish it. But I just wanted to say that this style isn't a replacement for 'singing with feeling' - call me cynical, but it's just more notes.
You haven't said what kind of music you play so it's a bit difficult to give a 'technique' but one thing that gets the rush of passion for me is when my band (rock bass/drums/guitar - I'm guitar & vocals) is throwing out a huge sounding tune which hits you in the chest- literally. So thating with that ..

Thing 1: Loudness.
I think there's some kind of instinctive excitement in hearing loud noises like drums and guitar. The sound thumps you in the body as well as your ears, and gets the adrenalin going.

For me, that excitement pushes my mood up and I just have to let it out with the sining !
Note that loud doens't necessarily mean ear-damaging : you can wear earplugs or maybe just the bass throbbing away will do the trick.

Thing 2: Learn from the masters.
Try listening to some motown or 60's pop/soul like Tina Turner, James Brown, Marvin Gaye, Creedence Clearwater Revival, Stevie Wonder. Also Janice Joplin. Pay attention to how they slide the notes about - especially sliding the pitch up to meet a note, and where they add vibrato in the voice. It's probably not contrived for these people - if they want to hit a certain note, it sounds like that when they do - but I think I can safely say that such soulful performers do so with style.
Thing 3: Strain a bit (but not much)
For most people, the tension in the voice gets more noticable as you reach the top of your range - or sing loudly- so that the voice gets a little gravelly. So try singing some songs where you have to try a bit to get the higher notes. Don't overdo it though as too much can damage your voice. Best bet is somethign that is generally comfortable but you have to stretch a bit now and then, either volume or pitch.
Thing 4: Don't apologise.
If you're going to sing, SING! Stand there and give it to them right between the ears. People want to hear it, so endow them in bucketfuls.

None of this relies on being able to sing accurately. Bob Geldof is a pretty flaky singer but gives plenty of emotion, as does Johnny Rotten (Sex Pistols). Or David Byrne (Talking Heads). There are plenty of examples like this- some of the 'emotion' comes from missing a note a bit or fluffing timing a little.
I hope you pick the guitar up again too !
Comment: I was talking about something like this. https://www.youtube.com/watch?v=yrOH_qz0x3E He's fully absorbed in his own music. He and I might be worlds apart skill-wise, but I've also seen amateur musicians give off a similar vibe.
Comment: On the guitar I used to play rock (mostly 80s and 90s) and classical. My play wasn't "wild enough" for rock, and as I said in an other comment I had mostly no idea what the classical songs were about. By the way it's not just about the sound, but also my body language. Whenever I played I only moved my hands, and I even tried to avoid any useless movements doing that. Right now I'd like to sing rock songs, some of them should sound aggressive, some emotional, but I can't seem to get the feeling right. I'm not sure about the guitar. It has been so long that I even forgot how to read notes.
Comment: @Chris It's interesting that you say you've forgotten how to read notes. .do you play by ear at all ? It occurs to me that the kind of music you'd like to sing is generally written / played by ear - ie - from within, rather than from a piece of paper. Maybe that will help ?
Comment: Regards body language : I too don't move much when playing and singing, but that's as a singer/guitarist, because you can't really. Your mouth has to be on the mic and your hands are busy. When I get a chance, there might be some limb or other trying to shake in time, or sometimes I break into a spot of headbanging when the guiar part is just chords. Have to watch that as it makes me dizzy lol
Comment: Here's a question : re body language.. Are you aiming at that thing where people kind of act out the meaning of the song as they're singing it (eg lyrics "I just want to say I'm sorry" .. ), or just some movement to get into the groove of the tune, almost like dancing, or actually dancing ?
Comment: @Chris Damn he's good! +1 just for posing the video. Honestly : I think movements like that are things that come as you get more confident about performing. Once you've got the physical dexterity part sorted (know the lyrics / know the song well) I think you'll find your body 'wants to' do more while you're playing / singing - there will certainly be more room on your muscle memory for things like that.
Comment: If you compare how a performer is when learning a piece or playing an unfamiliar one - very steady, concentrating, engaged- vs. a performer playing something they've done many times (assuming they're not bored) : calm, confident, probably more animate & able to squidle out the odd smile or a toe tapping etc. So I believe it comes from being well practiced- althoygh this could be subjective of course.
Answer: We all experience considerable frustration when learning a musical instrument.  Two years is, in general, simply not enough time to be expressing  emotions effectively in ones playing.
Depending on your age, this may just be a natural state of affairs.  Males in the mid-teens to early-twenties are often overwhelmed by other issues and may subconsciously keep everything buttoned down.
Ask yourself : "Am I an emotional person?"  If you answer no, then ask yourself again.  You may think you are as cool as a cucumber, but if you look hard enough inside I'm sure you'll find all of the usual emotional turmoil.
Other things to consider are :

Do you make an effort to understand the emotional content of the music you are playing/singing?
Is the style of music you are playing/singing suitable for your emotional profile (comfort zone)?

Sincerity is a necessary condition for expressing emotion effectively. Matters of taste can also effect emotional colour. Many things do as you will no doubt learn for yourself if you continue your studies.
On a more practical note, you might consider exploring techniques used by actors relating to setting a mood.  Many performing musicians use what often appear to be rather over-the-top physical mannerisms to enhance their performance.
Comment: I do consider myself emotional, it's just that I'm focusing so hard on playing every note perfect that I can't pay attention to anything else.

My problem can be divided into two things actually. Firstly, there are songs I can't associate with. Let's say there's a metal song about being angry at everyone and everything. I know what the song is about, I just can't get an agressive sound like that.
Secondly, there are songs like the Dance of the Sugar Plum Fairy. I like the way it sounds, so I wanted to play it, but I don't know what the song means, and probably never will unless I google it.
Comment: @Chris If you're talking about metal guitar playing, then 99% of the emotion you feel comes from the electronic effects employed by the guitarist.  If you unplug all the pedals and processors then most likely you will hear a guy playing Polly Wolly Doodle rather badly - maybe throwing in the odd diminished 5'th.  (bad joke).  I still think you are experiencing normal frustrations and that 2 years is not enough time.  Dance of the Sugar Plum Fairy is a perfect piece to develop a more emotional approach.  It is a fairy tale with a strongly nocturnal mood - dark and spooky yet fun and playful.
Answer: An actor in a play appears to be in a murderously bad mood. Is he really like that? Whilst you may not be a professional, it's entirely possible to put yourself in the right mood to play a particular piece. If it's a gentle lilting song, get the imagination working before you start - lying in a lush field, sun shining, all's right with the world, etc. On occasions, I've wound a pupil up into a frustrated mood before playing the blues, say. It does work. On gigs, your mood will affect your playing, like it or not. The trick is to be aware, and programme yourself accordingly. Of course, if you're still trying to play a piece well, but don't know it well enough, none of this will happen, as your mind is preoccupied with which note follows which note. As said in other answers, knowing a piece inside out is always the first important step.
Answer: Lacking a good connection to some cover song is OK: simply play something else - something you can feel. Or consider trying to write and sing your own songs, be they either vocal or instrumental. There's nothing wrong with hitting the right notes... sometimes the note itself is wrought with emotion and a perfectionist will find the emotion behind the note in order to nail it. In vocals that might mean a certain almost-cracking of the voice; in guitar it might be subtle vibrato, or bending up to a note, or sliding into it, or a touch of artificial harmonic for some squeal, etc...
Technique-wise, try performing with your eyes closed... something about not seeing anything seems to help for me.
Answer: Listen to the song a few times, then see if you can figure out the mood the composer was trying to convey.  Then, try forgetting about the timing and play with the song.  See if you can find a track with just the voice or guitar and playing/singing with it.  For example. swing.  The notation is two 8th notes, but you play/sing it as a quarter/eighth triplet, that gives you the feel.  Try finding the stylistic things that make the song sound how it does (vibrato, not being on time all the time, etc).  Basically, feel the song before you try to play/sing it.
Answer: Well...Emotion is a vague concept. People sometimes inherit it naturally while others have to practice it. When playing the guitar especially, you need to realize what you are doing and why you are doing it. Playing the note accurately is not about playing the guitar. Just check out 'Buckethead', one of my all time favorite guitarists. He brings out such an emotional feeling and makes the environment so tensed while listening to his songs. You need to realize that bringing out your inner emotions through your guitar playing or vocals cannot be achieved within 2 years of training. It takes years. Don't give up. Keep practicing. Watch other top / best guitarists playing and try to grab whatever you can and learn from them. Even the scales you are using to play have a bigger effect on bringing out emotions. Good luck.


Como passar uma função de referencia implementada diretamente no argumento?

Question: Como posso fazer isso em C++ ?
void run(Function){
//Faça algo
Function(); //Executar funcao do argumento
}

int main(){

//Criar e passar como parametro
run({
int a = 5;
int b = 10;
cout<< a+b << endl;
})

}

Aproveitando o topico, e se eu quisesse que ela retornasse um 'int' ou recebesse dois 'int' ?
Answer: Acabei encontrando a solução e envolve função anônima e passagem de função por parâmetro.
Para passagem da função por parametro:
  <tipo Retorno> (*<nomeFuncao>)(<argumentos>)
//Exemplo
void (*myFunc)();

Criar função anônima:
[<capture>] (<argumentos>) -> <tipo Retorno> { <corpo> }

Exemplo:
    [](int a) -> int {
    int a = 5;
    return a;
    }

Juntando os dois fica:
void run(void (*myFunc)()){
    myFunc();
}

int main()
{
    run([]() -> void {
        cout<<"LIDO"<<endl;
    });

//Caso não tenha argumentos e retorno, pode-se simplificar.
    run([]{
        int a=5;
        int b=58;

        cout<<a+b<<endl;
    });

    return 0;
}

Comment: Provavelmente _"uma solução"_ seria melhor descrição que _"a solução"_. Há muitas maneiras de fazer isso em C++.


How on earth do you reason about order of Jetpack Compose modifiers?

Question: I get that Jetpack Compose operates from the outside in for modifiers, but what about this one?
What I asked for:

8 dp padding
then 2 dp border
then background colour for component
then 16 dp padding

@Composable
fun App() {
    MaterialTheme(colors = darkColors()) {
        Surface(
            modifier = Modifier
                .background(MaterialTheme.colors.background)
                .fillMaxSize()
                .padding(8.dp)
        ) {
            Column {
                Card(
                    modifier = Modifier
                        .requiredSize(DpSize(300.dp, 200.dp))
                        .border(BorderStroke(2.dp, Color.White), RoundedCornerShape(8.dp))
                        .background(MaterialTheme.colors.surface)
                        .padding(16.dp)
                ) {
                    // Content still to come
                }
            }
        }
    }
}

What I got:

8 dp padding
then 2 dp border
--then 16 dp padding--
--then background colour for component--

How do I reason about the background colour not extending to the border here? I defined it immediately after.
Addendum:
By the way, swap border and background - same result:
Comment: I do realise that using the other parameters to `Card` produces better results in this case, but what I really want here is some way to understand why it's being quirky.
Answer: Modifier order has always been confusing, but they have a rule: they will append each other since a modifier extension returns a modifer.then. The closest and probably the best incongruity and analogy:

properties are not applied to a view collectively, forget css and xml styling
properties are sequenced, like factory workers on a production line, if padding is first, the next border call will be inside the padding. If border is first, padding will be inside the border

So be careful of positional modifiers.
Comment: Right, but my question shows a case where I swapped the modifiers and things still came out the same way. So it's more like, modifiers are applied from the outside-in _unless some other conditions are met, in which case it does something else_
Comment: Well, quirky is the way it will always be, after writing dozens of UI (for desktop as well) I feel like they cheated by forking off of androidx components without thoroughly explaining the onDraw call system


Lost my key file when I generated the csr


Code in jsfiddle not working that runs locally with expected results

Question: I've got a very simple example of a short angularjs app that works correctly when I run it on my local server.  Here is the fiddle:
http://jsfiddle.net/pkellner99/9sd6ggoq/3/
and the code is quite simple
<code>(function () {
'use strict';

    angular.module('app', []).
    myApp.controller('MyController', ['$scope', function ($scope) {
        $scope.myVal = 'ABCDE';
    }]);

}());
</code>
I get an error regarding app not injected correctly and the expression {{ myVal }} does not evaluate.
Comment: "Uncaught Error: [$injector:modulerr] Failed to instantiate module app due to:
Error: [$injector:nomod] Module 'app' is not available! You either misspelled the module name or forgot to load it. If registering a module ensure that you specify the dependencies as the second argument."
Answer: Two things. First, you're declaring your controller as part of a different module. No module named 'myApp' exists and you have incorrect dot syntax. Change the code to this:
<code>angular.module('app', []).
controller('MyController', ['$scope', function ($scope) {
    $scope.myVal = 'ABCDE';
}]);
</code>
Then, set the JS load to 'no wrap in head' in the fiddle.
Demo
Comment: thanks @isherwood.  I never noticed the location for the js code inserted.  Also, stupid error on my part with myApp.  I just missed that even though i looked at it 10x.  Time for a new year I think.


Is this because the go compiler optimized the code?

Question: package main

import "time"

func main() {
    i := 1
    go func() {
        for {
            i++
        }
    }()
    <-time.After(1 * time.Second)
    println(i)
}

The output is always 1.
However it's absolutely that 1s is enough for the for loop to go over many many times.
I think the i in the closure is the i in the main func.
See the code below.
package main

import "time"

func main() {
    i := 1
    go func() {
        for {
            i++
            println("+1")
        }
    }()
    <-time.After(1 * time.Second)
    println(i)
}

After many lines of "+1", the output is exactly a great number as expected.
Comment: The program does not print the expected value because there's a data race on variable `i`.
Answer:
The Go Memory Model
Version of May 31, 2014
Introduction
The Go memory model specifies the conditions under which reads of a
  variable in one goroutine can be guaranteed to observe values produced
  by writes to the same variable in a different goroutine.
Advice
Programs that modify data being simultaneously accessed by multiple
  goroutines must serialize such access.
To serialize access, protect the data with channel operations or other
  synchronization primitives such as those in the sync and sync/atomic
  packages.
If you must read the rest of this document to understand the behavior
  of your program, you are being too clever.
Don't be clever.
Synchronization
var a string

func hello() {
  go func() { a = "hello" }()
  print(a)
}

the assignment to a is not followed by any synchronization event, so
  it is not guaranteed to be observed by any other goroutine. In fact,
  an aggressive compiler might delete the entire go statement.

The assignment to i, via increment i++ (i = i + 1), is not followed by any synchronization event, so it is not guaranteed to be observed by any other goroutine. In fact, an aggressive compiler might delete the entire i++ statement.
For example,
package main

import "time"

func main() {
    i := 1
    go func() {
        for {
            i++
        }
    }()
    <-time.After(1 * time.Millisecond)
    println(i)
}

Output:
1

The goroutine reduces to:
"".main.func1 STEXT nosplit size=2 args=0x8 locals=0x0
    0x0000 00000 (elide.go:7)   TEXT    "".main.func1(SB), NOSPLIT, $0-8
    0x0000 00000 (elide.go:7)   FUNCDATA    $0, gclocals·2a5305abe05176240e61b8620e19a815(SB)
    0x0000 00000 (elide.go:7)   FUNCDATA    $1, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB)
    0x0000 00000 (elide.go:9)   JMP 0

To the compiler,
for {
    i++
}

can be implemented by incrementing a register forever, essentially a no-op for loop.
for { }

After inserting a print statement,
package main

import "time"

func main() {
    i := 1
    go func() {
        for {
            i++
            println("+1")
        }
    }()
    <-time.After(1 * time.Millisecond)
    println(i)
}

Output:
+1
+1
<< SNIP >>
+1
+1
432

The goroutine expands to,
"".main.func1 STEXT size=81 args=0x8 locals=0x18
    0x0000 00000 (elide.go:7)   TEXT    "".main.func1(SB), $24-8
    0x0000 00000 (elide.go:7)   MOVQ    (TLS), CX
    0x0009 00009 (elide.go:7)   CMPQ    SP, 16(CX)
    0x000d 00013 (elide.go:7)   JLS 74
    0x000f 00015 (elide.go:7)   SUBQ    $24, SP
    0x0013 00019 (elide.go:7)   MOVQ    BP, 16(SP)
    0x0018 00024 (elide.go:7)   LEAQ    16(SP), BP
    0x001d 00029 (elide.go:7)   FUNCDATA    $0, gclocals·a36216b97439c93dafebe03e7f0808b5(SB)
    0x001d 00029 (elide.go:7)   FUNCDATA    $1, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB)
    0x001d 00029 (elide.go:8)   MOVQ    "".&i+32(SP), AX
    0x0022 00034 (elide.go:9)   INCQ    (AX)
    0x0025 00037 (elide.go:10)  PCDATA  $0, $0
    0x0025 00037 (elide.go:10)  CALL    runtime.printlock(SB)
    0x002a 00042 (elide.go:10)  LEAQ    go.string."+1\n"(SB), AX
    0x0031 00049 (elide.go:10)  MOVQ    AX, (SP)
    0x0035 00053 (elide.go:10)  MOVQ    $3, 8(SP)
    0x003e 00062 (elide.go:10)  PCDATA  $0, $0
    0x003e 00062 (elide.go:10)  CALL    runtime.printstring(SB)
    0x0043 00067 (elide.go:10)  PCDATA  $0, $0
    0x0043 00067 (elide.go:10)  CALL    runtime.printunlock(SB)
    0x0048 00072 (elide.go:9)   JMP 29
    0x004a 00074 (elide.go:9)   NOP
    0x004a 00074 (elide.go:7)   PCDATA  $0, $-1
    0x004a 00074 (elide.go:7)   CALL    runtime.morestack_noctxt(SB)
    0x004f 00079 (elide.go:7)   JMP 0

The increased complexity of the goroutine means that the compiler no longer considers dedicating a register to the value of i. The in-memory value of i is incremented, which makes the updates visible, with a data race, to the main goroutine.
==================
WARNING: DATA RACE

Read at 0x00c420094000 by
main goroutine:
  main.main()
      /home/peter/gopath/src/lucky.go:14 +0xac

Previous write at 0x00c420094000 by
goroutine 5:
  main.main.func1()
      /home/peter/gopath/src/lucky.go:9 +0x4e

Goroutine 5 (running) created at:
  main.main()
      /home/peter/gopath/src/lucky.go:7 +0x7a
==================

For your expected result, add some synchronization,
package main

import (
    "sync"
    "time"
)

func main() {
    mx := new(sync.Mutex)
    i := 1
    go func() {
        for {
            mx.Lock()
            i++
            mx.Unlock()
        }
    }()
    <-time.After(1 * time.Second)
    mx.Lock()
    println(i)
    mx.Unlock()
}

Output:
41807838

Answer: Concurrent access to the variable i need to be synchronized:

synchronization is better done with channels or the facilities of the
  sync package. Share memory by communicating; don't communicate by
  sharing memory.

ref: https://golang.org/pkg/sync/atomic/

This is interesting, so I'm sharing my experiments:
0- Your code using time.Sleep(1 * time.Second) (not recommended-not synchronized):
package main

import "time"

func main() {
    i := 1
    go func() {
        for {
            i++
        }
    }()
    time.Sleep(1 * time.Second)
    println(i)
}

output:
1

1- Using i := new(int) (not recommended-not synchronized):
package main

import "time"

func main() {
    i := new(int)
    go func() {
        for {
            *i++
        }
    }()
    time.Sleep(1 * time.Second)
    println(*i)
}

output(CPU: i7-7700K @ 4.2GHz):
772252413

2- Synchronizing using atomic.AddInt64(&i, 1)(Package atomic provides low-level atomic memory primitives useful for implementing synchronization algorithms):
package main

import (
    "sync/atomic"
    "time"
)

func main() {
    i := int64(1)
    go func() {
        for {
            atomic.AddInt64(&i, 1) // free running counter
        }
    }()
    time.Sleep(1 * time.Second)
    println(atomic.LoadInt64(&i)) // sampling the counter
}

output:
233008800

3- Synchronizing using chan int:
package main

import "time"

func main() {
    ch := make(chan int)
    go func() {
        timeout := time.NewTimer(1 * time.Second)
        defer timeout.Stop()
        i := 1
        for {
            select {
            case <-timeout.C:
                ch <- i
                return
            default:
                i++
            }
        }
    }()
    //time.Sleep(1 * time.Second)
    println(<-ch)
}

output:
272702341

4- Synchronizing using sync.WaitGroup:
package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    var done sync.WaitGroup
    done.Add(1)
    i := 1
    go func() {
        defer done.Done()
        timeout := time.NewTimer(1 * time.Second)
        defer timeout.Stop()
        for {
            select {
            case <-timeout.C:
                return
            default:
                i++
            }
        }
    }()
    done.Wait()
    fmt.Println(i)
}

output:
261459418

5- Synchronizing using quit channel:
package main

import (
    "fmt"
    "time"
)

func main() {
    quit := make(chan struct{})
    i := 1
    go func() {
        for {
            i++
            select {
            case <-quit:
                return
            default:
            }
        }
    }()
    time.Sleep(1 * time.Second)
    quit <- struct{}{}
    fmt.Println(i)
}

output:
277366276

6- Synchronizing using sync.RWMutex:
package main

import (
    "sync"
    "time"
)

func main() {
    var i rwm
    go func() {
        for {
            i.inc() // free running counter
        }
    }()
    time.Sleep(1 * time.Second)
    println(i.read()) // sampling the counter
}

type rwm struct {
    sync.RWMutex
    i int
}

func (l *rwm) inc() {
    l.Lock()
    defer l.Unlock()
    l.i++
}
func (l *rwm) read() int {
    l.RLock()
    defer l.RUnlock()
    return l.i
}

output:
24271318

related topics:
The Go Memory Model
There is no equivalent to volatile and register in Go
Does Go support volatile / non-volatile variables?
Comment: The primary virtue of a program is correctness. You don't say which examples are incorrect. For example, at least examples 0, 1, and 2.
Comment: Dear @peterSO: I think example 2 has no problem, do you see any? (using `atomic.AddInt64(&i, 1)` )
Comment: The Go memory model contains no explicit mention of atomic variables. Some have argued for, at best, somethiing like this: An implementation should ensure that the last value (in modification order) assigned by an atomic operation will become visible to all other goroutines in a finite period of time. You would see updated values of `i` sometime between now and something less than infinity. Essentially, the value of `i` is undefined.
Comment: Yes, `LoadInt64 atomically loads *addr.` just samples at moment system provided, it is guaranteed  **atomic synchronization** but time of sampling is unknown.  so using example 2 here is OK, but it depends to use case. thanks again.


Angular: erro ao tentar manipular valor de input - property 'value' does not exist on type 'Element

Question: Eu estou tendo problemas ao tentar pegar o value de um input para o limpar, o código está num script .ts (usado no Angular 7), código abaixo:
<code>document.querySelector('.input').value = "";
</code>
Erro ao compilar o código

error TS2339: Property 'value' does not exist on type 'Element'

Está faltando algo? Pois fiz testes utilizando o console e ocorreu tudo certo.
Comment: O melhor seria pegar do seu NgModel ou do seu ReactiveForms dependendo de qual vc estiver usando. Isso de pegar o valor do input através do DOm era prática de jquery antiga e não é recomendado principalmente com angular.
Comment: Irei pesquisar sobre o NgModel e ReactiveForms, obrigada!
Answer: O problema era do tipo que não poderia ser Element, consegui solucionar com o código abaixo, convertendo de Element para Input:
<code>(document.querySelector('.input') as HTMLInputElement).value = "";
</code>


Private file path getting set as public local files

Question: I have a new Drupal 8 site up and running, and in the File System settings I have the Default download method set to Public local files served by the web server.
My public folder is set to site/default/files. When I set my private folder to site/default/private, I get the following error.

Writable (public download method)
The directory sites/default/private is not writable

The public folder is not set to sites/default/private, but sites/default/files, so why does this point to private instead?
My files and private are both writable by Drupal. The  permissions for bot the directories are set to 775, so they are writable.
If I remove the private folder setting, the error goes away, but I obviously need the private directory to be set.
What could possibly cause Drupal to be confused on what is the private and public directories?
Answer: It's not mixing up public/private directories; the File system report row on the Status report page is conveying multiple bits of information:

Your default download method ("Writable...")
Any other issues found with your file system config. In this case, permissions on the private directory.

Regarding the permission issue, you'll need to check what user/group the Drupal/Apache/PHP is running as to confirm 775 lines up with the private dir group.


How to perform advanced search of all process memory from Visual Studio debugger?

Question: I am a developer on a large commercial program, and I am trying to track down a particular C++ memory leak.  I would like to use Visual Studio to search the entire valid address space of my process, but I can not see how to perform this search effectively.
I am aware of the .s command documentation here, but it is not doing what I need.  For examle, I am 100% certain that the address 0xfdfd240 contains the value 0x0f0e34a8.  I can successfully use the .s command to search near that address like this:
<code>.s -d 0x0fdfd200 L256000000 0x0f0e34a8
Found match at
0xfdfd240
</code>
But my program has made many small allocations which have left me with many small non-contiguious sections of memory.  If I back up a few thousand bytes the search command fails:
<code>.s -d 0x0fd00000 L256000000 0x0f0e34a8
Memory location could not be read. Please specify a valid memory location.
</code>
And it also appears that the search command does not attempt to search forward any more when it finds it's first bad address, even if there are valid addresses beyond it.
<code>.s -d 0x0f000000 L256000000 0x0f0e34a8
No match was found
</code>
I'm vaguely aware that there is a way to ask windows what memory ranges are valid for a given process, so I'm considering writing a small throwaway program to harvest that information and then automate a series of search commands to the Immediate Window...but it seems like someone must have dealt with this before, and must have done something smarter.
Also, I can extract a dump file of the running process, so if anyone can recommend third party tools to feed the dump to that has stronger search functionality that should work out as well.
Any suggestions?
Edit:  I see this behavior in VS2008SP1 and VS2010SP1.
Comment: To find non obvious memory leaks you need to purchase a memory leak analysis program like purify.
Answer: Here's a small tool that finds the memory blocks1 in a target process, searches the blocks for a pattern, and prints out the addresses at which it found the pattern. Right now it's fairly primitive -- purely command line, expects you to supply the PID of the target process, only accepts the pattern as a single string on the command line. The search portion should be reasonably robust, so it's mostly a matter of a pretty weak UI -- nonetheless, it works fine as long as the pattern you're looking for is something you can enter as a string on the command line. If you want to include non-printable characters, it would be pretty easy to add in a translator I wrote years ago that would let you use C-style escape sequences on the command line.

1 skipping blocks that hold things like the code of the executable/DLLs for that process.

<code>#include <iostream>
#include <vector>
#include <string>
#include <windows.h>
#include <algorithm>
#include <iterator>

template <class outIter>
void find_locs(HANDLE process, std::string const &pattern, outIter output) {

    unsigned char *p = NULL;
    MEMORY_BASIC_INFORMATION info;

    // VirtualQueryEx does the hard part, telling use about each
    // block in the target process.

    for ( p = NULL;
        VirtualQueryEx(process, p, &info, sizeof(info)) == sizeof(info);
        p += info.RegionSize )
    {
        // buffer to hold copies of memory blocks from the target
        std::vector<char> buffer;
        std::vector<char>::iterator pos;

        // We only want committed memory that's mapped or private --
        // screens out things like the code in the target process.
        //
        if (info.State == MEM_COMMIT &&
            (info.Type == MEM_MAPPED || info.Type == MEM_PRIVATE))
        {
            DWORD bytes_read;

            // copy block from target to our buffer, search for pattern:
            buffer.resize(info.RegionSize);
            ReadProcessMemory(process, p, &buffer[0], info.RegionSize, &bytes_read);
            buffer.resize(bytes_read);

            // find all occurrences of pattern in buffer:
            for ( pos = buffer.begin();
                buffer.end()!=(pos=std::search(pos, buffer.end(), pattern.begin(), pattern.end()));
                ++pos)
            {
                // record each address in target where pattern was found:
                *output++ = p+(pos-buffer.begin());
            }
        }
    }
}

int main(int argc, char **argv) {
    if (argc != 3) {
        fprintf(stderr, "Usage: %s <process ID> <pattern>", argv[0]);
        return 1;
    }

    // Read the target PID and search pattern:
    int pid;
    sscanf(argv[1], "%i", &pid);
    std::string pattern(argv[2]);

    // Open the target process with rights to read its information and memory:
    HANDLE process = OpenProcess(
        PROCESS_VM_READ | PROCESS_QUERY_INFORMATION,
        false,
        pid);

    // Find the locations, and write them to standard output, one per line:
    find_locs(process, pattern,
        std::ostream_iterator<void *>(std::cout, "\n"));

    return 0;
}
</code>
Answer: When doing anything beyond simple debugging, I find that often times WinDbg (part of Debugging Tools for Windows) has far better capability.
And you don't even have to do all your debugging from WinDbg.  If you are in VS doing your thing and then want to search memory (or do other things which are not easy in VS), go to Debug -> Save Dump As...  Make sure to select "Minidump with Heap" file type.
This will create a complete snapshot file of your process.  Load that up into WinDbg and now you have oh some many cool commands at your disposal.  For easy access to the documentation, you can always type ".hh" in the command window.  All Commands are listed under Debuggers -> Debugger Reference -> Debugger Commands.
The one you want is s (Search Memory)
Comment: Thanks for pointing me to WinDbg, it's search command behaves as I wished.


Automatic git commit between Red, Green and Refactor steps?

Question: I'm always interested in trying new things with my workflow, and I thought it might be an interesting experiment to automatically commit between red, green and refactor steps, but then manually squash the commits down once I finish a specific feature (and before pushing).
I was just wondering if anyone else has tried this before? I thought I read about this once, but I'm unable to find any references.
I'm hoping one benefit might be to focus more on committing often, as well as being able to see my workflow visually so that I can improve it. For example, before squashing I can see if my time between red and green is too long, or if the number of code changes I make is larger than necessary between each step.
I was going to implement this as a guard plugin so that when I save a spec or library file, it runs the specs and commits the change with a commit message like:
<code>Green: 1621 examples, 0 failures, 2 pending (1659 tests/s, 0.0006 p/test)
</code>
The idea being that I could visually scan this when squashing and determine where to group the related Red/Green/Refactor commits by logical changes.
At worst I thought this might be a fun experiment, at best it might give me a different way of seeing how I work.
Answer: I like the idea.
Showing the new / updated spec could be a plus. :)
It might be tricky for this plugin to know when the code reached a "true" Red / Green state.
Would it:

commits --amend 'Red' when specs are not passing and no file other than 'spec' files are changed?
after that commits 'Green' as soon as specs are passing because of an update in lib?
Comment: I thought about that kind of workflow. For the start I was just going to commit after every spec run. Amending when there's no state change might cut down on the noise though.
Answer: Yeah, I think this would be a fun experiment to do as the information gathered would be interesting to analyze.  You could look at your average cycle times, and see what parts of the projects (files) have slower cycle times which could be veiwed as a code metric.  The more information in the git log the better.. i.e. which spec is failing etc.  Please share any progress and/or results that come from the idea.
Answer: I like this and have considered it myself a time or two. I'm not sure I completely see the value of it though. At first blush I thought it would make it easy to roll back to my last known green state. After seeing JUnitMax, it has the "revert to green" functionality built in, so I haven't felt the desire since to auto commit.
Answer: Wow!! Believe it or not, I had a similar idea a few days back (to do as a weekend project), when I was reading on Unit Testing, that I wanted to build a module for git doing similar thing.
My idea was not completely automating, but rather a part. For example, a custom commit will see what are the REDs and give them some ID, and subsequent GREEN will see what all REDs does it solve, and append a suitable comment in the commit message along with the test ID and the RED commits.
Some additional functionality may include browsing through these commits based on some criteria...
Anyways, if you find this reference that you are talking about, please update this thread.


Using pandas to perform regression, error: cannot concatenate 'str' and 'float' objects

Question: I've been writing a code based on this answer (Reading csv to array, performing linear regression on array and writing to csv in Python depending on gradient) in order to find out which days exhibited an increasing wind speed in the morning.
This is a sample of my data
hd,Station Number,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Local time,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Local standard time,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Universal coordinated time,Precipitation since last (AWS) observation in mm,Quality of precipitation since last (AWS) observation value,Air Temperature in degrees Celsius,Quality of air temperature,Air temperature (1-minute maximum) in degrees Celsius,Quality of air temperature (1-minute maximum),Air temperature (1-minute minimum) in degrees Celsius,Quality of air temperature (1-minute minimum),Wet bulb temperature in degrees Celsius,Quality of Wet bulb temperature,Wet bulb temperature (1 minute maximum) in degrees Celsius,Quality of wet bulb temperature (1 minute maximum),Wet bulb temperature (1 minute minimum) in degrees Celsius,Quality of wet bulb temperature (1 minute minimum),Dew point temperature in degrees Celsius,Quality of dew point temperature,Dew point temperature (1-minute maximum) in degrees Celsius,Quality of Dew point Temperature (1-minute maximum),Dew point temperature (1 minute minimum) in degrees Celsius,Quality of Dew point Temperature (1 minute minimum),Relative humidity in percentage %,Quality of relative humidity,Relative humidity (1 minute maximum) in percentage %,Quality of relative humidity (1 minute maximum),Relative humidity (1 minute minimum) in percentage %,Quality of Relative humidity (1 minute minimum),Wind (1 minute) speed in km/h,Wind (1 minute) speed quality,Minimum wind speed (over 1 minute) in km/h,Minimum wind speed (over 1 minute) quality,Wind (1 minute) direction in degrees true,Wind (1 minute) direction quality,Standard deviation of wind (1 minute),Standard deviation of wind (1 minute) direction quality,Maximum wind gust (over 1 minute) in km/h,Maximum wind gust (over 1 minute) quality,Visibility (automatic - one minute data) in km,Quality of visibility (automatic - one minute data),Mean sea level pressure in hPa,Quality of mean sea level pressure,Station level pressure in hPa,Quality of station level pressure,QNH pressure in hPa,Quality of QNH pressure,#
hd, 40842,2000,03,20,10,50,2000,03,20,10,50,2000,03,20,00,50,      ,N, 25.7,N, 25.7,N, 25.6,N, 21.5,N, 21.5,N, 21.4,N, 19.2,N, 19.2,N, 19.0,N, 67,N, 68,N, 66,N, 13,N,  9,N,100,N,  4,N, 15,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#
hd, 40842,2000,03,20,10,51,2000,03,20,10,51,2000,03,20,00,51,   0.0,N, 25.6,N, 25.8,N, 25.6,N, 21.5,N, 21.6,N, 21.5,N, 19.2,N, 19.4,N, 19.2,N, 68,N, 68,N, 66,N, 11,N,  9,N,107,N, 11,N, 13,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#
hd, 40842,2000,03,20,10,52,2000,03,20,10,52,2000,03,20,00,52,   0.0,N, 25.8,N, 25.8,N, 25.6,N, 21.7,N, 21.7,N, 21.5,N, 19.5,N, 19.5,N, 19.2,N, 68,N, 69,N, 66,N, 11,N,  9,N, 83,N, 13,N, 13,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#
hd, 40842,2000,03,20,10,53,2000,03,20,10,53,2000,03,20,00,53,   0.0,N, 25.8,N, 25.9,N, 25.8,N, 21.6,N, 21.8,N, 21.6,N, 19.3,N, 19.6,N, 19.3,N, 67,N, 68,N, 66,N,  9,N,  8,N, 87,N, 14,N, 11,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#
hd, 40842,2000,03,20,10,54,2000,03,20,10,54,2000,03,20,00,54,   0.0,N, 25.8,N, 25.8,N, 25.8,N, 21.6,N, 21.6,N, 21.6,N, 19.3,N, 19.3,N, 19.2,N, 67,N, 67,N, 67,N,  8,N,  4,N, 98,N, 23,N,  9,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#
hd, 40842,2000,03,20,10,55,2000,03,20,10,55,2000,03,20,00,55,   0.0,N, 25.7,N, 25.8,N, 25.7,N, 21.5,N, 21.6,N, 21.5,N, 19.2,N, 19.3,N, 19.2,N, 67,N, 68,N, 66,N,  8,N,  4,N, 68,N, 15,N,  9,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#
hd, 40842,2000,03,20,10,56,2000,03,20,10,56,2000,03,20,00,56,   0.0,N, 25.9,N, 25.9,N, 25.7,N, 21.7,N, 21.7,N, 21.5,N, 19.4,N, 19.4,N, 19.2,N, 67,N, 68,N, 66,N,  8,N,  5,N, 69,N, 16,N,  9,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#
hd, 40842,2000,03,20,10,57,2000,03,20,10,57,2000,03,20,00,57,   0.0,N, 26.0,N, 26.0,N, 25.9,N, 21.8,N, 21.8,N, 21.7,N, 19.5,N, 19.5,N, 19.4,N, 67,N, 68,N, 66,N,  9,N,  5,N, 72,N, 10,N, 11,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#
hd, 40842,2000,03,20,10,58,2000,03,20,10,58,2000,03,20,00,58,   0.0,N, 26.0,N, 26.1,N, 26.0,N, 21.7,N, 21.8,N, 21.7,N, 19.4,N, 19.5,N, 19.3,N, 66,N, 67,N, 66,N,  8,N,  5,N, 69,N, 13,N, 11,N,     ,N,1018.6,N,1017.5,N,1018.6,N,#

and this is my attempted code:
import glob
import pandas as pd
import numpy as np
from datetime import datetime

for file in glob.glob('X:/brisbaneweatherdata/*.txt'):
    df = pd.read_csv(file)

    col = 'Wind (1 minute) speed in km/h'
    mask = pd.notnull(df[col])
    df = df.loc[mask]

    for date, group in df.groupby(['Year Month Day Hours Minutes in YYYY', 'MM', 'DD']):
        morning_data = group[group.HH24.between(9, 12)]
        gradient, intercept = np.polyfit(morning_data.HH24, morning_data['Wind (1 minute) speed in km/h'], 1)
        wind_direction = np.average(morning_data['Wind (1 minute) direction in degrees true'])
        if gradient > 0:
            print("{0:%d, %b %Y} , {1:.2f}, {2:.2f}".format(datetime(*date), gradient, wind_direction))

However, this is producing
runfile('X:/python/linearregression.py', wdir='X:/python')
X:/python/linearregression.py:1: DtypeWarning: Columns (17,25,27,29,31,33,35,37,55,57,59) have mixed types. Specify dtype option on import or set low_memory=False.
  import glob
Traceback (most recent call last):

  File "<ipython-input-19-ace8af14da2c>", line 1, in <module>
    runfile('X:/python/linearregression.py', wdir='X:/python')

  File "C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 699, in runfile
    execfile(filename, namespace)

  File "C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "X:/python/linearregression.py", line 10, in <module>
    gradient, intercept = np.polyfit(morning_data.HH24, morning_data['Wind (1 minute) speed in km/h'], 1)

  File "C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\numpy\lib\polynomial.py", line 550, in polyfit
    y = NX.asarray(y) + 0.0

TypeError: cannot concatenate 'str' and 'float' objects

If I try to convert my year values to integers of floats, e.g. int('Year Month Day Hours Minutes in YYYY') or int('MM') it produces the error ValueError: invalid literal for int() with base 10: 'Year Month Day Hours Minutes in YYYY'
With help from Unutbu the TypeError issue has been resolved, however. This produces the error below.
runfile('X:/python/linearregression.py', wdir='X:/python')
X:/python/linearregression.py:1: DtypeWarning: Columns (17,25,27,29,31,33,35,37,55,57,59) have mixed types. Specify dtype option on import or set low_memory=False.
  import glob
C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\numpy\lib\polynomial.py:594: RankWarning: Polyfit may be poorly conditioned
  warnings.warn(msg, RankWarning)
Traceback (most recent call last):

  File "<ipython-input-24-ace8af14da2c>", line 1, in <module>
    runfile('X:/python/linearregression.py', wdir='X:/python')

  File "C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 699, in runfile
    execfile(filename, namespace)

  File "C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "X:/python/linearregression.py", line 17, in <module>
    wind_direction = np.average(morning_data['Wind (1 minute) direction in degrees true'])

  File "C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\numpy\lib\function_base.py", line 570, in average
    avg = a.mean(axis)

  File "C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\numpy\core\_methods.py", line 72, in _mean
    ret = ret / rcount

TypeError: unsupported operand type(s) for /: 'str' and 'int'

Comment: Can you include the full `TypeError` message, including the traceback that shows where the problem happened?
Comment: I have included the full error message in the question now.
Comment: I suspect this has to do with how the column headers were parsed.  Where did you source the data? You may want to try parsing the data itself without the column names. `df = pd.read_csv(file, header=None)`
Comment: If I parse that data without the column names, what do I use to group the columns into dates for the command                 'for date, group in df.groupby(['Year Month Day Hours Minutes in YYYY','MM','DD']):'
Answer: The error message
  File "C:\Users\kirkj\AppData\Local\Continuum\Anaconda2\lib\site-packages\numpy\lib\polynomial.py", line 550, in polyfit
    y = NX.asarray(y) + 0.0

TypeError: cannot concatenate 'str' and 'float' objects

can be reproduced if y is a Series containing a string:
In [14]: np.asarray(pd.Series(['',1.0])) + 0.0
TypeError: cannot concatenate 'str' and 'float' objects

Now if you peek at line 550 inside polynomial.py, you'll see that y is the second argument passed to np.polyfit. So this suggests strongly that morning_data['Wind (1 minute) speed in km/h'] is a Series containing a string.
The sample data that you posted does not show a string, but somewhere further along in the CSV you'll likely find a string in that column.
Now how can we find that string? One way would be to convert the Series to numeric values (coercing invalid strings to NaN):
col = 'Wind (1 minute) speed in km/h'
tmp = pd.to_numeric(morning_data[col], errors='coerce')

and then to look for NaN:
mask = pd.isnull(tmp)
print(morning_data.loc[mask, col])

This will show you all the values in the 'Wind (1 minute) speed in km/h' column which could not be converted to numerals.
You could then consider how you wish to handle these problematic rows.  If there
are just a few of them, you might edit them by hand. Or look into how the CSV
was generated and fix the error at the source. Or, if you want to discard these
rows, you could use
for file in glob.glob('X:/brisbaneweatherdata/*.txt'):
    df = pd.read_csv(file)

    for col in ['Wind (1 minute) speed in km/h',
                'Wind (1 minute) direction in degrees true']:
        df[col] = pd.to_numeric(df[col], errors='coerce')
        mask = pd.notnull(df[col])
        df = df.loc[mask]

    for date, group in df.groupby(['Year Month Day Hours Minutes in YYYY', 'MM', 'DD']):
        morning_data = group[group.HH24.between(9, 12)]
        if len(morning_data) == 0: continue
        gradient, intercept = np.polyfit(morning_data['HH24'], morning_data['Wind (1 minute) speed in km/h'], 1)
        wind_direction = np.average(morning_data['Wind (1 minute) direction in degrees true'])
        if gradient > 0:
            print("{0:%d, %b %Y} , {1:.2f}, {2:.2f}".format(datetime(*date), gradient, wind_direction))

and then the rest of the code should have a chance of working.
Comment: @JossKirk: I had forgotten to add `df[col] = pd.to_numeric(df[col], errors='coerce')` -- you'll see it if you refresh the web page. Without it, `df[col]` still contains numeric strings. Hence the `TypeError`.
Comment: Since the error message mentions `str` it is probably due the same kind of problem. I've update the code to apply the same fix to the `'Wind (1 minute) direction in degrees true'` column. Be sure to investigate what values you are discarding!
Comment: You could use `if len(morning_data) == 0: continue` to skip groups with no `morning_data`. I've edited the post above to show what I mean.
Comment: Thanks for your answer. Using the code in the question, and your code (which i have edited the question to include) I am recieving the same TypeError message, suggesting that the polyfit is still struggling.
Comment: Thanks! That seems to have resolved that issue. I have updated the question to include the new TypeError that is occurring. This time it is the Wind direction that is being troublesome. Would that also be resolvable through data coercion?
Comment: Thanks so much for you help. That seems to have resolved the issue. Perusing the data there are quite a few rows in which there is no data (presumably because the weather station malfunction for that time period) so it is likely those rows I am discarding. Thanks again!
Comment: Final question. This works well for most files until about 36 months in it gives the error TypeError("expected non-empty vector for x) with regards to the wind speed it seems. Perusing the data, there are many missing days for that particular date. Will this type of data coercion cease to work if there are too many missing rows? Ultimately I will probably skip that particular day, just asking out of curiosity.
Answer: I adjusted .between('9', '12') to .between(9, 12), the np.average calculation to only use morning_data['Wind (1 minute) direction in degrees true'], and added string formatting to the final print statement:
from datetime import datetime
for date, group in df.groupby(['Year Month Day Hours Minutes in YYYY', 'MM', 'DD']):
    morning_data = group[group.HH24.between(9, 12)]
    gradient, intercept = np.polyfit(morning_data.HH24, morning_data['Wind (1 minute) speed in km/h'], 1)
    wind_direction = np.average(morning_data['Wind (1 minute) direction in degrees true'])
    if gradient > 0:
        print("{0:%d, %b %Y} , {1:.2f}, {2:.2f}".format(datetime(*date), gradient, wind_direction))

which ends up working fine (at least without errors), producing:
20, Mar 2000 , 0.47, 83.67

This is the DataFrame I'm getting after copying your sample:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 9 entries, 0 to 8
Data columns (total 62 columns):
hd                                                             9 non-null object
Station Number                                                 9 non-null int64
Year Month Day Hours Minutes in YYYY                           9 non-null int64
MM                                                             9 non-null int64
DD                                                             9 non-null int64
HH24                                                           9 non-null int64
MI format in Local time                                        9 non-null int64
Year Month Day Hours Minutes in YYYY.1                         9 non-null int64
MM.1                                                           9 non-null int64
DD.1                                                           9 non-null int64
HH24.1                                                         9 non-null int64
MI format in Local standard time                               9 non-null int64
Year Month Day Hours Minutes in YYYY.2                         9 non-null int64
MM.2                                                           9 non-null int64
DD.2                                                           9 non-null int64
HH24.2                                                         9 non-null int64
MI format in Universal coordinated time                        9 non-null int64
Precipitation since last (AWS) observation in mm               9 non-null object
Quality of precipitation since last (AWS) observation value    9 non-null object
Air Temperature in degrees Celsius                             9 non-null float64
Quality of air temperature                                     9 non-null object
Air temperature (1-minute maximum) in degrees Celsius          9 non-null float64
Quality of air temperature (1-minute maximum)                  9 non-null object
Air temperature (1-minute minimum) in degrees Celsius          9 non-null float64
Quality of air temperature (1-minute minimum)                  9 non-null object
Wet bulb temperature in degrees Celsius                        9 non-null float64
Quality of Wet bulb temperature                                9 non-null object
Wet bulb temperature (1 minute maximum) in degrees Celsius     9 non-null float64
Quality of wet bulb temperature (1 minute maximum)             9 non-null object
Wet bulb temperature (1 minute minimum) in degrees Celsius     9 non-null float64
Quality of wet bulb temperature (1 minute minimum)             9 non-null object
Dew point temperature in degrees Celsius                       9 non-null float64
Quality of dew point temperature                               9 non-null object
Dew point temperature (1-minute maximum) in degrees Celsius    9 non-null float64
Quality of Dew point Temperature (1-minute maximum)            9 non-null object
Dew point temperature (1 minute minimum) in degrees Celsius    9 non-null float64
Quality of Dew point Temperature (1 minute minimum)            9 non-null object
Relative humidity in percentage %                              9 non-null int64
Quality of relative humidity                                   9 non-null object
Relative humidity (1 minute maximum) in percentage %           9 non-null int64
Quality of relative humidity (1 minute maximum)                9 non-null object
Relative humidity (1 minute minimum) in percentage %           9 non-null int64
Quality of Relative humidity (1 minute minimum)                9 non-null object
Wind (1 minute) speed in km/h                                  9 non-null int64
Wind (1 minute) speed quality                                  9 non-null object
Minimum wind speed (over 1 minute) in km/h                     9 non-null int64
Minimum wind speed (over 1 minute) quality                     9 non-null object
Wind (1 minute) direction in degrees true                      9 non-null int64
Wind (1 minute) direction quality                              9 non-null object
Standard deviation of wind (1 minute)                          9 non-null int64
Standard deviation of wind (1 minute) direction quality        9 non-null object
Maximum wind gust (over 1 minute) in km/h                      9 non-null int64
Maximum wind gust (over 1 minute) quality                      9 non-null object
Visibility (automatic - one minute data) in km                 9 non-null object
Quality of visibility (automatic - one minute data)            9 non-null object
Mean sea level pressure in hPa                                 9 non-null float64
Quality of mean sea level pressure                             9 non-null object
Station level pressure in hPa                                  9 non-null float64
Quality of station level pressure                              9 non-null object
QNH pressure in hPa                                            9 non-null float64
Quality of QNH pressure                                        9 non-null object
#                                                              9 non-null object
dtypes: float64(12), int64(24), object(26)
memory usage: 4.4+ KB

Comment: Needed  to `from datetime import datetime`, see update, also included what the data look like.
Comment: As you can see, I am not getting an error because the data sample does only contain numeric values for the `Wind...` variable. Just fix the data type by coercing and you'll be fine. But also keep the various other adjustments I suggested.
Comment: 2 suggestions: simplify your variable names, and always look at `df.info()` so you can see right away if a data type is off for what you are planning to do, like `string` (`object` in `pandas`) for a regression, etc
Comment: take a look at `.value_counts()` or `.describe()` for that column.
Comment: I have used your code, provided here, and have imported datetime. But it still produces the same type error. I have updated the question to include the new code and the new error message. And if I make wind direction column float it also produces a type error.
Comment: Thanks for your help, TypeError is still occurring, will have to keep working on it.
Comment: Thanks that makes sense. Noob error. I am having problems now with the wind direction variable, but perhaps coercing that variable should resolve that issue as well.
Comment: Good advice. I will have to figure out why the dataframe is interpreting the wind direction as an object. Thanks for your help!


Drupal 8 status code set 301 instead of 302 for urls with out lang code

Question: I have an article with url alias '/gettting-started' and it is translated in several languages.
Lang codes for other languages is set like the following.
For German:de-de, English: en-us.
In 'admin/config/regional/language/detection' Detection and selection languages, priority is set to 'URL' then 'Browser'.
My browser locale set to German first,so when i try to visit the article with the /getting-started, it is redirected to /de-de/getting-started with the status 301.
The status code 301 is creating the problem for me as the response is cached at varnish cache (for 301 response codes).
The other user, with English as priority in Browser language visits /getting-started redirects to /de-de/getting-started
So, instead of sending 301, how can i send 302 redirect for urls without language code as a prefix.
Answer: That redirect isn't a drupal standard feature, it is done by the redirect.module.
It is currently using the default response header of the redirect configuration in \Drupal\redirect\EventSubscriber\RouteNormalizerRequestSubscriber::onKernelRequestRedirect(). That is a bit weird as that is the default for new manually created redirects which usually should be cached.
I think there is an issue in the redirect module to change that to a different configuration or always using 302, if not, feel free to create one.
Comment: Changing 'Default redirect status' in Redirect settings page worked for me.


Problemas com APK react-native

Question: Quando eu compilo o projeto ele gera o APK na pasta C:\Users<usuario>\Desktop\projetos\appTeste\android\app\build\outputs\apk.
Se eu pegar o APK e colocar em um celular ele instala certinho, porém se eu alterar alguma coisa no código, recompilar e gerar o novo APK, por mais que eu exclua o anterior e pegue o novo gerado, ao instalar no celular ele instala a versão antiga do código.

Por exemplo:
  Tenho um texto escrito "Olá".
  Se eu pegar o APK e instalar no celular, ele instala e mostra o texto "Olá".
  Se eu alterar o texto para "Olá nova versão" recompilar e pegar o novo APK gerado, ele instala no celular mas ainda mostra apenas o texto "Olá", que seria da versão anterior do projeto.

Se eu usar um APK Assinado, feito o build direto pelo Android Studio também da o mesmo problema.
Comment: Amigo estou com o mesmo problema, você conseguiu solucionar?
Answer: Tenta "resetar" a build anterior e gere novamente, e alterar a versão do código também, assim como o brother acima disse

Problemas com APK react-native


Python hacking around read only property

Question: I am using some third party code that has some read only properties.
I obviously want to alter them regardless.
I tried using del but I keep getting: AttributeError: can't delete attribute
Here is my code:
for page in suggested_pages:
    url = page.url + '?source=smart_404'
    full_url = page.full_url + '?source=smart_404'

    del page.url
    del page.full_url

    page.url = url
    page.full_url = full_url

Edit people have been asking for the code of the properties
@property
def full_url(self):
    """Return the full URL (including protocol / domain) to this page, or None if it is not routable"""
    url_parts = self.get_url_parts()

    if url_parts is None:
        # page is not routable
        return

    site_id, root_url, page_path = url_parts

    return root_url + page_path

@property
def url(self):
    """
    Return the 'most appropriate' URL for referring to this page from the pages we serve,
    within the Wagtail backend and actual website templates;
    this is the local URL (starting with '/') if we're only running a single site
    (i.e. we know that whatever the current page is being served from, this link will be on the
    same domain), and the full URL (with domain) if not.
    Return None if the page is not routable.
    """
    url_parts = self.get_url_parts()

    if url_parts is None:
        # page is not routable
        return

    site_id, root_url, page_path = url_parts

    if len(Site.get_site_root_paths()) == 1:
        # we're only running a single site, so a local URL is sufficient
        return page_path
    else:
        return root_url + page_path

The code came from Wagtail CMS
Comment: Why do you try to del them first? Why not just assign page.url directly?
Comment: You're going to need to show the code of whatever class `page` is.
Comment: The page class has a `self.url_path` which you can probably alter. Altering it will probably break something. Just because you can change things doesn't mean that changing them does what you want.
Comment: I get this `AttributeError: can't set attribute`
Comment: Obviously there is no setter for `url`, why are you trying to override that behavior?
Comment: @selcuk because I want to alter the url...
Comment: I understand that, but the 3rd party library is obviously not meant to be used like that. You can possibly fork it and create your own version.
Comment: So your code shows that `url` is calculated dynamically. What would it even mean to "set" it?
Comment: I thought that with a dynamic language anything was possible, never mind what the library creator intended?
Comment: @DanielRoseman, so instead of getting the url, I want the url plus a query string.
Comment: @L3viathan https://github.com/torchbox/wagtail/blob/master/wagtail/wagtailcore/models.py#L789
Comment: The main point is that `url` isn't really a property. It's actually a method, but they've defined a getter so that it can be accessed like a property.
Comment: @Barmar but is there anyway i can wipe it off the class definition dynamically, and replace it with my own implementation like my first code snippet?
Comment: @Lee Can you tell us why you want to change the url? Do you want a side to appear under a specific url?
Comment: @syntonym i want to use a query string for metrics collection. This code is part of a library that is itself used in other applications, so i just want to update the urls with no work on the "client" apps
Comment: @Lee So you have an application with wagtail and you want to dynamically change the url of an endpoint? How does the query string play into that?
Answer: In Python, read only properties are generally implemented through the property function.
Example:
class A(object):
    def __init__(self, val):
        self._val = val
    @property
    def val(self):
        return self._val

That means that the read only property is not an attribute of the object, but a special attribute of the class itself.
Hopefully, you can easily add a setter to a property:
def setval(self, val):
    self._val = val
A.val = A.val.setter(setval)
a = A(5)
a.val
-> 5
a.val = 6
-> 6

Or you can also completely replace the property. For your classes, you could do (I use clazz for the name of the class you want to change)
clazz.orig_url = url
def set_url(self, url):
    self._url = url
clazz.url = property((lambda self: self._url), seturl)

You can then do:
page.url = url
print(page.url)

The main caveat is that you have to modify the owning class and not only the instance.


How to back up Cached data in Asp.Net MVC applications?

Question: I am building an application which builds a cache which is of big size. The cost of building the cache is again if it is lost is huge.
Is there any way to save the cached data in asp.Net MVC3 applications ?
Comment: Cached Data? Could mean anything. What are you caching?
Comment: I am caching a list of string (actually html strings that will be fetched from different application using WbeClient). The list can be as huge as of 10,000 +
Answer: I think you should take another approach. could be, create database views for example
Keep in mind a few things, you can't have cached whole time because the app pool it's gonna recycle, so even you change the time interval, always gonna happen.
plus, changes in your app that can trigger the recycle
What trigger a immediate recycle?
anything changes in:

Web.config
Machine.config
Global.asax
Bin directory
App_Code
Views strong typed

What trigger a delayed recycle?
Occur with changes in .aspx, .cs/.vb files.
Comment: Agreed Zach. What should i follow then ?
Comment: reading your other comment, +/- 10k strings from different aplications, you can built database table, drop and create each time the app starts. I think will be more healthy


Why is my multi-layer (with identity activation) neural net converging petter than single layer perceptron?

Question: Before I ask my actual question, I just want to verify that the following is correct:
A multi-layer neural network with an identity activation function is equivalent to a single layer (also with identity activation function) neural network? This is mathematically obvious unless I am missing something.

Moving on to my actual question:
I generated some synthetic data for training. The independent variable is a 100 element array for each sample. The first sample is
$$
S_0 = [0  \ \ 1\ \  2\ \  3\ \  \ldots\ \  99]
$$
There are 1500 training samples. Each training sample follows this relation $$
S_i = S_0 + i*100   \ \ \ \forall i = 1,\ldots,1499
$$
The training labels are simply:
$$
\ell_i = i
$$
I am using pytorch, with MSE as the objective function, and batch gradient descent as the optimization method.
The 4 layer network contains 100, 500, 250, 1 node(s), respectively. The 1 layer network contains 1 node, the output node.
The first plot below shows the MSE error vs. epoch # for 1 layer network. The second plot shows the MSE error vs. epoch # for the the 4 layer network. The y-axis is on a log-scale, and you can see that the 1 layer network converges to a value that is about 3-4 orders of magnitude larger than the 4 layer network. Shouldn't they be converging to the same value because of what is in italics at the top of this post?
This result is very puzzling to me and I do not understand why this is the case.
Answer: For starters, let us show that indeed the underlying relationship of this dataset of size $M$ is linear. We are given that the data is generated by the relationship
\begin{align}
S_i &= S_0 + 100 i \boldsymbol{1}^T \tag*{$\forall i \in \lbrace 0, 1, \cdots, M-2, M-1 \rbrace$}
\end{align}
where $\boldsymbol{1} \in \lbrace 1 \rbrace^{100}$ and $S_0 = [0, 1, \cdots, 98, 99]$. We are also told the label $l_i$ for $S_i$ is $l_i = i$. With the definitions above, it is clear that the analytical solution to this model is indeed $l_i = S_i w$, where $w = [w_0, w_1, \cdots, w_{99}]^T = [10^{-2}, 0, \cdots, 0]^T$. We see this because
\begin{align}
l_i &= S_i w \\
&= S_0 w + 100 i \boldsymbol{1}^T w \\
&= 100 i w_0 \\
&= i
\end{align}
Thus, the data can indeed be captured by a linear model and consequentially one can model it exactly using a single layer neural network with linear activations.

Now the question is, why are you struggling to converge to the unique global minimum of your corresponding optimization? While this data described above looks quite simple, it is actually interesting to see that the data produces a poorly conditioned optimization problem. We first note that your single layer neural network with linear activation is effectively defined as
\begin{align}
f(S) = Sw + b
\end{align}
where $w$ and $b$ need to be estimated using the data. For simplicity, I will define $\hat{S} = [S, 1]$ and $\hat{w}^T = [w^T, b]$, making the neural net $f(\hat{S}) = \hat{S} \hat{w}$. Since you are training with a Mean Squared Error (MSE) cost function, we know the optimization problem to solve is
\begin{align}
\min_{\hat{w}} \frac{1}{2N} \left(X \hat{w} - L\right)^T \left(X \hat{w} - L\right)
\end{align}
where $X^T = [S_0^T, S_1^T, \cdots, S_{M-1}^T]$ is a stacked set of the inputs and $L^T = [l_0, l_1, \cdots, l_{M-1}]$ is a stacked vector of all the output labels. This cost function is convex since the hessian is $H = X^T X$, which is clearly positive semi-definite. Since this function is a paraboloid, we can investigate the condition number of the hessian $H$ to gain insight into how easy it will be to traverse the error surface with first order optimization methods.
Performing a simple computation, one can find for $M = 1500$ that $\text{cond}(H) = O\left(10^{21}\right)$, which is very large. This is telling us that the ellipsoid isosurfaces for the error surface have extremely skewed shapes. An example of this and the optimization progress can be seen in the visual below:

As one can imagine based on the visual, the extremely skewed shapes greatly impact convergence to the actual local (global) minimum and can require many more iterations. In our actual example, since the condition number is so large, the skew is much, much worse than what's shown in the visual.

Anyway, the situation affecting convergence in the above example is largely due to a poorly conditioned optimization problem. The poor convergence rates in the first image make me skeptical about the optimization code being used because a simple momentum-based gradient descent with line searches, which I wrote in a few minutes to investigate this problem, converges to $O(10^{-2})$ error in less than 10 iterations. However, even with a properly coded optimization algorithm, first order optimization methods should struggle to converge quickly for this problem.


lower enveloper and upper enveloper

Question: Let see $A \subset \mathbb{R}^n$, Define $\overline{f}(x) = \limsup_{y\to x} f(y)$
and $\underline{f}(x) = \liminf_{y\to x} f(y)$
$$\underline{\chi_A} =\chi_{A^o}, \text{   }\overline{\chi_A} =\chi_{A^-} $$
I wanna prove that. How can I approach?
Comment: @Marcos I got it! thanks for your help
Answer: If $x \in A^{\circ}$ there exist $B_\varepsilon(x) \subset A$. Then $\inf \{\chi(x): x \in B_\varepsilon(x) - \{x\} \}= 1 = \chi(x)$. If $x \notin A^{\circ}$ there exist $z \in B_\varepsilon(x)- \{x\}$ such that $x \notin A^{\circ}$ and $\inf\{\chi(x): x \in B_\varepsilon(x)- \{x\} = 0 = \chi(x)$.
Hence $ \underline{\chi_{A}} = \chi_{A^{\circ}}$.

lower enveloper and upper enveloper


How to solve this combination problem quickly?

Question: In how many ways can 3 men and their wives be made stand in a line such that none of the 3 men stand in a position that is ahead of his wife?
What is the best way to tackle such problems?
<code> Here is how I am tackling this problem

    M1 M2 M3  -> Men
    W1 W2 W3  -> Wives

    a.
     1  2   3   4    5    6
     W1 M1  W2  M2   W3   M3     ->

    With the above in mind, W1 M1 is a pair and W2 M2 is a pair, W3 M3 is a pair.
    so, there can be 3 interchanges - if we assume that W1 M1 stick to each other.

    b.

     W1 W2  M1  M2   [W3   M3]

W1 W2 can interchange in 2 ways.
M1 M2 can interchange in 2 ways

4 ways..* 2 ( when W3 M3 will be at first) = 8 ways.

    c.  W1 W2  W3  M1   M2   M3

    W1 W2 W3 can be arranged in 3 ways
    M1 M2 M3 can be arranged in 3 ways.
    total ways are 9 ways.

    -----------------

    so adding all -  20 ways.

However, the answer is not 20 ways. What is wrong in my thought process here?
</code>
Answer: Choose two positions for man 1 and woman 1 (husband-wife pair).  $6\choose 2$ ways.  Put woman 1 in the earlier of these two positions.
Choose two remaining positions for man 2 and woman 2.  $4\choose 2$ ways.  Put woman 2 in the earlier of these two positions.
Put man 3 and woman 3 into the two remaining slots, with woman 3 in the earlier position.
${6\choose 2}{4\choose 2}$ ways to do the process.
Answer: A good, if not the best, way to tackle this problem is pick two positions for the first couple to occupy, then another two for the second couple to occupy, which will leave just two positions for the third couple.  This can be done in
$${6\choose2}{4\choose2}{2\choose2}=15\cdot6\cdot1=90$$
ways.  Note that if you remove the condition that each man must stand behind his wife, the number of ways increases by a factor of $2\cdot2\cdot2=8$, which would give $90\cdot8=720$ ways, which makes sense since $720=6!$, which counts the number of ways of arranging $6$ people in a line with no restrictions.
Comment: Ah, I see paw88789 beat me by a minute.  Great minds, and all that....
Comment: I am not getting why divide by 2. How did you get that it should be divided by 2? You are saying for couple 1, they will be half time ahead and half time behind. How did you get that they will be half time ahead and behind?
Comment: @dexterous_stranger, where do you see a division by $2$?  Does paw88789's answer make more sense?
Comment: 6/2 then 4/2 then 2/2 . I am seeing division by 2 for each three position. Where one position is combined by the couple,
Comment: @dexterous_stranger, it sounds like you are unfamiliar with the notation for binomial coefficients.  The symbol ${5\choose 3}$, for example, doesn't mean $5/3$.  Is this what you are unclear on?
Comment: it's basically 5C3
Comment: @dexterous_stranger, yes, and the others are $6C2=15$, $4C2=6$, and $2C2=1$.  The combinatorial interpretation of binomial coefficients involves *counting*, not division (although the computation of them is often done with some division steps).  If you are still unclear on how my answer (and paw88789's) works, I'll try editing it with extra explanation.  Please let me know.
Comment: Ok, I understand C notation very well. Sorry, I didn't know the binomial notation. I might have studied but have forgot by this time. You can elaborate your answer.
Answer: I can see two slips.
Firstly, you forgot WMWWMM. [edit, sorry, you didn't.  However, in b, there are three choices for which pair is together - you have W3M3 together, but it might be W1M1 or W2M2.]
Secondly, WMWMWM can be arranged in six ways, not three:
W1M1W2M2W3M3,
W1M1W3M3W2M2,
W2M2W1M1W3M3,
W2M2W3M3W1M1,
W3M3W1M1W2M2,
W3M3W2M2W1M1.
Similarly for WWWMMM.
Comment: What is the best way to tackle such questions? Do we need to make pairs? If we need to make pairs then what's the point of counting using combinations.
Comment: Your method is good. to check all possibilities.  I suggest, for this one, a shortcut:  Out of the 6!=720 arrangements, M1 is ahead of W1 in half, M2 is ahead of W2 in half, and M3 is ahead of W3 in half.  So only one eighth of the 720 are valid.
Comment: I didn't understand that - "Why should we divide by 8 here - 6!/ 2! 2! 2!?" You say that 2! are the ways where M1 W1 can be arranged. However, this shouldn't be. But, then why are we dividing it?
Comment: The 6! ways include both M1 before W1 and W1 before M1.  There is an equal number with M1 before W1 and with W1 before M1, so half of them have W1 before M1.  That leaves 360.  Then look at W2 and M2.


Ajuda Sintx error

Question: Gostaria de saber onde se encontra meu erro nesta classe em php, se alguém puder ajudar agradeço, o erro é o seguinte:

Parse error: syntax error, unexpected '$_POST' (T_VARIABLE) in
  C:\wamp\www\exercicio1\classe\function.class.php on line 4

class Pessoa{

    var $codigo = $_POST["codigo"];
    var $nome = $_POST["nome"];
    var $altura = $_POST["altura"];
    var $idade = $_POST["idade"];
    var $dt_nasc = $_POST["dt_nasc"];
    var $escol = $_POST["escol"];
    var $salario = $_POST["salario"];

    function crescer($centimetros){
    $this->altura+=$centimetros;

    }

    function formar($titulo){
    &this->idade+=$anos;

    }

    function envelhecer($anos){
    $this->idade+=$anos;

    }

}

Answer: O erro esta no $_POST, não é possível iniciar valores dos atributos da classe desse jeito.
O ideia é você criar um construtor recebendo o parâmetro.
class Pessoa{

    var $codigo;
    var $nome;
    var $altura;
    var $idade;
    var $dt_nasc;
    var $escol;
    var $salario;

    function __construct($POST){
        $this->codigo = $POST['codigo'];
        //.. aqui você inicializa todos os parametros
    }

    function crescer($centimetros){
        $this->altura+=$centimetros;
    }

    function formar($titulo){
        $this->idade+=$anos;
    }

    function envelhecer($anos){
        $this->idade+=$anos;
    }

}

$Pessoa = new Pessoa($_POST);

E na linha &this->idade+=$anos; em vez de $ você colocou &
Answer: Você não pode fazer esse tipo inicialização na propriedes, se precisar definir uma valor padrão dinamico faço no construtor.
Substitua o var que era usado no php4 por uma das tres visibilidade disponivel, public, protected e private.
class Pessoa{

public function __construct($info){
    $this->codigo = $info["codigo"];
    $this->nome = $info["nome"];
    $this->altura = $info["altura"];
    $this->idade = $info["idade"];
    $this->dt_nasc = $info["dt_nasc"];
    $this->escol = $info["escol"];
    $this->salario = $info["salario"];
}

//demais métodos.

Manual - visibilidade

Ajuda Sintx error


Conditional Joint Distributions

Question: I am given random variables x and y where $f(x,y) = 1$ for $0 <= x, y <= 1$ and 0 elsewhere.  I am asked to find $E[X | Y > X]$.
Here's what I have.  I know I made a mistake somewhere or am misunderstanding something so any help would be appreciated.
I get $f_{x|y}(x, y > \frac{1}{2}) = \frac{f_{xy}(x, y > x)}{f_{y}(y > x)}$.
I get $f_{xy}(x, y > x) = \int_{0}^{1} \int_{x}^{1} 1 {\,dy} {\,dx} = \int_{0}^{1} (1-x) {\,dx}  = 1/2$
I also get ${f_{y}(y > x)} = \int_{x}^{1} 1 {\,dy} = 1-x$
However, that means $f_{x|y}(x, y > \frac{1}{2}) = \frac{1}{2(1-x)}$.
This can't be right since when I plug this into $E[X | Y > X] = \int_{0}^{1} x * \frac{1}{2(1-x)} {\,dx}$, this is an undefined value...so clearly I messed up.  What step am I doing wrong?  Or am I not even approaching the problem in the right way?
Thanks!
Comment: Does this answer your question? [How to calculate the conditional mean of $E(X\mid X<Y)$?](https://math.stackexchange.com/questions/2685845/how-to-calculate-the-conditional-mean-of-ex-mid-xy)
Answer: Not sure where $\frac{1}{2}$ is coming from. You can use the formula
$$\mathbb{E}\left(X\mid X < Y\right) = \frac{\mathbb{E}\left(X1_{\left\{X<Y\right\}}\right)}{P\left(X<Y\right)}.$$
Both the numerator and denominator can be computed using
\begin{eqnarray*}
P\left(X<Y\right) &=& \int^1_0 \int^y_0 1 \,dx\, dy,\\
\mathbb{E}\left(X1_{\left\{X<Y\right\}}\right) &=& \int^1_0 \int^y_0 x \,dx\, dy.
\end{eqnarray*}
Comment: Thank you for your answer.  It seems like I am not even using the right formula than in my case.  My question is for your second integral, what does that mean?  Like inside the integral is it just x*f_x(x) which is then x?  Like if f_x(x) = 2 instead, would it be 2x inside that integral?  Or is it f(x,y) * x inside the integral? Is that just saying the general expected value of X where X < Y?  Also I noticed your integral is dx dy rather than dy dx.  Does that make a difference?  Thanks!
Comment: The integral is the unconditional expectation of $f\left(X,Y\right)$, where the function $f$ is given by $f\left(x,y\right) = x1_{\left\{x<y\right\}}$, so you are integrating this function with respect to the joint density. Also $x$ should be on the inside because its value depends on $y$. You first pick a value for $y$ in the outer integral and then integrate over all $x < y$.
Comment: Ok thank you.  I understand.  Sorry, one more question just to make sure I understand.  I see you are taking the integral dx dy.  Can I do dy dx or is there a preferred order I should do it? I guess if I do it dy dx, I have to take it first from y = x to y =1? Nvm, you answered it in your edit!  Thanks!
Comment: You can do it in any order, but either way you have to integrate over the same set, so you need to change the limits of integration accordingly.

Conditional Joint Distributions


using sass to generate RTL & LTR css files in Omega 4

Question: is there any convenient way of using Omega 4 to compile rtl and ltr css files automatically. I mean something like this or this, but for drupal.
Answer: I guess it depends on what you define as being convenient.
The SASS part is easy, since sass support is already setup in Omega. You just need to setup some sort of a sass structure to write the CSS.
To change the language settings, you'll want to invoke the template_preprocess_html() function to override the html that's rendered.
function template_preprocess_html
Comment: thank you, but I'm very lazy and I found simplest answer, although I couldn't find anything on the net.
Answer: found the answer! no need to do any sass related work. I was working with layouts, and all I did was adding a file named mylayoutname.layout-rtl.scss under sass/layouts/mylayoutname/ solved my problem :) it compile automatically to appropriate css file without any hassle.


What NBA players have won both Defensive Player of the Year and Most Valuable Player?

Question: Which NBA players have won both the awards of Defensive Player of the Year (DPOY) and Most Valuable Player?
Have any players won both awards in different seasons across their career? Have any players won both awards in the same season?
If any players have done so, in which season and with which team did these players earn the awards?
Answer: Won MVP and DPOY during the same season:

Michael Jordan, Chicago Bulls, 1988.
Hakeem Olajuwon, Houston Rockets, 1994.

Source

With James expected to again be named NBA Most Valuable Player, it means he will have come up just short in joining Michael Jordan (1988) and Hakeem Olajuwon (1994) as the only players to be named MVP and Defensive Player of the Year for the same season.

Won MVP and DPOY during the course of their careers:

David Robinson, San Antonio Spurs, 1992 DPOY, 1995 MVP
Kevin Garnett, Minnesota Timberwolves/Boston Celtics, 2004 MVP (Timberwolves), 2008 DPOY (Celtics)
Answer: Giannis antetokoumpo has won both awards in the same year in 2020
Source
Comment: Welcome to Sports SE. Please add a source to your answer