Gradle 6.4, Kotlin and JPMS

Question: So, 6.4 is (finally) out and has JPMS support
Eager to give it a try, I started to modify one project of ours in order to play with it.
jjohannes has a (kotlin) example on one of his repo
I tried to copy/paste what mattered into our project here
But unfortunately I didn't succeded.
Whenever I try to run, I always get:

C:\Users\elect\IdeaProjects\kotlin-unsigned\src\main\java\module-info.java:4: error: module not found: kotlin.stdlib.jdk8
      requires kotlin.stdlib.jdk8;

Referring to the module-info.java:
module com.github.kotlin_graphics.kotlin_unsigned {

//    requires kotlin.stdlib;
    requires kotlin.stdlib.jdk8; // error

    exports unsigned;
}

stdlib or stdlib-jdk8 doesn't change much
What am I doing wrong?
Answer: I checked your repo.
You will need these fixes.

Added java plugin.

plugins {
    kotlin("jvm") version "1.3.72"
    java
}

java {
    modularity.inferModulePath.set(true)
}

Use kotlin("stdlib") in all 3 places (2 in build.gradle.kts and 1 in module-info.java).

dependencies {
    implementation(kotlin("stdlib"))
    ...
    components { withModule<ModularKotlinRule>(kotlin("stdlib")) }
}

module com.github.kotlin_graphics.kotlin_unsigned {

    requires kotlin.stdlib;
//    requires kotlin.stdlib.jdk8;

    exports unsigned;
}

For kotlin("stdlib-jdk8"), you need to add it alongside kotlin("stdlib")
dependencies {
    implementation(kotlin("stdlib"))
    implementation(kotlin("stdlib-jdk8"))
    ...
    components { withModule<ModularKotlinRule>(kotlin("stdlib")) }
    components { withModule<ModularKotlinRule>(kotlin("stdlib-jdk8")) }
}

Then, your module-info.java can be
module com.github.kotlin_graphics.kotlin_unsigned {

//    requires kotlin.stdlib;
    requires kotlin.stdlib.jdk8;

    exports unsigned;
}

./gradlew build built successfully for me.
I also add to add missing test dependency:
testImplementation("io.kotlintest:kotlintest-runner-junit5:$kotlintest_version")

UPDATE: Listed for both stdlib and stdlb-jdk8
Comment: Thanks, it finally works. I also tried `stdlib-jdk8` though and it keeps saying `e: Module kotlin.stdlib cannot be found in the module graph`. Have you succedeed with it?
Comment: I made the correction to the answer.  Listed for both `stdlib` and `stdlb-jdk8`
Comment: It works. Is this safe though? Isn't there gonna be problems importing them both this way? (Any interference or whatever)
Comment: No, it should not interfere.  `stdlib-jdk8` internally uses `stdlib`.  In normal cases, you do not have to add `stdlib` since it is transitively available via `stdlib-jdk8`.  You can also experiment by adding it compiletimeonly and see if it works at runtime.
`compileOnly(kotlin("stdlib"))`

For your library, I would recommend to use `stdlib` which makes the library usable on older versions of Java for Android.

Gradle 6.4, Kotlin and JPMS


How to separate introduction from main theme with Lilypond

Question: I'm trying to reproduce a music sheet with Lilypond, and it has an introduction separate from the main theme, but still on the same "line / staff / group of staves", just like so :

And I don't know how to reproduce that. What command would help me in that case ?
My code so far :
<code>%%%%%%%%%%
%% VERSION
%%%%%%%%%%

\version "2.20.0"

%%%%%%%%%%%
%% PIANO UP
%%%%%%%%%%%

mff = ^ \markup { \italic Introduction }

pianoUpper = \relative c' {
    \override DynamicLineSpanner.staff-padding = #3
    \override BreathingSign.text = \markup {
        \musicglyph #"scripts.caesura.straight"
    }

    \clef "treble"
    \key d \minor

    \time 4/4

    <f a>2 \mff \f <f c'> |
    <f bes d>1 |
    \> << {bes4( a)} \\ {d,2} >> <e g>2 |
    %   \once \override Hairpin.to-barline = ##f
    f \fermata \breathe \! \bar "||" <a f'> \f |
    <c e> <b d> |
    <c e> <a c> |
}

%%%%%%%%%%%%
%% PIANO LOW
%%%%%%%%%%%%

pianoLower = \relative c {
    \override BreathingSign.text = \markup {
        \musicglyph #"scripts.caesura.straight"
    }
    \clef "bass"
    \key d \minor

    f2 a |
    bes1 |
    g2 c, |
    f \fermata \breathe f |
    c g' |
    c, f |
}

%%%%%%%%
%% SCORE
%%%%%%%%

\score {
    <<
    \new PianoStaff <<
    \new Staff = "upper" \pianoUpper
    \new Staff = "lower" \pianoLower
    >>
    >>
}
</code>
produces :
Answer: This trick puts your introduction inside of a score inside of a markup inside of the pianoStaff instrumentName.

See the code here: lilypond code of answer in lilybin.com
Comment: That's really cool ! A question, though. Is there a way of knowing the length of the introduction, in order to align with the rest of the sheet when indenting ? Especially since I'm trying to write a collection, and similar alignment would be neat within the different pieces.
Comment: @JambonSama I'm afraid for now there is none I am aware of. Thankfully it is a single value that you have to manually code though.


Cannot insert UTC_TIMESTAMP() with pdo prepared statement

Question: How can I use UTC_TIMESTAMP() with this prepared insert statement?  It is not liking the format/binding and errors out.
// prepare the statement
$stmt = $db->prepare("INSERT INTO accounts (
        account_username,
        account_password,
        account_email,
        fname,
        lname,
        dtCreated
    ) VALUES (
        :account_username,
        :account_password,
        :account_email,
        :fname,
        :lname,
        :dtCreated
    )
");

//set bindings
$binding = array(
    'account_username' => $_POST['username'],
    'account_password' => $newhash,
    'account_email' => $_POST['email'],
    'fname' => $_POST['fname'],
    'lname' => $_POST['lname'],
    'dtCreated' => UTC_TIMESTAMP()
    );

// execute the insert
$stmt->execute($binding);

Answer: Specify the UTC_TIMESTAMP() in the statement itself and not as a parameter.
$stmt = $db->prepare("INSERT INTO accounts (
        account_username,
        account_password,
        account_email,
        fname,
        lname,
        dtCreated
    ) VALUES (
        :account_username,
        :account_password,
        :account_email,
        :fname,
        :lname,
        UTC_TIMESTAMP()
    )
");

//set bindings
$binding = array(
    'account_username' => $_POST['username'],
    'account_password' => $newhash,
    'account_email' => $_POST['email'],
    'fname' => $_POST['fname'],
    'lname' => $_POST['lname']
);

// execute the insert
$stmt->execute($binding);

If you do this:
$stmt = $db->prepare("INSERT INTO accounts (
        account_username,
        account_password,
        account_email,
        fname,
        lname,
        dtCreated
    ) VALUES (
        :account_username,
        :account_password,
        :account_email,
        :fname,
        :lname,
        :dtCreated
    )
");

//set bindings
$binding = array(
    'account_username' => $_POST['username'],
    'account_password' => $newhash,
    'account_email' => $_POST['email'],
    'fname' => $_POST['fname'],
    'lname' => $_POST['lname'],
    'dtCreated' => UTC_TIMESTAMP()
);

// execute the insert
$stmt->execute($binding);

Then UTC_TIMESTAMP() gets turned into a string (because its not a PHP function) and the SQL statement ends up being turned into:
    INSERT INTO accounts (
        account_username,
        account_password,
        account_email,
        fname,
        lname,
        dtCreated
    ) VALUES (
        'jblow',
        'password',
        'email@example.com',
        'Joe',
        'Blow',
        'UTC_TIMESTAMP()'
    );

And if you notice, UTC_TIMESTAMP() is specified as a string which will not work.
Answer: You can do that using the NOW() MySQL function I guess.
Comment: NOW() is the local server time... I want to keep things consistent and always store UTC.


Checking for Model Errors in Entity Framework

Question: In my controller, I have an action that takes in 3 arguments (primary_key, property, and value) and uses reflection to set the value on corresponding model. (Found by its primary key)
I thought I could catch the model if it was invlaid with ModelState.IsValid but it evaluates as true. Now it goes to db.SaveChanges(); which throws exception.
The ModelState is valid. (Apparently it is no the model instance as found by the primary key and actually refers to my three inputs).
I thought I could check my model for errors with the following line...
if (System.Data.Entity.Validation.DbEntityValidationResult.ValidationErrors.Empty)

But I am getting a "missing object reference" error.
I have no idea what that means. (New to C# and everything else here.) Any help?
EDIT 1 - SHOW MORE CODE:
Validations
[Column("pilot_disembarked")]
[IsDateAfter(testedPropertyName: "Undocked",
             allowEqualDates: true,
             ErrorMessage = "End date needs to be after start date")]
public Nullable<System.DateTime> PilotDisembarked { get; set; }

Custom Validatior
public sealed class IsDateAfter : ValidationAttribute, IClientValidatable
{
    private readonly string testedPropertyName;
    private readonly bool allowEqualDates;

    public IsDateAfter(string testedPropertyName, bool allowEqualDates = false)
    {
        this.testedPropertyName = testedPropertyName;
        this.allowEqualDates = allowEqualDates;
    }

    protected override ValidationResult IsValid(object value, ValidationContext validationContext)
    {
        var propertyTestedInfo = validationContext.ObjectType.GetProperty(this.testedPropertyName);
        if (propertyTestedInfo == null)
        {
            return new ValidationResult(string.Format("unknown property {0}", this.testedPropertyName));
        }

        var propertyTestedValue = propertyTestedInfo.GetValue(validationContext.ObjectInstance, null);

        if (value == null || !(value is DateTime))
        {
            return ValidationResult.Success;
        }

        if (propertyTestedValue == null || !(propertyTestedValue is DateTime))
        {
            return ValidationResult.Success;
        }

        // Compare values
        if ((DateTime)value >= (DateTime)propertyTestedValue)
        {
            if (this.allowEqualDates)
            {
                return ValidationResult.Success;
            }
            if ((DateTime)value > (DateTime)propertyTestedValue)
            {
                return ValidationResult.Success;
            }
        }

        return new ValidationResult(FormatErrorMessage(validationContext.DisplayName));
    }
 }

Controller Action
    [HttpPost]
    public ActionResult JsonEdit(string name, int pk, string value)
    {
        Voyage voyage = db.Voyages.Find(pk);

        var property = voyage.GetType().GetProperty(name);

        if (Regex.Match(property.PropertyType.ToString(), "DateTime").Success)
        {
            try
            {
              if (Regex.Match(value, @"^\d{4}$").Success)
              {
                var newValue = DateTime.ParseExact(value, "HHmm", System.Globalization.CultureInfo.InvariantCulture);
                property.SetValue(voyage, newValue, null);
              }
              else if (value.Length == 0)
              {
                  property.SetValue(voyage, null, null);
              }
              else
              {
                var newValue = DateTime.ParseExact(value, "yyyy/MM/dd HHmm", System.Globalization.CultureInfo.InvariantCulture);
                property.SetValue(voyage, newValue, null);
              }
            }
            catch
            {
                Response.StatusCode = 400;
                return Json("Incorrect Time Entry.");
            }
        }
        else
        {
            var newValue = Convert.ChangeType(value, property.PropertyType);
            property.SetValue(voyage, newValue, null);
        }

        if (ModelState.IsValid)
        {
            db.SaveChanges();
            Response.StatusCode = 200;
            return Json("Success!");
        }
        else
        {
            Response.StatusCode = 400;
            return Json(ModelState.Keys.SelectMany(key => this.ModelState[key].Errors));
        }

    }

Comment: You need to show us more of your code.  Show us the model and the function throwing the exception.
Comment: @Amy - Added more code...!
Comment: What line is throwing the Exception?
Answer: When any value of your model is null at that time ModelState.IsValid.So first check your model data.


Filter with callback misses some condition

Question: I have the following code to filter an array in PHP;  it works fine.  But after adding more filtering conditions, it stopped working.
Working code
$s_item = array_values(array_filter($itmlist, function ($itmlist) {
    return ($itmlist['itmslsprc'] > 0);
    }
));

Code that doesn't work after adding more conditions
$s_item = array_values(array_filter($itmlist, function ($itmlist) {
    return ($itmlist['itmslsprc'] > 0 && $today >= $itmlist['itmslsfr']  &&  $today <= $itmlist['itmslsto']);
    }
));

Both $today and the other dates have been initialised in format 2021-01-12 00:00:00.
Can anyone tell what might have gone wrong?
Thanks in advance for any advice.
Comment: where is `$today` defined
Comment: if you want to use $today in a nested function you have to pass it! Probably your $today variable is undefined
Comment: What's the problem with different formats? "Doesn't work" sounds pretty broad, so what have you tried to debug it?
Answer: if your today is outside the function you can use it inside the function like this :
$s_item = array_values(array_filter($itmlist, function ($itmlist) use($today) {
    return ($itmlist['itmslsprc'] > 0 && $today >= $itmlist['itmslsfr']  &&  $today <= $itmlist['itmslsto']);
    }
));

Answer: As mentioned, you can just import the variable in the the callback's scope, relying on use statement.
PHP Closures don't bind automatically to the outer scope (like it happens in languages like Javascript).
But if you stick to a more functional programming paradigm (and work with PHP >=7.4), you can avoid the manual import and rely on PHP Arrow Functions.
$today = // your value here

$s_item = array_values(array_filter($itmlist, fn($itmlist) =>
    $itmlist['itmslsprc'] > 0 &&
    $today >= $itmlist['itmslsfr'] &&
    $today <= $itmlist['itmslsto']
));

fn() => returnExpression will have access to $today.
Note
This is a copy-by-value, as reported in the manual:

When a variable used in the expression is defined in the parent scope it will be implicitly captured by-value.

In case you want to modify the outer variable, and want no additional closure parameters, then use statement is your best option.
Answer: if your today is outside the function you can use "use"
Comment: Please avoid duplicating existing answers, unless you want to add something that hasn't been covered in other answers


Difference between assigning any value to a[ ] and a[ : ]

Question: I am a beginner in image processing i am learning it using OpenCV library.
While working with it i came across the following code snippet
import numpy as np
img = np.zeros((300, 512, 3), np.uint8)
img[:] = [b, g, r]

When i do img=[b,g,r]
It does not work as expected. I understand that this is may be because of the dimension of the array as initialized with numpy but i am not able to visualize it. Can anyone explain how this syntax work ?
Comment: From where the `b`/`g`/`r` came from? Did you receive an error message, if so can you add it?
Comment: I think you will find the answer here https://stackoverflow.com/questions/29810632/different-ways-of-deleting-lists
Comment: Without knowing what you did expect, I'm not sure how I can answer your question. Please provide an MCVE.
Answer: img[:] will do in-place reassignment. It'll essentially reassign img to a whole new list, but in the same place in memory.
If you do img = [b,g,r], your variable img will still point to an array [b,g,r], however, it is pointing to a completely new location now. The idea is that if you have img already, there's no need to allocate new space for [b,g,r].
Comment: sorry i made a mistake i meant img=[b,g,r]
Answer: Let's simplify your example a bit, and define the 3 unknowns:
In [140]: b, g, r = 1,2,3
In [141]: x = np.zeros((4,2,3),int)

The img=[b,g,r] just assigns the list to the variable img.  The fact that you already had assigned a value to that variable doesn't make a difference.  In Python we don't "initialize" variables (as is sometimes done in other languages).
In [142]: [b,g,r]
Out[142]: [1, 2, 3]

Assigning that list to x[:] results in:
In [143]: x[:] = [b,g,r]
In [144]: x
Out[144]:
array([[[1, 2, 3],
        [1, 2, 3]],

       [[1, 2, 3],
        [1, 2, 3]],

       [[1, 2, 3],
        [1, 2, 3]],

       [[1, 2, 3],
        [1, 2, 3]]])

Effectively the list is converted to a (3,) array, np.array([b,g,r]).  Those values are assigned, with broadcasting thus:
(3,) => (1,1,3) => (4,2,3)

And by the broadcasting rules, this would only work if b,g,r were scalars.


Delete All Children on table that REFERENCES itself

Question: Let's say we have a table that looks like this:
CREATE TABLE Test
(
    Id INT NOT NULL PRIMARY KEY IDENTITY(1,1),
    [Name] NVARCHAR(50) NOT NULL,
    ParentId INT NULL FOREIGN KEY REFERENCES Test(Id)
)

In this table, we have a hierarchy of data that may look something like:
INSERT INTO Test (Name)
VALUES ('ABC'), ('DEF'), ('HIJ');
GO

INSERT INTO TEST (Name, ParentId)
VALUES ('KLM', 1), ('NOP', 1), ('QRS', 2), ('TUV', 2), ('XYX', 3)
GO

INSERT INTO Test (Name, ParentId)
VALUES ('AAB', 4), ('AAC', 4), ('AAD', 4)

How can I delete id 1 and all of its children without using cascade delete?
Answer: You would have to use an rCTE (recursive Common Table Expression) to recurse through the hierachy. Then you can JOIN that data to your table and delete the relevant rows:
DECLARE @ID int = 1;

WITH rCTE AS(
    SELECT T.Id
    FROM dbo.Test T
    WHERE T.Id = @ID
    UNION ALL
    SELECT T.Id
    FROM rCTE r
         JOIN dbo.Test T ON r.Id = T.ParentId)
DELETE T
FROM dbo.Test T
     JOIN rCTe r ON T.Id = r.Id;

Note that unlike some some examples you may have seen, such as where a CTE is used to DELETE duplicate rows, a rCTE is not updatable (due to the use of the UNION). As such you can't simply perform a DELETE operation on the CTE at the end (DELETE FROM rCTE) and you have to use it as a JOIN.
DB<>Fiddle
Comment: can you please explain this, I don't understand what this is doing?
Comment: My opening sentence explains this, @3xGuy: *"You would have to use an rCTE (recursive Common Table Expression) to recurse through the hierachy. Then you can JOIN that data to your table and delete the relevant rows:"* [Guidelines for Defining and Using Recursive Common Table Expressions](https://learn.microsoft.com/en-us/sql/t-sql/queries/with-common-table-expression-transact-sql?view=sql-server-2017#guidelines-for-defining-and-using-recursive-common-table-expressions)
Comment: If you change the `DELETE` to a `SELECT` you can return the data from the rCTE, @3xGuy . Effectively an rCTe is a self referencing Common Table Expression, and one of it designs is to specifically deal with hierarchical data.


Available EMP protection devices?

Question: This weekend there was a solar CME warning that finally had no perceptible impact. However, I was wondering what could be done to mitigate the effects in case a large one took place. Are there any devices available in the market to protect servers from the pernicious electromagnetic effects (or help paliate them in some way)? (Like for example, an special power source or cage)
I'm not thinking on protecting a large server farm, but perhaps one or two critical machines.
UPDATE:
Actually there are RF/EMI shielded racks on the market, what I don't know is if they could withstand a solar flare. Also, critical server rooms can be built as a Faraday cage as one of the answers pointed. It would be interesting to know the percentage of server rooms that are built with this in mind.
Answer: Certainly having your equipment on surge-protected power supplies, ideally UPSs, makes some sense for this scenario but the only way to add significant EM protection would be by placing your hardware inside a well grounded Faraday cage of some sort. I'm not aware of any commercially available products in this area and obviously the size of installation would be key to the resources needed to carry out this work but I do know that given the time and money it would be possible to build an entire data centre inside an effective Faraday cage if you so desired.
Comment: "Into/conductors going in and out" - odd way of putting it but no, that said the whole concept of 'Earth' is a variable one, and if the EM was strong enough then certainly it could affect the earth levels somewhat but then that would still be common to all local devices so the impact would be limited and also if it were that bad, well there's be no users to complain because we'd all be dead - as for the power lines I mentioned using surge-protection and/or UPSs which would deal with that issue.
Comment: Fascinating topic.  Wouldn't the EM event make its way into the Faraday cage via the conductors going in and out?  While the data connectivity could be fiber, it could still be carried on the power lines...
Comment: I grant that I am no scientist, but it is my understanding that an EM event would contain a surge across a wide part of the EM spectrum, and as such, the conductors penetrating the cage would be susceptible to pick up as well as re-radiate this energy, particularly at higher RF frequencies.  Probably not something a typical UPS would be able to filter or prevent?
Comment: @TevoD: No, but if the UPS is close enough to the faraday cage (either inside or outside) then the conductors (antenna's) will be too short to effectively radiate.
Answer: The surge generated by a solar event is generally caused by swift oscillations in Earth's magnetic field which in turn induces current in long conductors. In extreme cases (I think that 1980's era X-class solar flare that blacked out a lot of Canada was one) the magnetic fields from the CME itself reach the ground for big-big fun; we haven't yet had a flare that size in the Internet era.
This means there are two surges you have to defend against:

The indirect-effect surge coming down your mains feed thanks to those long transmission lines acting as antenna for the magnetic fields.
The direct-effect induced current in your own electrical plant.

The first type can be handled by Ye Olde Surge Suppressors, or if you want to get fancy with it a line conditioner will do the job better.
The second type is a lot harder to defend against. Happily, you're not likely to be vulnerable to it since it requires long, straight runs of conductor. Long twisted-pair runs may get higher incidents of bad signals on the wire but shouldn't induce enough voltage to smoke equipment. Long power runs may pick up dirtier power than normal, so equipment sensitive to such small voltage changes may act badly.
A Farraday Cage does offer some protection, so long as it doesn't share a ground with the powered equipment inside.
Comment: So it is not a good idea to ground the Faraday cage to a regular AC socket ground contact.

Available EMP protection devices?


Is the audio quality of a digital piano sufficient for enjoying classical music on YouTube?

Question: Some months ago, I have started composing piano music in mostly romantic style (somewhere between Brahms, Rachmaninoff and Albeniz for most pieces).
I have recorded some of them as videos using my digital piano but so far I have just posted those videos publicly on my Facebook page. I have used for that recording a BOYA BY-LM10 microphone. Even though that seems to be rather intended for recording speech, the quality is better than when using the built-in microphone of my tablet with which I record the videos.
My first question:

Is such an audio quality acceptable for posting on YouTube?

This being said, I will soon have the opportunity of accessing a grand piano at the home of friends where I may possibly do some recording. But that would involve much more logistics, and apart from that I am wondering:

Would it be in that case advisable to use as well a different kind of microphone (i.e. an omnidirectional one)? Would that make a huge difference in audio quality, how useful for YouTube?

A friend of mine who is very knowledgeable about social networks told me after listening to one on my Facebook page that in his opinion, the audio quality of the digital piano would be largely sufficient even for YouTube. But I am not very sure, especially as that friend is much less knowledgeable about *classical * style music. Others say (generally speaking) that for a YouTube video in order to be appreciated, the quality of the audio is much more important then the one of the image.
Comment: It's a digital piano - is there some way you can plug direct instead of using a mic? With the grand, it's worth recording in stereo. I tend to  listen on cans, and stereo recordings are usually much more listenable.  For those who just use their speaker phone, it's not going to make much difference to what they hear, but I'd have thought listening to serious music won't be done using a tinny little speaker!
Comment: https://www.soundonsound.com/techniques/piano-recording

Your mic specs:

 Frequency response:    35Hz-18KHz
 Signal/Noise:    74dB or more
 Reception sensitivity:    -30dB±3dB / 0dB=1V/Pa,1KHz
Answer: The main advantage of a digital piano for recording is that you don't need a microphone at all.  Every digital I've seen has a headphone jack and/or a line out; with the right cable you should be able to plug your piano directly into a tablet or laptop and get a good recording without a lot of fuss.
Your friend's grand piano may sound better, but choice and placement of mics is a complicated subject; a web search for "piano recording" should give you lots of resources.
I'm not sure what "good enough for YouTube" means.  YouTube videos can represent audio with high fidelity, so the limiting factor will be how they're played back; for example, somebody using headphones is going to hear more than somebody using phone speakers.
Comment: Well I have tried connecting the digital piano to the computer but it doesn't take into account the pedal (damper). I have sought on various forums, and that problem seems to be common! No solution suggested...
Comment: BTW feel free to get an impression of how it sounds right now, by searching the 3 words "wolfgang berndt karfreitag" on youtube. I'd say not too bad!
Answer: I don't see why not. I have a Yamaha digital piano and the sound quality is great, even in the speakers, much better then the older acoustic piano I had. I bet you couldn't tell it apart from an actual piano.
Of course there are different digital pianos, so it depends on the model you have, but based on the ones I knew I believe it's true.
Considering the little loss of quality in any recording you make, the difference between a good digital piano and an acoustic piano would be even more unnoticeable on Youtube.
Comment: Well I guess it is also (maybe most) about the feeling when playing on a real piano. Even though the touch quality of the Yamaha is good, you just feel you can make so much less nuances, compared to a real one.


How to handle strongly correlated but not perfectly collinear dummies

Question: I am using several dummy variables in a GLM model implemented in R with a logit link function.
However, the coefficient of one of the dummy variables is not shown in the results with the warning that "Coefficients: (1 not defined because of singularities)".
When I check the correlations between the dummy variables with the cor() function, I see that 2 of them are indeed strongly, though not perfectly correlated (-0.6..).
I understand that one of the dummy variables could be removed if we had perfect collinearity. But how can this be handled in a situation where the correlation is "only" strong but not perfect without losing information (as I can not perfectly predict the correlated dummy)?
Comment: This seems like an issue of multicollinearity. A correlation of 0.6 doesn't sound like it should cause a problem. Does your model matrix have full rank?
Answer: The note you see from R about a missing coefficient is not caused simply by correlations between columns (which are not in themselves a problem) but rather by the fact that your model matrix is not of full rank. Basically, your model is over-parametrized.
This is a problem caused by all the columns of your design matrix together. It is not a problem that can be analysed in terms of pairwise correlations alone. When all the columns of the model matrix are considered together, they are in fact perfectly collinear in one particular dimension, otherwise you would not get the message that you quote.
You will get a similar message as part of the output from summary(fit) whenever fit is a linear model or generalized linear model fit and there are one or more NA coefficients.
It is not a warning or message in the formal R sense, but just a helpful note attached to the formatted table of coefficients.
Looking down the table of coefficients will tell you which coefficient is NA.
Usually the message is a signal that you should reformulate your model matrix to avoid the over-parametrization.
You might also find alias(fit) useful.
Although the output from that function is brief, it attempts to show you exactly which coefficients are aliased with which others.
People have asked about similar messages many times on this help forum, for example

Logistic regression "1 not defined because of singularities"
Coefficients: (5 not defined because of singularities)
How to deal with an error such as "Coefficients: 14 not defined because of singularities" in R?

If you provided more information about your data and glm call we might be able to give specific advice about your model matrix should be reformulated.


What are the hypothesis of using linear discriminant analysis (LDA)?

Question: I want to use LDA for a classification problem.

I have 14 independent variables including 11 continuous and 3 categorical variables.
My response factors is binary {0, 1}.
I have checked the normality of my continuous variables and it is OK.

My questions are

Could I use the categorical variables with LDA?
Should I report the co-variance?

If yes, what is acceptable for the covariances? Do they need to be close together?
Answer: Short answer might be no. One of the most cited arguments against LDA is that it is a parametric method (it assumes unimodal Gaussian likelihoods). If the distributions are significantly non-Gaussian, the LDA projections may not
preserve complex structure in the data needed for classification. LDA will also fail if discriminatory information is not in the mean but in the variance of the data. Logestic regression tends to yield better estimate than does LDA if predictors include categorical variables.


c++ random number from 1 to 8 digits

Question: I'm trying to generate a random from 1 - 99,999,999
However, its not ever getting that high. I believe it has something to do with the size of an int.
Here is what I'm trying
<code>(1 + rand() % 99999999)
</code>
Thanks
Comment: Apart from being a terrible formula, I don't suppose you know the value of RAND_MAX for your current implementation? Methinks thats about the limit you will see using this mechanism. (and in case you're using VC++, their 32bit RAND_MAX is only 0x7FFF, or 65535). See [](http://en.cppreference.com/w/cpp/numeric/random) in the C++ 11 standard library, and use *that*. It's much more robust than the crap-legacy interface and manages the modulus probability balancing for you. `std::random_device` + `<>` + `std::uniform_int_distribution` together do exactly what you want.
Comment: `RAND_MAX` is most likely 32k. Use ``. It's a lot better anyway.
Comment: If you have access to c++11 - http://channel9.msdn.com/Events/GoingNative/2013/rand-Considered-Harmful - Don't use rand.
Comment: what platform/compiler?
Comment: I'm pretty sure an `int` can contain much more than that.
Answer: your code will not work because value returned by rand have a maximum value RAND_MAX

int rand (void);
Generate random number
Returns a pseudo-random integral number in the range between 0 and
RAND_MAX.

RAND_MAX
Maximum value returned by rand This macro expands to an
integral constant expression whose value is the maximum value returned
by the rand function.
This value is library-dependent, but is guaranteed to be at least
32767 on any standard library implementation.
Comment: @Phlip: `sizeof RAND_MAX` will tell you exactly nothing.
Comment: @Phlip this question have nothing to do with data type size.
Comment: hence, printing out `sizeof rand()`, and `sizeof RAND_MAX` would shed some light on what platform `int` you have.
Answer: To generate a random number between min and max, and they should be long data type use:
<code>long randamNum = rand()%(max-min + 1) + min;

(Includes max and min)
</code>
For more clearity refer Generating random integer from a range
Comment: I'll bet you `sizeof(long) == sizeof(int)` for the OP. 16-bit ints aren't that common anymore and I have yet to know of something with 64-bit one but not the other.
Comment: lol, no. An `int` can contain that number. That's not the problem.
Comment: I can't see how type `long` or even `long long` will change anything

c++ random number from 1 to 8 digits


Matrix equation of Kronecker product

Question: $\ I ⊗ I ⊗ A = Z ⊗ I$
$A$ is a known matrix. $I$ is the identity matrix. $A$ and $I$ are n by n matrices. $⊗$ is the Kronecker product.
Is there a way to find $Z$ in terms of $A$ and $I$?
Thanks in advance.
Answer: A general approach how you can solve equations involving Kronecker products for one of its factors is vectorization: since $A \otimes B$ is linear in $A$ for given $B$ and vice versa, you can rewrite it as ${\rm vec}\{A \otimes B\} = \tilde{B} \cdot {\rm vec}\{A\} = \tilde{A} \cdot {\rm vec}\{B\}$. This would allow you to solve for the vectorized version of your desired matrix in standard ways (e.g., using Least Squares).
Whether your equation even has a solution will of course depend on the column space of the matrix $\tilde{A}$ or $\tilde{B}$, respectively.
That said, in your particular problem you are trying to find $Z$ such that $I \otimes A = Z \otimes I$ (note that the first two identity matrices can be combined into one since the Kronecker product is associative and $I \otimes I = I$). Solving this will be tough: $I \otimes A$ is a block-diagonal matrix with $A$ appearing as repeated blocks. $Z \otimes I$ is a matrix containing identity matrices stacked horizontally and vertically and scaled by the elements of $Z$. I would claim that this equation only has a solution when $Z$ and $A$ are diagonal. In this case, you don't need the matrices and their vectorizations, just extract the diagonals and equate those.

Matrix equation of Kronecker product


Diagonalizable operator $D$ on $R^3$ and a nilpotent operator $N$

Question: Let T be the linear operator on R3 which is represented by the matrix
$\begin{pmatrix}
3  &1 &-1  \\
 2 &2& -1  \\
 2 &2& 0  \\
\end{pmatrix}$
in the standard ordered basis. Show that there is a diagonalizable operator $D$
on $R^3$ and a nilpotent operator $N$ on $R^3$ such that $T = D + N$ and $DN = ND$.
Find the matrices of $D$ and $N$ in the standard basis.
We have the relationships
$D=c_1E_1+...+c_kE_k$
and
$N=(T-c_1I)E_1+...+(T-c_kI)E_k$
but I do not know how to use it. Can anyone explain? In this case the characteristic polynomial is $(x-1)(x-2)^2$. I do not know if I should find the matrix of E_i, also I did not understand either if In this formula should I consider the repetition of eigenvalue?
Answer: The minimal polynomial for $T $ is $p (x)=(x-1)(x-2)^2$. Define  $$\left\{
\begin{array}{c}f_1 (x):=\dfrac{p (x)}{x-1}=(x-2)^2\\f_2 (x):=\dfrac{p (x)}{(x-2)^2}=(x-1)\end{array}\right.. $$ Since  $f_1$ and $f_2$ are relatively prime, you can find  $g_1$ and $g_2$ such that $f_1 g_1+f_2g_2=1$(use Euclidean algorithm). Now define  $E_i=f_i (T)g_i (T),i\in\{1,2\}.$ Finally take $D=E_1+2E_2$ and $N=(T-I)E_1+(T-2I)E_2$.
Comment: @KumarNilesh In this case $(x-2)^2-(x-1)(x-3)=1$. You can use [polynomial long division](https://en.wikipedia.org/wiki/Polynomial_long_division) and [Euclidean algorithm.](https://en.wikipedia.org/wiki/Polynomial_greatest_common_divisor#Euclidean_algorithm)
Comment: I'm not familiar with this method. How does it work in general? Does it work for more than two eigenvalues?
Comment: @TheoBendit Yes, it will work. But I don't think this is a good computational technique for higher order matrices. You can find a detailed explanation in *Linear Algebra* by Hoffman and Kunze(section 6.8).
Comment: I got it in the form of jordan. I'll try with this other method.
Comment: @ThomasShelby please tell me how to find $g_1$ and $g_2$. I know there exist such polynomial but I don't know how to find them.
Answer: Steps:

Find a Jordan basis for $T$
Express $T$ as a matrix in terms of this Jordan Basis (i.e. in Jordan Normal Form).
Separate this matrix into a sum of two matrices: a diagonal matrix consisting of all the diagonal entries, and an upper triangular matrix consisting of the off-diagonal $1$s. The former is diagonal, the latter is nilpotent, and their product commutes.
If you want, find the standard basis representations for these operators, to present the solution in the same form as you were presented the problem.

Let me know if you need more help than this.
Comment: Fortunately, the matrix is small. I was trying to find the $E_i$ matrix by Lagrange interpolation, but it is not working.
Comment: @RicardoFreire Actually, while I have your attention, I have a small bone to pick with you. You deleted your question about finding the Jordan basis for a $6 \times 6$ matrix, and I only noticed when I had basically finished writing my answer up. :-(
Comment: I exclude because I wrote wrong and I thought no one would help. I can try to rescue if you want to post your answer.
Comment: @RicardoFreire It's too late. The answer was saved only in the clipboard of a computer that is now switched off. While new questions tend to get the most attention, there are a good number of people who will try to answer the tougher questions that didn't get answered.
Answer: The Jordan normal form of your matrix will be $J=\begin{pmatrix}1&0&0\\0&2&1\\0&0&2\end{pmatrix}$.  That is, there is a basis $\{v_1,v_2,v_3\}$ rel which the matrix for $T$ is the above matrix.
Let $P$ be the matrix whose columns are the basis vectors.   Then we have $P^{-1}TP=J$.
But $J=\begin{pmatrix}1&0&0\\0&2&0\\0&0&2\end{pmatrix}+\begin{pmatrix}0&0&0\\0&0&1\\0&0&0\end{pmatrix}= D'+N'$.
Now just solve for $T$.  That is, $$T=PJP^{-1}=P(D'+N')P^{-1}=PD'P^{-1}+PN'P^{-1}=D+N$$.
Finally you need the Jordan basis (a basis of generalized eigenvectors), $\{v_1,v_2,v_3\}$ to find $N$ and $D$ explicitly.
To this end, I have computed the following Jordan basis:  $\{(1,0,2),(-1,0,1),(1,1,2)\}$.


p:selectOneMenu preselects previous item when noSelectionOption item is present

Question: There is a problem with p:selectOneMenu selection when adding a f:selectItem.
View:

    <f:selectItem itemLabel="Select" noSelectionOption="true" />
    <f:selectItems value="#{selectionTest.allNames}" var="varName" itemLabel="#{varName}" itemValue="#{varName}" />

Model:
private List<String> allNames;
private String selectedName;

public MenuSelectionTestBean(){
    allNames = new ArrayList<String>();
    allNames.add("Ahmed");
    allNames.add("Mohamed");
    allNames.add("Ibrahim");
    allNames.add("Walid");

    selectedName ="Walid";
}

Result:

The item "Walid" should be selected, however "Ibrahim" is selected instead. I think that PrimeFaces selects the item depending on its index in the list and not the value of the item.
How is this caused and how can I solve it?
Answer: I could reproduce your problem. I looked in the PrimeFaces source code. It's a bug in PrimeFaces SelectOneMenuRenderer which exposes only when  an item with noSelectionOption="true" is been used. The renderer won't generate the HTML <option> element of the backing <select> element for that, but it will generate the
 of the visual
 list for that. In other words, the <select> contains in your case 4 items and the
 5 items. The
 is selected by JavaScript based on the index of the <option selected>, exactly as you suspected. That explains your problem.
Your best bet is to report it as a bug to PrimeFaces guys. The renderer should also generate the <option> for that, or the JavaScript should select the option based on value instead of on index.
In the meanwhile, replacing the noSelectionOption="true" by itemValue="#{null}", or also itemValue="" when JSF is configured to interpret empty string submitted values as null, should solve your problem.
Comment: it is realy an issue http://code.google.com/p/primefaces/issues/detail?id=4178&colspec=ID%20Stars%20Type%20Status%20Priority%20TargetVersion%20Reporter%20Owner%20Summary&start=100 i found it after posting this problem; thanks


How do I dynamically adjust the tooltip box of a zingchart legend to be wrapped around the text?

Question: In the following chart, I modified the name of the Test n5. I want to have a text box that can wrap the text by doing a line break. Right now with the current code the box has the right maximal width but the text and the red box around it do not extend in height. What should I change?
<!DOCTYPE html>

  <meta charset="utf-8">
  <title>ZingSoft Demo</title>

Answer: full HTML will work only if you set 'html-mode': true, on the tooltip object as I've demonstrated in this demo. Please read here for more details on how to use HTML tooltips in ZingChart


dict is not ordering the content

Question: I installed the Anaconda 3 and I have a problem, I have read that dict is ordered automatically by the keys, but this is not happening in my installation
In the book that I am reading there is this example, wrote in a jupyter notebook:

this is what supposed to happen

I tested on https://colab.research.google.com and I got the output ordered, so is my anaconda installation the problem? is this just a visual representation?
Sorry for my english, thanks in advance
Comment: Python 3.7+ dicts are *insertion-ordered*, not sorted.
Comment: Where did you read that it sorts keys alphabetically?
Comment: dict objects *maintain insertion order*, they don't maintain a *sorted* order.
Comment: Dicts don't sort alphabetically...
Comment: Does this answer your question? [How to sort dictionary by key in numerical order Python](https://stackoverflow.com/questions/22264956/how-to-sort-dictionary-by-key-in-numerical-order-python)
Comment: @MichaelH: That doesn't sort either.
Comment: @alec it says in the book that I am reading( is in portuguese), the example is written in a jupyter notebook, I tested on https://colab.research.google.com and I got the output ordered
Answer: If you want your dictionary to be sorted alphabetically, you can do the following:
<code>>>> print(dict(sorted(dic_estados.items())))
{'AM': 'Amazonas', 'BA': 'Bahia', 'MG': 'Minas Gerais', 'PR': 'Parana', 'RN': 'Rio Grande do Norte'}
</code>
Comment: Thanks for the help.
Answer: Here is how you can achieve the desired output:
<code>dic_estados = {"PR":"Parana","MG":"Minas Gerais","BA":"Bahia","RN":"Rio Grande do Norte","AM":"Amazonas"}
sorted_dic = {k : v for k, v in sorted(dic_estados.items())}

print(sorted_dic)
#{'AM': 'Amazonas', 'BA': 'Bahia', 'MG': 'Minas Gerais', 'PR': 'Parana', 'RN': 'Rio Grande do Norte'}
</code>
Comment: Thanks for the help.
Answer: as everybody pointed out the dict are not ordered by the keys, the output in the jupyter notebook uses a function called pprint to show ordered by the keys, and for some reason on my installation this pprint is not working. is not an error
Thanks all

dict is not ordering the content


Pass variables from one preprocess function to another in .theme

Question: I am developing a theme in Drupal 8 and I have 2 twig files. One is page.html.twig and the other is status-messages.html.twig.
The problem is that status-messages has a variable named message_list (an array) but this variable is protected, so other files don't have access to it. I have tried to call this variable from theme_preprocess_page() like so  $variables['testvar'] = $variables['message_list']; with result of $variables['testvar'] = null.
In the next step I tried to copy the functionality of status-messages.html.twig from Drupal\Core\Render\Element\StatusMessages to mythemem.theme file, like so:
use Drupal\Core\Render\Element\StatusMessages;

function MYTHEME_preprocess_page(&$variables) {

  // Create Dynamic path for project root elements (in general files that are in our theme project)
  $variables['theme_path'] = base_path() . $variables['directory'];

  // this here gives us the array of renderMessages() but  '#message_list' has no data in it
  $variables['testvar'] = StatusMessages::renderMessages();
}

function mythemem_preprocess_status_messages(&$variables) {

  // this here dose gives the value but it can be accessed  only in status-messages.html.twig
  $variables['mymessages'][] = $variables['message_list'];

  // same as in mythemem_preprocess.page
  $variables['testvar'] = StatusMessages::renderMessages();
}

My aim here is to understand how to correctly use .theme files, how to pass protected variables from one preprocess function to another.

To be more clear I would like to know/understand how to get the same effect for my custom.html.twig file as it is here
To achieve this, I want to take $variables['message_list'] to process it as I want.
Comment: Please explain why you need to get this var, your final point. I'm sure there is some other logic to achieve what you need. This could be put it at a other place, simply altering it, or many other thing.
Comment: Messages are now in a block, no longer in the page template, see https://www.drupal.org/node/2446647. And you can't access the variables from the block in page directly. You render the messages by placing the block in a region and the region on the page.
Comment: @4k4 After some debugging with kint, i am sure that there is no such variable like 'message_list'
Comment: Are you trying to modify the message_list, duplicate it or move it at a specific  ?
Comment: @Cawet i am trying to duplicate this variable... but it is protected, is there a way to duplicate protected variable and pas it to the  higher level than it is right now (as i know preprocess.page function is executed before preprocess_status_messages. As level, status message is in a lower level). if i am wrong pleas correct me
Comment: @GeorgeChanturia, this is very difficult to predict what code runs in which order. One example, the message block puts only a placeholder in the page, this page gets cached, then you have a request with a cache hit for the page, no code for the page is run, but the placeholder gets replaced with the messages from the message list.
Comment: To be more clear i would like to know/understand ho to make same effect for my custom.html.twig file as it is  [link]https://api.drupal.org/api/drupal/core!modules!system!templates!status-messages.html.twig/8.2.x)

To achieve this, i must take thirst

$variables['message_list']
and than process it as i want...
Comment: This variable is only available in the status_messages template, so you would have to use this template to customize the messages.
Comment: OK i appreciate all your answers, i stil don't get why i cant duplicate some variables of drupal... I will stick to only drupal way stile so on...  Thanks
Comment: because it's like this!
More seriously it's because u must make change at one place and only one. If you don't, this could be very hard to remember where you did it, and it's why some var is protected
Answer: well after finking a lot about this problem, i realized that i don't wont to duplicate this variable at all.
lets start from beginning i wanted to copy the $variables['message_list'] from status-messages.html.twig to make my custom layout for different situation.
for example i wanted to have different layout/design of status message on  my custom login page and another layout/design for my custom register page. but now i understend that i can have this two in one place (in status-messages.html.twig) i can check if my URL is /login tean  with help of twig if condition i can render one layout and if  my URL is /register then i could have second layout! and in aver situation u would have default layout.
is that correct? how would you solve the problem if you ware in my place


Python: for loop - print on the same line

Question: I have a question about printing on the same line using for loop in Python 3. I searched for the answer but I couldn't find any relevant.
So, I have something like this:
def function(s):
    return s + 't'

item = input('Enter a sentence: ')

while item != '':
    split = item.split()
    for word in split:
        new_item = function(word)
        print(new_item)
    item = input('Enter a sentence: ')

When a user types in 'A short sentence', the function should do something with it and it should be printed on the same line.
Let's say that function adds 't' to the end of each word, so the output should be
At shortt sentencet

However, at the moment the output is:
At
shortt
sentencet

How can I print the result on the same line easily? Or should I make a new string so
new_string = ''
new_string = new_string + new_item

and it is iterated and at the end I print new_string?
Answer: Use end parameter in the print function
print(new_item, end=" ")

There is another way to do this, using comprehension and join.
print (" ".join([function(word) for word in split]))

Answer: The simplest solution is using a comma in your print statement:
>>> for i in range(5):
...   print i,
...
0 1 2 3 4

Note that there's no trailing newline; print without arguments after the loop would add it.
Comment: Note this only works in Python 2, and OP seems to be using Python 3
Answer: As print is a function in Python3, you can reduce your code to:
while item:
    split = item.split()
    print(*map(function, split), sep=' ')
    item = input('Enter a sentence: ')

Demo:
$ python3 so.py
Enter a sentence: a foo bar
at foot bart

Even better using iter and partial:
from functools import partial
f = partial(input, 'Enter a sentence: ')

for item in iter(f, ''):
    split = item.split()
    print(*map(function, split), sep=' ')

Demo:
$ python3 so.py
Enter a sentence: a foo bar
at foot bart
Enter a sentence: a b c
at bt ct
Enter a sentence:
$

Comment: I always had this doubt, can we use `iter` to stop when any of the items in a list is encountered?
Comment: @thefourtheye I don't think so, because we don't have access to `item` inside `iter`, but we can use `itertools.takewhile` for that.
Comment: Downvoter, could you please explain why?
Comment: @thefourtheye No, just `takewhile`: `for x in takewhile(lambda x: x not in my_list, (x() for x in repeat(f)))`, where `f` is `f = partial(input, 'Enter a sentence: ')`
Comment: @hcwhsa You mean we can combine `iter` and `takewhile` to do that?


Bootstrap Resposive Button Link In A Column

Question: Need to place 5 buttons in a download and finally i came with the following code.

<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" />

      Click Here For Music & Rhyme Section

      Click Here For Music & Rhyme Section

      Click Here For Music & Rhyme Section

      Click Here For Music & Rhyme Section

      Click Here For Music & Rhyme Section

But the issues here is that the text is collapsed in the button. We want the entire the text to visible in the button.
Comment: use btn-block class
Comment: it looks well in snippet
Comment: @Kenry Sanchez You can find the issuse when you click full page
Comment: @SaurabhMistry When i use btn-block full text is not shown
Answer: @rakeshkumar you just need to  add class="col-lg-2" to all your divs. it solves my problem.
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" />

      Click Here For Music & Rhyme Section

      Click Here For Music & Rhyme Section

      Click Here For Music & Rhyme Section

      Click Here For Music & Rhyme Section

      Click Here For Music & Rhyme Section


input data in python - get it to stick

Question: while True:
    print("")
    print("Welcome to Torrey's Restaurant")
    print("=====Select Option to Continue=====")
    print("1- Reserve a Table")
    print("2- Clear Reservation")
    print("3- Status of Tables")
    print("0- Exit")
    choice = int(input("Choice? "))

    count = 0
    tables = []
    name = " "

    for i in range(20):
        tables.append("AVAILABLE")

    if choice == 0:
        print("Thank you for using this program")
        break

    if choice == 1:
        table = int(input("Please select your table number(0-19): "))
        if tables[i] in tables:
            name = input("What is your name?: ")
            tables.insert(table, name)
        elif name != "AVAILABLE":
            print("The table you selected is unavailable")

How can I get the user input data to stick? When a user inputs their name, it inserts itself into the list, however, when the second user attempts to reserve the same table as the first, the original input data is overwritten, and the new data is stored, causing my elif to Never run.
any advice?
Comment: There is the issue of you defining the `tables` inside the while loop and then some issues with how your if-else is working. Check my answer for details and fixes.
Comment: Example., `if tables[i]` should be `if tables[table]` because its the variable `table` that holds the index of the table as input from user. Also, `elif name != "AVAILABLE":` should be either `elif tables[table] != "AVAILABLE"` or just `else` because `name` variable holds the input from user, but you want to check the table index instead.
Comment: Hope that helps.
Answer: Rather than using a list, use a dict. The keys are table numbers (1, 2, ..., n); the values are availability status (e.g. AVAILABLE, BUSY, ...). Checking whether a table is occupied or not then becomes an average-case constant time operation, and you can still access table states (and occupant names) by using that class' values and keys methods.
from enum import Enum, auto

NUMBER_OF_TABLES = 20

# If we know a thing can only be in one of N states,
# an enum.Enum object gives us an easy tool to enforce
# that constraint.
class TableStatus(Enum):
    AVAILABLE = auto()
    BUSY = auto()

# All tables start available. We're using a tuple
# here to pair the status-name pairs, but it'd be
# more readable if you used a NamedTuple-based class
# to store things in an easier-to-interpret format.
tables = {
    idx: (TableStatus.AVAILABLE, "")
    for idx in range(NUMBER_OF_TABLES)
}

# Prompt for a table.
check_table = input("What table do you want? ")

try:
    check_table = int(check_table)
    if tables[check_table][0] == TableStatus.AVAILABLE:
        name = input("What's your name? ")
        tables[check_table] = (TableStatus.BUSY, name)
    elif tables[check_table][0] == TableStatus.BUSY:
        # Handle the busy case.
        print(
            "Sorry, that table's already occupied. "
            f"The occupant's {tables[check_table][1]}."
        )
except KeyError:  # The customer asked for a table number we don't have.
    print("Sorry, that table doesn't exist.")

Comment: Thank you. I should have mentioned this. This is a school assignment, we have not yet covered functions. Instructor may not accept functions at this point. My apologies.
Comment: I see. In that case: in the context of your original problem, notice that `tables[i]` is *always* in `tables`. You don't want to check whether the element is in the list: you want to check whether the *value* at that element is `AVAILABLE`. If I ask you for table `x`, you should check whether `tables[x]` is available for me to sit in. In your current implementation, _every_ insert creates a _new_ table (not what you want).
Comment: I'd also note that you don't need to store occupancy status *and* occupant name. Given `n` tables, store `n` occupant names, with the caveat that an unoccupied table has no occupant name (e.g. `["John", None, "Jack", "Jill"]` has four tables, of which one is unoccupied. As your model becomes more complicated (e.g. a table can be busy even if it's unoccupied, like when a table is being bussed), you'd need to relax this assumption.
Comment: if I am understanding this correctly, instead of validating the name in my elif, I should check the status of tables[x]?
Comment: Instead of checking whether `tables[i] in tables`, you should check whether `tables[i] == "AVAILABLE"`. If it's available, the requestor occupies it (and you mark it as busy). If it's not available, the requestor should be prompted for a different table (or—if all tables are busy—asked to try again later). Note that you should *not* use `insert` to mark occupancy,  since that _adds a new table_ to the set of tables.
Comment: I hadn't noticed this, but you are correct, tables.insert adds a table at the end. is there any way to update a table without using a function? I originally had tables.append(name)
Answer: The issue is that you have tables defined inside the while loop and it resets to the list of "AVAILABLE" each time. Second issue is your if-else loop. I have updated both and marked the differences.
Try this modification -
tables = []                            #<---------- defined outside while loop

while True:
    print("")
    print("Welcome to Torrey's Restaurant")
    print("=====Select Option to Continue=====")
    print("1- Reserve a Table")
    print("2- Clear Reservation")
    print("3- Status of Tables")
    print("0- Exit")
    choice = int(input("Choice? "))

    count = 0
    name = " "

    for i in range(20):
        tables.append("AVAILABLE")

    if choice == 0:
        print("Thank you for using this program")
        break

    if choice == 1:
        table = int(input("Please select your table number(0-19): "))
        if tables[table] == 'AVAILABLE':           #<--------- check if table available
            name = input("What is your name?: ")
            tables.insert(table, name)             #<--------- insert name at position of table
        else:                                      #<--------- else error message
            print("The table you selected is unavailable")

Here is a demo result of when the same table #1 is selected by a new user.
Welcome to Torrey's Restaurant
=====Select Option to Continue=====
1- Reserve a Table
2- Clear Reservation
3- Status of Tables
0- Exit
Choice? 1
Please select your table number(0-19): 1
What is your name?: John

Welcome to Torrey's Restaurant
=====Select Option to Continue=====
1- Reserve a Table
2- Clear Reservation
3- Status of Tables
0- Exit
Choice? 1
Please select your table number(0-19): 1
The table you selected is unavailable

Welcome to Torrey's Restaurant
=====Select Option to Continue=====
1- Reserve a Table
2- Clear Reservation
3- Status of Tables
0- Exit
Choice? 0
Thank you for using this program

Once done, printing the tables variable should retain the values that you want -
print(tables)

['AVAILABLE', 'John', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE',
 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE',
 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE',
 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE',
 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE',
 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE',
 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE',
 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE',
 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE', 'AVAILABLE']

input data in python - get it to stick


String manipulation in Python

Question: I am trying to write a command but I do not want one long line that looks untidy. I am looking to add the strings together to be executed as on command. I have some code below which is part of an email function:
msg = MIMEText("The nightly build status was a SUCCESS\n\nBuild File: http://www.python.org\n\n Build Results File: http://10.51.54.57/sandboxes/", project, "\n")

This shows the one line, I am hoping for a better way to do this. I have tried the below code but it does not work.
msg = MIMEText("The nightly build status was a SUCCESS\n\nBuild File: ")
msg += MIMEText("http://www.python.org\n\n Build Results File: ")
msg += MIMEText("http://10.51.54.57/sandboxes/", project, "\n")

Thanks for any help.
I have tried the below code but get:
msg = MIMEText("""The nightly build status was a SUCCESS\n\n
   Build File: """,
   build_file, """
   \n\n
   Build Results File: """,
   build_file, """
   \n\n
   Sandbox Folder:""",
   sandbox, """
   \n\n
   Antibrick File: """,
   antibrick, "\n\n")

Now I get the message:
Traceback (most recent call last):
  File "test_email.py", line 45, in <module>
    if __name__ == '__main__': myObject = email_success()
  File "test_email.py", line 32, in email_success
    antibrick, "\n\n")
TypeError: __init__() takes at most 4 arguments (10 given)

Any ideas?
Thanks S.Mark, I tried this but when the email is sent it is not as a hyperlink but sent as:
The nightly build status was a SUCCESS

Build File: ('http://10.67.54.57/sandboxes/', '2010-01-05/new_sandbox', 'basebuild')

Build Results File: ('http://10.67.54.57/sandboxes/', '2010-01-05/new_sandbox', 'basebuild')

Sandbox Folder: ('http://10.67.54.57/sandboxes/', '2010-01-05/new_sandbox')

Antibrick File:

Comment: Please specify the error you've got.
Comment: Probably that `MIMEText` cannot be concatenated together like that.
Comment: the error says it all!! you have 10 arguments passed to MIMEText. construct your message properly.
Answer: Try:
msg = MIMEText("""The nightly build status was a SUCCESS

Build File:
http://www.python.org

Build Results File:
http://10.51.54.57/sandboxes/""", project, "\n")

If the additional space at the beginning of each line is a problem, remove them with a regexp (r'^\s+')
Answer: How about
msg = MIMEText(
"The nightly build status was a SUCCESS\n\n"
"Build File: http://www.python.org\n\n"
"Build Results File: http://10.51.54.57/sandboxes/"
, project
, "\n"
)

Or
msg = MIMEText("""The nightly build status was a SUCCESS

Build File: http://www.python.org

Build Results File: http://10.51.54.57/sandboxes/""", project, "\n")

Or
msg = MIMEText("The nightly build status was a SUCCESS\n\n"
"Build File: http://www.python.org\n\n"
"Build Results File: http://10.51.54.57/sandboxes/"
, project, "\n")

UPDATE: because OP added another question
msg=MIMEText("""The nightly build status was a SUCCESS\n\n
    Build File: %s
    \n\n
    Build Results File: %s
    \n\n
    Sandbox Folder: %s
    \n\n
    Antibrick File: """ % (build_file,build_file,sandbox),
    antibrick,
    "\n\n"
)

Answer: Why not
msg= MIMEText("The nightly build status was a SUCCESS\n\nBuild File: "+ \
  "http://www.python.org\n\n Bu..... ")

etc.
(I.e., use the line continuation backslash).
Also note  that the following each give you abcdef
s ="abc" "def"

s= "abc" \
     "def"

Also,
 s="""xyz
    wvu"""

gives you
'xyz\nwvu'

Answer: Python supports multi-line strings using triple quotes:
text = """The nightly build status was a SUCCESS\n\nBuild File:
http://www.python.org\n\n Build Results File:
http://10.51.54.57/sandboxes/"""
msg = MIMEText(text, project, "\n")

Answer: you can use triple quotes
>>> s="""The nightly build status was a SUCCESS

Build File: http://www.python.org

Build Results File: http://10.51.54.57/sandboxes/"""

>>> msg=MimeType(s,project,"\n")

Answer: Hmm, what exactly is the module are you using? I am guessing, that it is deprecated, because the modern interface is email (if I've guessed your intentions correctly). More specifically, to create a MIMEText object you use this class. The signature is
email.mime.text.MIMEText(_text[, _subtype[, _charset]])

String manipulation in Python


Why are classes necessary for the avoidance of paradoxes in the construction of infinite sets?

Question: I've been told that the notion of proper classes exists because you run into paradoxes if you try to make certain large infinite sets the elements of a set. Could someone give an example of such a paradox and what qualifies a class as a proper class?
Sorry I should probably have specified that I am speaking in the context of ZF
Comment: It is something of an exaggeration to say they are "necessary".  They are useful for avoiding contradictions, but other ways to avoid inconsistency have been proposed.
Comment: @Dan: You are not avoiding the paradoxes "without classes". You simply decide to use the term *class* to describe collections which do not form sets.
Comment: https://en.wikipedia.org/wiki/Russell%27s_paradox
Comment: There's nothing really that qualifies a class as proper class except that it fails to be a set
Comment: In what set theory are you referring to?
Comment: The paradoxes can be avoided without using classes by being more careful about the rules for constructing sets: You can't have unrestricted comprehension, but you can have, for example, subsets, power sets and Cartesian products of other sets presumed to exist.
Comment: @AsafKaragila I'm just saying you don't need to invent things called "classes" to avoid the paradoxes of set theory. Classes are not formalized in ZFC and it somehow manages to avoid the paradoxes.
Comment: @Dan: Then what do you do with $\{x\mid x\notin x\}$, what do you call this collection, which you most certainly define?
Comment: @AsafKaragila  I would that say it is doesn't exist. Recall that for any binary relation $R$, we have $\neg \exists x: \forall y: [y\space R \space x \iff \neg  y \space R \space y]$ Why would you need it to exist? What problem does it solve?
Answer: Classes are simply collections we can describe but they do not form a set. For example $\{x\mid x\notin x\}$ is something we can clearly define using the language of set theory, but it is not a set. (Why?)
This is Russell's paradox, and several other paradoxes have been found at the turn of the 20th century. Cantor's paradox and the Burali-Forti paradox, to name a few.
But just because something is not a set does not mean we cannot talk about it, and we cannot use it in proofs or statements. And this is exactly where classes come in.
Comment: It may be noteworthy that the meaning of classes depends on the set theory you are using. What Asaf calls classes in his answer, I like to call 'virtual classes' in the setting of ZFC. They are not objects of our language, but nonetheless we can handle them. Taking these virtual classes as second order objects, any ZFC model then yields a model of 'NBG minus global choice'. On the other hand, we can have models of NBG with classes that are not virtual classes, if we cut down or model to its canonical first order model of ZFC, i.e. they can't be defined over this smaller model.
Comment: Strictly speaking, the language of ZF does not contain the symbols "{" and "|", and so it's not clear what exactly you meant to convey by "can clearly define using the language of set theory". Of course, I know you are not speaking strictly, but perhaps it would be clearer to the asker if you clarified your intended meaning. Side question: if you believe "$\{ x : x \notin x \}$" describes a collection, do you also believe that we can describe collections of classes? If so, how are we allowed to reason about it and why? (Just briefly will do.) =)
Comment: @user21820: Maybe. They had two weeks, and more, to ask, though. I'm not putting that much time and effort (which frankly, nowadays have far far superior uses before this site) to expand on something which is clear in its essence, but the OP didn't bother to even leave one comment to indicate they have read the answer and didn't understand something. If the answer was clear, great; if not, then they can ask for specific clarifications...
Comment: @AsafKaragila: Yea sorry I didn't mean to suggest you had any obligation to update your answer for the asker. And if you have time, do you have a quick answer to my side question? =)
Comment: @user21820: About 2-classes? Well, of course we can. Because I can talk about $\{V\}$ (look!!! I just did! :-P). How can we do that formally? Well, it's easy if you have a set-theoretic meta-theory, since then it's just a set in the metaverse. But if you don't, there are probably easy type-theoretic approaches that can circumvent these issues, by defining what are 2-classes and so on syntactically, with the needed axioms we want for them.
Comment: @AsafKaragila: What stumps me is how we can conceive of iterating this climb up this hierarchy, but we seem to get stuck somewhere. For instance, if we use ordinals from ZFC to index these levels, we still do not have the union of all levels, despite being able to conceive of it. And if we restrict to syntactic methods, we can reach the $k$-th level for any computable ordinal $k$, but cannot reach the $ω_1^{CK}$-th level. The type theory COC underlying Coq has this kind of universe hierarchy but only to finite levels, and indeed syntactically.
Comment: @user21820: Of course we have the limit steps. An $\omega$-class is a collection that every object in that class is an $n$-class for some $n$. You're right that we fail below $\omega_1^{CK}$. But why is that an issue? If you want things to be computable, they end up being below that wretched ordinal...
Comment: @AsafKaragila: Nothing is wrong except that we seem to be unable to capture what we can describe. This is different from the phenomenon with the incompleteness theorem because there we are actually limited to computable ordinals when we iteratively add consistency axioms. In ZFC, we aren't limited in this way, and yet seem to be able to reason beyond the boundary of any such extension of ZFC via extending the hierarchy of universes. Put another way, I cannot see a level at which we can justify stopping, and yet we certainly cannot go on indefinitely either.
Comment: @user21820: This seems like a discussion to be had over drinks, rather than here in the comments. But let me sign off from it by saying that we can *never* capture via syntax what we feel about the semantics. These things will forever be with discrepancy. Roger Zelanzy hit that nail on the head in "Lord of Light", http://people.nas.nasa.gov/~pulliam/lord_of_light.html
Answer: ZF
In ZF, we cannot construct as an object a proper class, such as "$\{ x : x \notin x \}$" and "$\{ x : x = x \}$", which are technically not even valid to write down. The reason is that the axioms of ZF do not permit constructing them, since the axiom of specification only allows you to construct a set by specifying it as the elements of some previously constructed set which satisfy some formula. And the axioms of ZF were designed that way because if you could construct either of these two sets, you could then prove a contradiction using the axioms of ZF, which would make ZF useless.
$
\def\eq{\leftrightarrow}
$
But ZF, like any first-order theory, can be equipped with what is called full abbreviation power or definitorial expansion. Applied to ZF, it allows us to create a new predicate symbol $R$ defined by $R(x) \equiv x \notin x$, for example, and we can now prove (in ZF) that $R$ does not correspond to a set in the following precise sense:

$\neg \exists x\ ( \forall y\ ( R(y) \eq y \in x ) )$.

This is the technical meaning when people say that the Russel class does not exist. After all, "Russel class" is not a phrase that can be expressed in ZF. However, if you consider proper classes as simply the predicate symbols or their defining formulae as mentioned above, then you could say that ZF can handle classes in a limited way, namely they can be quantified over but not manipulated as objects.
Some common examples include the class of graphs, the class of models of a theory, the class of ordinals and the class of fields. One can say "For every graph $G$, either $G$ has an odd cycle or $G$ is bipartite." where "graph", "has an odd cycle" and "bipartite" have to be translated to formulae and cannot be represented by sets. Of course, "For every graph, ..." becomes "$\forall G\ ( graph(G) \to \cdots )$", and likewise for the existential quantifier. One can see that various useful meta-theorems for quantifiers hold, such as that "Not every graph is ..." is equivalent to "Some graph is not ...".
MK
However, if you want to actually talk about sets and classes separately, MK is essentially what you will get by adding to ZFC the ability to construct any arbitrary class (of sets) defined by a first-order formula over ZFC as an actual object. Formally, we can have two sorts called "set" and "class", so that we can state the axioms for ZFC for sets. (Or we can have just one sort but also a predicate for "sethood".) There are two main points of note:

A class only contains sets as members. So there is still no class of classes. But there is trivially a class of all sets, and classes form a boolean lattice under inclusion with a top and a bottom.
The usual axioms of ZFC apply to sets, not classes. So there is no such thing as the powerclass. However, given any class $C$ one can construct the class of all sets that are subsets of $C$.

MK is strictly stronger than ZFC, since MK proves the consistency of ZFC, but MK is easier to use than ZFC precisely because we can talk about classes directly.


For each loop through sheets with Selection.AutoFill

Question: This doesn't compile.
Option Explicit
Sub test3()
    Dim sht As Worksheet

    For Each sht In Sheets
        sht.Range("C1").FormulaArray = "=A1&B1"
        sht.Range("C1").Select
        sht.Selection.AutoFill Destination:=Range("C1:C3"), Type:=xlFillDefault
    Next sht
End Sub

The error

Compile error: Method or data member not found

is on:
sht.Selection.AutoFill Destination:=Range("C1:C3"), Type:=xlFillDefault

I also tried
Option Explicit
Sub test3()
    Dim sht As Worksheet

    For Each sht In Sheets
        sht.Range("C1").FormulaArray = "=A1&B1"
        sht.Range("C1").Select
        Selection.AutoFill Destination:=Range("C1:C3"), Type:=xlFillDefault
    Next sht
End Sub

This can work depending on the sheet I have active. For example if I have sheets(1) active it works on Sheets(1) but then on Sheets(2) I get an error:

Run-time error '1004': Select method of Range class failed
Comment: I forgot to mention... if I step through the code... and I manually select each tab as the code moves from one sheet to the next... it works fantastically on all sheets....
Comment: You can't select a sheet's cell without selecting the sheet first. So above the line `sht.Range("C1").Select` you'd put `sht.Select`
Comment: That said, it's recommended that you rewrite your function to not use `.Select` at all
Comment: Thank you!! Since everybody and their brother says "it's recommended that you rewrite your function to not use .Select at all" I have basically no experience using ".Select" which explains why I didn't know/remember that I needed to select the sheet. That said, what is a good alternative to using select in this situation?
Comment: [How to avoid using Select in Excel VBA](https://stackoverflow.com/a/10718179/2727437)
Answer: This will work for the activesheet (Using sheets array is referencing the active workbook).
For Each sht In Sheets
    sht.Range("C1").FormulaArray = "=A1&B1"
    sht.Range("C1").AutoFill Destination:=sht.Range("C1:C3"), Type:=xlFillDefault
Next sht

Comment: Thank you!! Much appreciated!!


How should I implement this interaction between a continuous and categorical predictor?

Question: I have a continuous outcome variable. I understand that if I have a binary predictor, and a continuous predictor, and an interaction, then the model looks like this:
$y_{i} = \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \beta_{3}x_{1}x_{2} + \varepsilon_{i}$
However, I'm thinking of making the binary predictor a categorical one instead, with three categories.

What will the model equation look like once I make that change? I understand that I'll need to have two dummy variables, but I can't conceptualize what the interaction will look like under that circumstance.
Does it make any difference whether I use 0,1 to code the binary predictor, or some other values like 1,2? This question also applies to the model with the categorical predictor, since in that case I'll have to decide how to code the dummy variables.
Prior to the interaction term being in my model I was encouraged to center the continuous predictor. Is centering (still) a good idea now that I have this interaction term?
Comment: (1) Write down the dummy coding scheme for each of the three levels. (2) As well as including terms for each of the two dummies, include terms where *each* is multiplied by the continuous variable. (3) Understand what's going on by re-writing the regression equation for each level of the categorical variable. (4) Investigate changing coding schemes/centring, paying attention to how the interpretation of coefficients changes, & whether the model as a whole changes (makes different predictions).
Answer:
Assuming your continuous variable is $x_1$ and we expand the binary $x_2$ to include $x_3$ then I suggest using:

$$
y_{i} = \beta_0 + \beta_1x_1 + \beta_2x_2 + \beta_3 x_3 + \beta_4x_1x_2 + \beta_5x_1x_3 + \varepsilon_i
$$
So if the continous variable interacts with the reference category, it will be included in the model by default. If there is an interaction with the second or third category then $\beta_4$ or $\beta_5$ will contain the difference from the reference category. Also, as you're suggesting, it wouldn't make any sense to put an interaction effect between $x_2$ and $x_3$.

The indicator variables should be coded with 0,1. In this case, if the indicator is not not true, the variable is zero and the corresponding $\beta$ drops out of the equation. This makes for much easier interpretation of the coefficients.

For example, if your category is the second category ($x_2=1$). Then for a given $x_1$, the interpretation of the category effect is $\beta_0 + \beta_2$. And given $x_2=1$, for every unit increase in $x_1$ there is a $\beta_1 + \beta_4$ increase in your response variable.

Here is a wonderful post about centering variables. As a short answer, if you were going to center your variable before, then adding interaction effects shouldn't change that. Actually, adding interaction effects is one reason that some people start centering variables to reduce collinearity.
Comment: (+1) Concise & clear explanation.  @user1205901: It might help you to rewrite the equation explicitly for each allowable combination of dummies to see the intercept & slopes for each level of the categorical variable. (Note that e.g. the second category is defined in full by $x_2=1$, $x_3=0$.)


When does my autoencoder start to overfit?

Question: I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has patterns but also varies a lot, hence, is partly stochastic in nature.
For understanding purposes, I trained a (complete) autoencoder with dimensions input = $500$, hidden = $500$, output = $500$ and sigmoid functions in the hidden and output layer. My training data has dimension $X\in[0,1]^{5000\times500}$ (500 variables, 5000 samples). I used $3$ algorithms, with learning rate $0.01$, mini batch size $64$, and pretty much the standard algo-parameters in Keras/TensorFlow:

standard stochastic gradient descent (SGD)
advanced/extended SGD with Nesterov momentum $0.9$ and learning rate decay $10^{-8}$
Adam optimizer ($\beta_1=0.9$, $\beta_2=0.999$, learning rate decay $0$)

The image below shows the corresponding error curves. In my case both keep decreasing (except for Adam) so I would say "keep training". On the other hand I know intuitively that I should not train so long because there must be some overfitting going on. So how do I know when to stop training, how would you interpret the result below? Would I be right, to just take Adam and use 250 epochs (even though it has a wide bias between training/validation sets)?
Comment: For an autoencoder, *approximating* the identity function is the entire point! Indeed, this is exactly the objective function. What you don't want is for the autoencoder to "learn nothing" and actually **be** the identity function. The tricks to prevent the AE from learning nothing are straightforward: use a bottleneck, regularize some or all parts of the encoder and decoder, learn from noisy inputs, etc. If your AE is obtaining 0 training error and you're learning through a bottleneck or similar, your model has learned how to reconstruct the input from limited information: a representation!
Comment: Your hidden layer should be smaller than your input. From there as long as your validation error doesn't start to increase you should be fine.
Comment: It seems like this question could be answered by (1) positing a definition of overfitting and (2) examining whether or not you observe phenomena which meet that definition. So, what's your definition of overfitting?
Comment: In the context of the autoencoder, it would seem to me that overfitting would be equivalent to the model approximating an identity function, i.e. if the reconstruction error approaches $0$ and it starts to simply pass inputs to the output layer without forming a latent space and learning feature patterns. However, I am not sure if the ADAM algorithm result above would already constitute this "approximation" due to its low error close to $0$, or if such a low error is still reasonable (my inputs take values in $[0,1]$).
Comment: Thanks a lot for this very helpful response. Am I correct in understanding that we want to construct a "constrained" AE (be it via bottleneck, noisy inputs, or regularization) and then we would like to reduce the reconstruction error as much as possible (ideally $0$)? So as long as we enforce a restriction, which forces the AE to form representations, we really want the error to go to ideally $0$?
Comment: Training error going to 0 is not related to overfitting. The reason we care about overfitting is because an overfit model has poor out-of-sample qualities. Poor out-of-sample generalization is measured by the validation data. So the question you should be asking is "Is my out-of-sample error good enough for my needs?"  *Elements of Statistical Learning* has a good discussion of overfitting and related concepts.
Answer: As validation tells about generalization of the algorithm. And from your graph, ADAM is working very good, with a biased response. But for sure, there is no overfitting sign in there.
For biasing check, you can try out k-fold method, and check the response of algorithm for each fold. Then you can find, whether this is irreducible error or something else.
Answer: @Sycorax answer is what comes the closest to the answer.
Usually, overfitting is described as the model training error going down while validation error goes up, which means the model is learning patterns that don't generalize beyond the training set.
In the case of an autoencoder, you're training the model to reproduce the input. At an extreme your model could simply be output = input and both validation and training loss would be 0.
But that's not what you want from an autoencoder for anomaly detection.
You want the model to learn an abstract representation of what the input should look like, so the model develop the ability to generate instances on it's own when presented with an input.
The distance or error between what the model "think it should look like" and the input is what tells us if the input is "abnormal" or not.
Best way to assess it would be to present the model with valid and invalid instances and monitor the reconstruction error. If it tends to be the same on both set your model is most probably learning the identity function and needs to be modified.


If $\frac{a_{n+1}}{a_n}$ converges to a real value L, then $\sqrt[n]{a_n}$ converges to $L$.

Question: To prove:
If {$a_n$} is a sequence of positive real numbers, and the sequence $\frac{a_{n+1}}{a_n}$ converges to a real value L, then $\sqrt[n]{a_n}$ converges to $L$.
Now I'm having trouble proving this. I tried the following:
Suppose that $\sqrt[n]{a_n}$ does NOT converge to value L. Then there is an $\epsilon$ such that there exists an $n_1>n_0$ for any $n_0$ such that:
$|\sqrt[n]{a_n}-L|\ge\epsilon$.
Then for that same value $n_1$ we still know:
$|\frac{a_{n+1}}{a_n}-L|<\epsilon$
It follows $|\sqrt[n]{a_n}-L|>|\frac{a_{n+1}}{a_n}-L|\ge\frac{a_{n+1}}{a_n}-L$
Now take a look at 2 different situations:
a) $\sqrt[n]{a_n}$ converges to a value greater than $L$: then
$\sqrt[n]{a_n}-L>\frac{a_{n+1}}{a_n}-L$ for certain $n_1$ that still satisfies the above conditions.
$\sqrt[n]{a_n}>\frac{a_{n+1}}{a_n}$
$a_n^{n+1}>a_{n+1}^n$
Now as $n$ approaches $\infty$, $a_{n+1}\rightarrow La_n$.
So
$a_n\lim\limits_{x \to \infty} a_n^{n} \ge \lim\limits_{x \to \infty} L^na_n^n$
$a_n \ge L^n$
I have no idea what I'm doing anymore.. Can someone please help :(
Also this was an exercise in a chapter concerning Cauchy's root and ratio test, but I don't see a way to employ that.
Answer: $\newcommand{\eps}{\varepsilon}$
Here is a more direct approach. Take $\eps>0$ and suppose $L-\eps<\frac{a_{n+1}}{a_n}<L+\eps$ for $n\geq N$. Then
$$a_n=a_1\frac{a_2}{a_1}\frac{a_3}{a_2}\dots\frac{a_{N}}{a_{N-1}}\frac{a_{N+1}}{a_N}\dots\frac{a_n}{a_{n-1}}.$$
We can bound all but a fixed number of terms. Show that after taking $n$th roots  the other terms become insignificant.
Comment: So $a_n<a_1\frac{a_1}{a_2}...(L+\epsilon)^{n-N)}$ and $\sqrt[n]{a_n}<\sqrt[n]{a_1\frac{a_1}{a_2}...(L+\epsilon)^{n-N)}}$ which gives by squeeze theorem $\lim\limits_{n \to \infty} \sqrt[n]{a_n} \le \lim\limits_{n \to \infty} \sqrt[n]{a_1\frac{a_1}{a_2}...(L+\epsilon)^{n-N}}$
Comment: @QuestionMaker Precisely. Can you find the limit you have on the RHS? Note that $N$ is fixed once $\varepsilon$ is fixed.
Comment: The RHS limit should be equal to $\lim\limits_{n \to \infty} \sqrt[n]{a_1\frac{a_1}{a_2}...\frac{a_N-1}{a_{N-2}}} \lim\limits_{n \to \infty}\sqrt[n]{(L+\epsilon)^{n-N}}$. Now the left limit clearly goes to 1, as all under the square root is just a constant.
Comment: The right limit equals the limit of $(L+\epsilon)^{1-\frac{N}{n}}$ Now again since $L+\epsilon$ is just a constant we look at the power, which goes to $1$ as $n$ goes to $\infty$. So the limit equals $L+e$. Now using the squeeze theorem we can do the same with a limit using $L-\epsilon$ and then we prove that the the $n$th root of $a_n$ converges to $L$. Correct?
Comment: @QuestionMaker Actually, since in the RHS limit we let n go to infinity, the limit is just $L+\varepsilon$. Other than that, all is correct.
Comment: Which step are you talking about?
Comment: @QuestionMaker Actually nvm, I've misread a part of your comment. It is correct.
Answer: Hint
$$\lim_{n\to+\infty}\frac{a_{n+1}}{a_n}=L$$
$$\implies \lim_{n\to+\infty}\ln(\frac{a_{n+1}}{a_n})=\ln(L)$$
$$\implies \lim_{n\to+\infty}\frac{\sum_{k=1}^n\ln(\frac{a_{k+1}}{a_k})}{n}=\ln(L)$$
using Cesaro sequence.
$$\implies \lim_{n\to+\infty}\frac{\ln(\frac{a_{n+1}}{a_1})}{n}=\ln(L)$$
Yes, you can conclude.
Comment: Thanks for your response but we've never covered a Cesaro sequence?
Answer: Let us first prove the following:

Lemma. If $(a_n)$ is convergent to $\ell$, then $\displaystyle\left(\frac{1}{n}\sum_{k=0}^{n-1}a_k\right)$ is also convergent to $\ell$.

Proof. Let $\varepsilon>0$, there exists $N_1\in\mathbb{N}$ such that: $$\forall n\geqslant N_1,|a_n-\ell|\leqslant\varepsilon.$$
Let $n\geqslant N_1$, using triangle inequality, one has: $$\left|\frac{1}{n}\sum_{k=0}^{n-1}a_k-\ell\right|=\left|\frac{1}{n}\sum_{k=0}^{n-1}(a_k-\ell)\right|\leqslant\frac{1}{n}\sum_{k=0}^{N_1-1}|a_k-\ell|+\frac{(n-N_1+1)}{n}\varepsilon.$$
Since $\displaystyle\frac{1}{n}\underbrace{\sum_{k=0}^{N_1-1}|a_k-\ell|}_{\textrm{constant}}\rightarrow 0$, there exists $N_2\in\mathbb{N}$ such that: $$\forall n\geqslant N_2,\frac{1}{n}\sum_{k=0}^{N_1-1}|a_k-\ell|\leqslant\varepsilon.$$
Furthermore, since $\displaystyle\frac{n-N_1+1}{n}\rightarrow 1$, there exists $N_3\in\mathbb{N}$ such that: $$\forall n\geqslant N_3,\frac{n-N_1+1}{n}\leqslant 2.$$
Finally, for all $n\geqslant\max(N_1,N_2,N_3)$, one has: $$\left|\frac{1}{n}\sum_{k=0}^{n-1}a_k-\ell\right|\leqslant 3\varepsilon.$$
Whence the result. $\Box$
From this lemma, one derives the following:

Proposition. If $(a_n)$ is such that $(a_{n+1}-a_n)$ converges to $\ell$, then $\displaystyle\left(\frac{a_n}{n}\right)$ converges to $\ell$.

Proof. Applying the lemma, one has that: $$\left(\frac{1}{n}\sum_{k=0}^{n-1}(a_{n+1}-a_n)\right)=\left(\frac{a_n}{n}-\frac{a_0}{n}\right)$$
converges to $\ell$. Whence the result, since $\displaystyle\left(\frac{a_0}{n}\right)$ converges to $0$ and $\displaystyle\left(\frac{a_n}{n}\right)=\left(\frac{a_n}{n}-\frac{a_0}{n}\right)+\left(\frac{a_0}{n}\right)$. $\Box$
Can you take it from here?


How can my WinForms app get the users MachineID or ProcessorID?

Question: I'm looking for a way to retrieve the user's MachineID or ProcessorID using a VB.net WinForms app? Any thoughts?
Comment: Bear in mind that the ProcessorID is not unique.
Comment: What would be unique that is not affiliated with the HD?
Comment: @redtail, practically nothing
Comment: As mentioned below, I'd suggest the MAC address from one of the network cards.
Answer: Another option is to get the MAC address of one of the network cards in your system.  This is a unique number  The following routine gets a mac address of the first network card, and then appends the bytes in the address into a unique ulong (sorry for the C# instead of VB.NET):

using System.Net.NetworkInformation;
NetworkInterface[] nics = NetworkInterface.GetAllNetworkInterfaces();
if (nics.Length != 0)
{
    byte[] addressBytes = nics[0].GetPhysicalAddress().GetAddressBytes();
    if (addressBytes.Length > 6)
        throw new ApplicationException("Unexpected length for MAC address");

    ulong address = 0;
    foreach (byte b in addressBytes)
    {
        address
Answer: You can access that information using WMI.
<code>string s = "SELECT SerialNumber FROM WIN32_BaseBoard";
System.Management.ManagementObjectSearcher oWMI = new
System.Management.ManagementObjectSearcher(s);
System.Management.ManagementObjectCollection oSerialNumbers = oWMI.Get();

foreach (System.Management.ManagementObject mo in oSerialNumbers)
{
Console.WriteLine("Serial Number: " +
mo.Properties["SerialNumber"].Value.ToString());
}
</code>
Referenced from here:
http://bytes.com/groups/net/411975-how-can-i-get-processor-id-my-computer
Another example to get processor id is here:
http://www.devasp.net/net/articles/display/149.html


Excel VBA to print the date and page number in the header and footer

Question: I would like that whenever my page prints on the top left corner for the date to be printed and the bottom left for the page number.
How can this be done using the head command in excel.
Head = "value of the date"
and
Foot = "Page number"
I know there are other ways to do this however I was asked to do this using specifically these functions.
Comment: I may have made a mistake editing out the vb script you had written.  Exactly how are you accessing/implementing this print?
Answer: With ActiveSheet.PageSetup
  .LeftHeader = Format(Now(), "dd/mm/yy")
  .LeftFooter = "&P"
End With

Comment: @FreeSoftwareServers wow, all these years and no one caught that, should be the "&", edited now.
Comment: see i'm not supposed to use leftheader
Comment: I have to use head and foot this is my main problem
Comment: @Tom, I know of no function called head and foot, so unless your company has a custom function, I think this is what they meant.  There is no 'Head' or 'Foot' in the Excel help.
Comment: here is the full back story, our company said the page was printed too slowly and contacted toshiba. Toshiba said to use head instead of .leftheader, I worked all day but head just does not work and I have to use it.
Comment: @tom, ok, I'll do some more research, in the meantime you might want to look at this post: http://stackoverflow.com/questions/230382/how-to-print-faster-in-excel-vba
Comment: @LanceRoberts 15K views too, I'm glad your still around to fix it so it's correct moving forward!
Answer: With ActiveSheet.PageSetup
  .LeftHeader = Format(Now(), "dddd, mmmm dd, yyyy")                                     ' Saturday, July 13, 2019
  .LeftFooter = "Page &P of &N"
End With

Date Formats --> https://powerspreadsheets.com/vba-format-date/#Format-Date-Using-VBA-Display-Several-Items-Of-The-Date
Update:
I used to use .LeftFooter = "Page " & ActiveSheet.Index & " of " & ActiveSheet.PageSetup.Pages.Count ' Page 1 of 2 but this gave inconsistent results and I found it better to use .LeftFooter = "Page &P of &N"
See here for more &Foo examples --> https://learn.microsoft.com/en-us/office/vba/excel/concepts/workbooks-and-worksheets/formatting-and-vba-codes-for-headers-and-footers


Is it possible to create a custom theme function?

Question: I want for example to create a mytheme_table function to override theme_table etc. (in template.php) but I do not want to use it everywhere in my project.
I need to replace the table with a div or to remove some hardcoded wrappers.
$output .= theme('table_custom', array('header' => $header, 'rows' =>
$rows, 'attributes' => array('id' => $table_id, 'class' =>
array('field-multiple-table'))));

Comment: what about just rewriting the individual field in a tpl or preprocess function and leave the tables alone ?
Answer: you don't have to use hooks in your theme functions.   you could create your own function called theme_table_custom()  and use that in your template files.
if for some reason you must use the hook, you could include the original drupal code in the function and include a check for a table id or something, which would then fire your custom code.
Comment: That is the point. I just duplicated theme_table (mytheme_table in my case) into mytheme_table_custom and it does not work. "mytheme_table" will apply everywhere.
Comment: I think I understand.  so you want to use the second solution I mentioned there. keep mytheme_table() and include a check for the table ID or some unique thing such as the node id.
Comment: Yes, thanks, I could do that. Actually I already did something similar for some custom fields..But it is more like plumbing. I was thinking if there is a more elegant solution. If I have 5 forms and 20 custom fields in a theme will be a mess. :)


Integration of a vector function through a cilinder

Question: I need to integrate $\vec F(x,y,z) = (2x,-3y,z)$ through a surface $\Sigma$.
$\Sigma$ is defined as the surface of the cylinder $x^2+y^2=1$ (so without the top and the bottom) that is confined between the plane $z=0$ and $z=x+2$.
I think I can't project it to my x-y plane since that didn't really work out when I tried it, so then I tried splitting it in 2 along the x-axis and then projecting it to the z-x plane.
Then I split the projection in 2 again so I get a bottom and a top half, resp. a rectangle and a triangle.
Now lets say I take the top half (triangle) of the part where y is positive.
My parametric representation would be:
$x=u$
$y=\sqrt{1-u^2}$
$z=v$
My boundaries would be:
$u=-1 \rightarrow u=+1$
$v=1 \rightarrow v=x+2$
Now I know everything to complete the formula:
$\iint_\Sigma\vec F.d\vec\sigma = \iint_K\vec F(\vec\phi(u,v)).(\frac{\partial\vec\phi}{\partial u}\times\frac{\partial\vec\phi}{\partial v})(u,v)du dv$
The cross product gives me $(\frac{-u}{\sqrt{1-u^2}};1;0)$ and so I get the following:
$\iint_K\frac{-2u^2}{\sqrt{1-u^2}}-3\sqrt{1-u^2} dudv$
I'm unsure how to proceed from here or if what I did was even correct...
My issues are:

I don't know how to solve the integral, I know I can split it up in the integral of an integral and then integrate one to $v$ and the other to $u$ (with the appropriate boundaries I defined earlier) but I'm unsure how to actually solve it.
How will I add these 4 parts up? Do I just add them or is there more to it?
Comment: Might I respectfully suggest that you try parametrizing $\Sigma$ in cylindrical coordinates $(\theta,z)$?
Comment: I figured it would be easier in cylindrical coordinates, but I'm still curious if what I've done so far (although it's more difficult) is correct?
Answer: @Joshua: Clumsy as your solution is :), it's almost right. There won't be four pieces, just two. You're integrating over a trapezoid in $uv$-space for each piece.
First, we have to decide how $\Sigma$ is oriented. Is the normal pointing outwards? Your cross-product should have been $\left(\dfrac{-u}{\sqrt{1-u^2}},-1,0\right)$, which points inward rather than outward. You need to change the sign (formally, switch $u$ and $v$ to get the right orientation).
Your integral should be [note your $v$ limits were wrong and there's a sign problem on the cross-product]
$$\int_{-1}^1 \int_0^{u+2} \left( \frac{2u^2}{\sqrt{1-u^2}} - 3\sqrt{1-u^2}\right) \,dv\,du.$$
Then switching the sign on $y$ will parametrize the other part. Again, be careful about the normal. Add the two.
By the way, your answer should be $-2\pi$. :)
Comment: So when adding two integrations I just need to make sure my normal should point in the "right" way? (i.e. either both outwards or both inwards?) Thanks for your help!
Comment: Yes, you need the orientation everywhere consistent with what is specified.


I'm setting up java on vs code and (1) problem keeps popping up even if the code is running and a small line below the p in public class

Question: enter image description here
I cant figure out what I am doing wrong?
Comment: Hover over it and check the warning message. You are missing the package name I suppose and welcome to Java.
Answer: The comment under the issue Warning: Classpath is incomplete. Only syntax errors will be reported. gave the explanation:

This warning is displayed when you open a single java file, because at
that point, no classpath can be configured thus no information on
types other than the default JDK can be provided. In that case, only
syntax errors are reported.

Open Command Palette and choose Java: Create Java Project, then code in your new project, there won't be this kind of errors.
Comment: @AhmedMahmoud. Please [accept and upvote](https://stackoverflow.com/questions/67806353/module-pyautogui-is-not-importing-correctly-despite-installing-successfully-in) my answer if it solves your question. Don't hesitate to let me know if you have any other problems.