Is there a object-change-tracking/versioning Java API out there?

Question: I know of at least two byte-code enhancer that modify the "object model" at runtime to allow transaction to be performed transparently. One of them is part of Versant VOD, which I use at work every day, and the other is part of Terracotta. There are probably quite a few others, for example in ORM, but Versant takes care of that at my company.
My question is, is there such an open-source API that can be used on it's own, independent of the product that it was designed for? You could say an "hackable" API. It should only track changes, not read access, which would slow down the code significantly. In other words, it should not require explicit read/write locking. This requires either access to all classes that perform changes, not just to the data model, or it requires to keep some form of "previous version" in memory to do a comparison.
The problem that I'm trying to solve is that I have "large" (32K to 256K) object graphs that are "serialized" in a (NoSQL) DB. They are long-lived and must be re-serialized regularly to have an "history" of the changes. But they are rather expensive to serialize, and most changes are minor.
I could serialize them fully each time and run a binary diff on the stream, but that sounds very CPU intensive. A better solution would be an API that modify write operations on the model to protocol the changes, so that after the initial "image" is stored, only the protocol need to be stored.
I've found some questions talking about Apache Commons Beanutils to compare objects, but that is not useful for in-place changes; I would need to make a complete clone of the model between every "business transaction".
To reiterate, I'm looking for an "in-memory" API, within the same JVM, which does not involve any external server application. APIs involving native code are OK if they are available on Win, Mac & Linux. The API does not have to be currently packaged independently; it just has to be possible to extract it from the "parent project" to form an independent API (the parent project license must allow this).
My object graphs will involve many large arrays, and so that needs to be supported efficiently.
The changes are not desired only for auditing, but so that they can be replayed, or undone. More precisely, with the deserialized initial graph, and a list of changes, I should arrive at an identical end graph. Also, starting with the end graph, it should be possible to go back to the initial graph by applying the changes in reverse. This uses exactly the same functionality, but requires the change protocol to keep the old value in addition to the new value.
The API license should be compatible with commercial use.
[EDIT] So far I did not get a useful answer, and it does not seem like what I want exists. That leaves me with only one option: make it happen. I'll post a link here as answer when I have a working implementation, as this is the next step in my project and I cannot go forward without it.
[EDIT] I found by accident this somewhat related question: Is there a Java library that can "diff" two Objects?
Comment: if you switch your serialization over to something like BSON (http://bsonspec.org/) it would be easier to isolate the "diff" to specific attributes (since the binary for the rest of the attributes would be the same)
Comment: probably would be. you might be able to "diff" the object with a prev ious copy of it while serializing but its "O(n)" either way unless you use something like ASM/javassist to dynamically subclass all of your domain classes and overwrite all the setters to mark "dirty" fields.

but even if you mark dirty fields you might miss out on things like:

someDomainObject.getSomeInternalClass().nonSTandardMethodThatCHangesState()

comparing the serialized form is the only "bulletproof" way (especially if you have no control over 3rd parties). using BSON just allows you to narrow down diffs to fields
Comment: Not totally sure what you're after.  But its sounds analogous to subversion.  If the objects were serialized to a file, that was checked into SVN, then the change history would be there.  Not very helpful if you're after programmatic access to the changes.  What do you need?
Comment: LOL! I see what you mean, but that doesn't fit at all to my use case. Firstly, I should all be "in-memory", not client-server based, and secondly, all RCS are inherently bad at binary data, which is what I am trying to version. No XML or JSON involved here. I'll update the question.
Comment: I don't know of any such API, but maybe you could think in order approaches. Instead of trying the transparent-API approach, it sounds that it is actually a requirement for your application. You may store the initial graph once, and then build a tree with the versions and the new data.
Comment: It does not have to be transparent. My problem is that I want to also use "third-party" beans, and I could not rely on them to write the "change tracking" code manually without errors. A code-generator (bean-interface to change-tracking-bean) could also be acceptable, depending on it's limitations.
Comment: @radai BSON looks alright, although it's missing int16, of which I will probably have large arrays. But how would I create a diff? Using a binary diff impl on the complete serialized graph? That sounds expensive.
Comment: What's wrong with Terracotta? Or for that matter with VmWare Gemfire (except for being enterprise)
Comment: @radai I also think that serializing the whole graph is safer that modifying the bytecode to keep track of "dirty fields". I think I got you wrong before; you meant creating a diff at the BSON API level, not using a generic binary diff, right? That sounds reasonable.
Comment: @StephanEggermont I never heard of Gemfire; going to have a look at it. Unfortunately, I liked Terracotta but it hates me. *Twice* I tried to replace an home-grown comm API with it: it works on the test servers perfectly, but fails as soon as you put it on the productive server. That meant cancelling two maintenance releases. There is no way I'm going to let it make a foul of me a third time. Also, it's no good with untrusted clients across the Internet; you have to thrust your clients, and they have to work synchronously over a low latency network due to the global locking.
Comment: Fail as in hardware differences or as in performance under load?
Comment: @StephanEggermont Fails as in the client JVM get stuck at start because the TC agent cannot connect to the TC server. Test servers and productive server where configured the same way, without a firewall in between. We never found out why. But this is off-topic and should not be discussed here.
Answer: Kryo v1 had a serializer that knows about the last data that was serialized
and only emits a delta. When reading, it knows about the last data received
and applies the delta. The delta is done on at the byte level. Here is the
serializer. Most of the work is done by this class. This could be used in a few useful ways, eg networking similar to Quake 3.
This was omitted in Kryo v2 because AFAIK it had never been put to use.
Also, it did not have an extensive set of tests. It could be ported though
and may do what you need, or serve as the basis for what you need.
Above also posted on JVM serializers mailing list.
Doing it at the object level would be a bit tricky. You could write something similar to FieldSerializer that walks two object graphs simultaneously This would be standalone code though, not a Kryo serializer. At each level you could call equals. Write a byte so that when you read you know if it was equals. If not equals, use Kryo to write the object. Equals would be called many times for the same object, especially for deeply nested objects.
Another way you might do it is to only do the above for scalars and strings, ie only values written by the Output class. The problem is walking two object graphs. To use Kryo I think you'd have to duplicate all the serializers to know about the other object graph.
Possibly you could use Kryo with your own Output that collects values in a list instead of writing them. Use this to "serialize" your old object graph. Now write another version of your own Output that takes this list and use it to serialize your new object graph. Each time a value is written, first check it with the next object in your list. If equals, write a 1. If not equals, write a 0 and then the value.
This could be made more space efficient by using the first Output twice, once on the old and once on the new graph. Now you have two lists of values. Use these to write a bitstring denoting which are equal. This saves space over writing a whole byte for each value, but has the overhead of an extra list. Finally, write all the values that are not equal.
To finish this idea, you need to be able to deserialize the data. You'll need an your own version of the Input class that takes a list of values from the old object graph. Your Input first reads the bitstring (or a byte per value). For a value that was equal, it returns the value from the list instead of reading from the data. If a value was not equal, it calls the super method to read from the data.
I'm not sure if this would be faster than doing it at the byte level. If I had to guess I'd say it probably would be faster. Storing all values in a list will be lots of boxing/unboxing, and this approach still assigns all fields even if they haven't changed. I doubt performance will be a problem either way, so I'd probably just choose the easier approach. Hard to say which that is tho... resurrect the delta stuff or write your own Output/Input classes.
If you feel like contributing back to Kryo, that would of course be great. :)
Comment: This might well be my best bet. :) But please write something about it here too, so SO readers don't have to follow the link.
Comment: Just looked at Kryo again. I had dismissed it before due to lack of facility for long-term storage (schema evolution), but V2 seems to solve that. :)
Comment: Updated with some more info. TaggedFieldSerializer is the next step up from FieldSerializer schema evolution-wise. After that there is CompatibleFieldSerializer.
Answer: Take a look at Content repository API for Java, it is used by Artifactory  to control maven dependencies. The Apache Jackrabbit is the reference implementation of this JSR (JSR-283 version 2)
Comment: This is very interesting. And it says this: "Another popular type is the versionable type. This makes the repository track a document's history and store copies of each version of the document." :)
Answer: As far as I know, GemFire is a Gemstone (now VmWare) enterprise product doing something similar to the Gemstone smalltalk OODB, but then for java. James Foster has created a series of videos on how Gemstone works. I found them very interesting. Gemstone has a free version to build small (Seaside web) systems with.
Answer: I do not know such API, but it cant be that complicated:

A better solution would be an API that modify write operations on the model to protocol the changes, so that after the initial "image" is stored, only the protocol need to be stored.

I would say you need only 2 components: Action and ActionProcessor
You only need to persist a list (protocol) of performed actions.
<code>interface ActionProcessor{
    void perform(Action action);
    void undoToDate(Date date);
}

iterface Action{
    Date getDate();
    void perform();
    void undo();
}
</code>
Comment: This would be the http://prevayler.org/ approach. While it would work, the amount of code required to *manually* implement the undo functions for every action would be tremendous, which would result in many bugs, which would result in failure to undo correctly. That would be the last solution, if all else fails. Secondly, once you bring out a new version of your code, your Action impl would change, causing you to loose reproducibility, and "undoability", because you would have recorded the Action ID and it's parameters, instead of the real change.


Rollup unexpectedly generates a file for reactive state

Question: I'm kinda of new to rollup, and been having several issues with it over the last couple of weeks.
I'm currently using rollup to build Lit components to production.
And for some unknown reason, when it generates a file which contains, what I believe is the reactive state for every component - query-assigned-elements-78bcb1b1.js.
Every component that has reactive state then imports this file.
I need to do not output this file, and keep the state in the actual files, since the components needs to be used alone, without any dependencies and imports.
rollup-config.mjs:
...

export default (args) => {
  const isProd = args.prod ? true : false;
  const isDev = args.dev ? true : false;

  console.log(
    `Building ${
      isProd ? 'production' : isDev ? 'development' : 'local'
    } build..`
  );
  const distProd = 'dist';
  const distDev = 'dev/dist';
  const distLocal = 'dev/components';
  let envPath = isProd ? distProd : isDev ? distDev : distLocal;
  let build = [
    {
      input: ['src/**/[!_]*.ts'],
      output: {
        dir: envPath,
      },
      plugins: [
        resolve(),
        multiInput.default({
          relative: 'src/',
        }),
        commonjs(),
        typescript({
          compilerOptions: {
            outDir: envPath,
            sourceMap: false,
            declaration: false,
            declarationMap: false,
            inlineSources: false,
          },
        }),
        template(),
        terser({
          ecma: 2017,
          module: true,
          warnings: true,
        }),
        copy({
          patterns: '**/assets/*.{svg,png,jpg,gif,webp}',
          rootDir: './src',
        }),
      ],
      preserveEntrySignatures: 'strict',
    },
  ];
  return build;
};

Every component then has import{r as t,_ as e,e as o,n as i,s as r,$ as a}from"../query-assigned-elements-78bcb1b1.js";. Any suggestions would be greatly appreciated.
Answer: The issue appears to occur when the input of the rollup config is too broad:
input: ['src/**/[!_]*.ts']

If each individual lit element is placed in its own separate folder, rollup needs to be applied to each lit element individually. If rollup input is too broad (in the sense that it takes all elements as an input), it will see that the given dependency (query-assigned-elements) is used across all of the given elements, and instead of embedding it individually in each file, it creates a separate external file (query-assigned-elements-78bcb1b1.js) and references is in import.
Changing input to something like
 input: ['./src/<element-folder>/[!_]*.ts']

and running rollup for each element separately fixes the issue.


How to prove two definitions of antisymmetric are equivalent

Question: The first definition of antisymmetric of a relation is $(a,b) \in R$ and $(b,a) \in R \implies a=b$. There is another saying if $a \neq b$ and $(a,b)\in R$, then $(b,a) \not\in R$. I could intuitively understand this or even by truth table, but I want to show how to transform one into another. I had gone with contrapositive and logical equivalence, neither work out. Here is what I got so far:
$a\neq b \implies (a,b)\not\in R $ or $(b,a)\not\in R$ I cannot go further with this one.
Could someone show me how to do it, please?
Answer: Let $p$ be $(a,b)\in R$, $q$ be $(b,a)\in R$, and $r$ be $a=b$.
$(p\wedge q)\to r$ is equivalent to $p\to( q\to r)$ is equivalent to $p\to( \neg r\to \neg q)$ is equivalent to $(p\wedge \neg r)\to \neg q$.
Comment: Nice !! Thank you so much! Exactly what I am looking for
Comment: @LawrenceGuo The best way to say thanks here is upvoting and accepting the answer
Comment: @Matthew Daly I don't see why does $ (p\wedge q) \to r $ is equivalent to $p\to( q\to r)$


Listview does not scroll

Question: The listview I have used in the application I am making with flutter cannot be scrolled. I tried everything but couldn't find the solution.There is no other scroll in the code tree.I also added pysics but it still didn't work. How can i fix this? I'm leaving all the code below. Thank you from now.
Here is my code:
Container(
      height: ScreenSize.getScreenHeight(context),
      width: ScreenSize.getScreenWidth(context),
      decoration: Background.setBackground,
      child: Column(
        children: [
          const SizedBox(
            height: 30,
          ),
          const Text(
            "Oylama Oluştur",
            style: TextStyle(
              fontSize: 30,
              fontWeight: FontWeight.w600,
            ),
          ),
          const SizedBox(
            height: 15,
          ),
          const Divider(
            color: Colors.black,
            indent: 25,
            endIndent: 25,
          ),
          const SizedBox(
            height: 50,
          ),
          Padding(
            padding: const EdgeInsets.all(8.0),
            child: Container(
              //height: ScreenSize.getScreenHeight(context)/2-20,
              width: ScreenSize.getScreenWidth(context) / 1.2,
              decoration: const BoxDecoration(
                borderRadius: BorderRadius.all(const Radius.circular(10)),
              ),
              child: TextFormField(
                onChanged: null,
                decoration: const InputDecoration(
                  hintText: '   Oylama başlığı giriniz',
                  hintStyle: TextStyle(
                    color: Colors.grey,
                    fontSize: 16.0,
                  ),
                  //labelText: "Name" ,
                  labelStyle: TextStyle(
                    color: Colors.black,
                  ),
                  border: OutlineInputBorder(
                    borderRadius: BorderRadius.all(Radius.circular(10)),
                    borderSide: BorderSide(color: Colors.black, width: 2),
                  ),
                  focusedBorder: OutlineInputBorder(
                    borderRadius: BorderRadius.all(Radius.circular(10)),
                    borderSide: BorderSide(color: Colors.black, width: 2),
                  ),
                ),
              ),
            ),
          ),
          Padding(
            padding: EdgeInsets.all(8),
            child: SizedBox(
              width: ScreenSize.getScreenWidth(context) / 1.2,
              child: Row(
                mainAxisAlignment: MainAxisAlignment.spaceBetween,
                children: [
                  Container(
                    //height: ScreenSize.getScreenHeight(context)/2-20,
                    width: ScreenSize.getScreenWidth(context) / 2,
                    decoration: const BoxDecoration(
                      borderRadius:
                          BorderRadius.all(const Radius.circular(10)),
                    ),
                    child: TextFormField(
                      onChanged: null,
                      decoration: const InputDecoration(
                        hintText: '  Ekle',
                        hintStyle: TextStyle(
                          color: Colors.grey,
                          fontSize: 16.0,
                        ),
                        //labelText: "Name" ,
                        labelStyle: TextStyle(
                          color: Colors.black,
                        ),
                        border: OutlineInputBorder(
                          borderRadius:
                              BorderRadius.all(Radius.circular(10)),
                          borderSide:
                              BorderSide(color: Colors.black, width: 2),
                        ),
                        focusedBorder: OutlineInputBorder(
                          borderRadius:
                              BorderRadius.all(Radius.circular(10)),
                          borderSide:
                              BorderSide(color: Colors.black, width: 2),
                        ),
                      ),
                    ),
                  ),
                  SizedBox(
                    width: ScreenSize.getScreenWidth(context) / 4,
                    height: 60,
                    child: ElevatedButton(
                      onPressed: () {},
                      style: ElevatedButton.styleFrom(
                          primary:
                              const Color.fromARGB(222, 167, 162, 162),
                          shape: RoundedRectangleBorder(
                              borderRadius: BorderRadius.circular(10))),
                      child: const Text("Ekle",
                          textAlign: TextAlign.center,
                          style: TextStyle(
                            fontSize: 20,
                            fontWeight: FontWeight.bold,
                          )),
                    ),
                  ),
                ],
              ),
            ),
          ),
          ListView.builder(
            itemCount: entries.length,
            scrollDirection: Axis.vertical,
            shrinkWrap: true,

            itemBuilder: (BuildContext context, int index) {
              return ListTile(
                title: Text('Item ${entries[index]}'),
              );
            },
          )
          //button(context),
        ],
      )),

Here is the list:
final List<String> entries = <String>['A', 'B', 'C', 'B', 'C', 'B', 'C', 'B', 'C', 'B', 'C', 'B', 'C', 'C', 'B', 'C', 'C', 'B', 'C'];

Answer: Wrap your ListView with Expanded to get all the space available in column:
Expanded(
    child: ListView.builder(
      itemCount: entries.length,
      scrollDirection: Axis.vertical,
      shrinkWrap: true,
      itemBuilder: (BuildContext context, int index) {
        return ListTile(
          title: Text('Item ${entries[index]}'),
        );
      },
    ),
  )

Comment: if this is not working could you include an image of the result on your phone? @Mert
Comment: Thanks for the answer but there is no another scroll
Answer: In your code, your ListView is within a Column which can't scroll. So wrap your ListView in an Expanded widget:
  Expanded(
            child: ListView.builder(
              shrinkWrap: true,

              itemCount: entries.length,
              scrollDirection: Axis.vertical,
              itemBuilder: (BuildContext context, int index) {
                return ListTile(
                  title: Text('Item ${entries[index]}'),
                );
              },
            ),
          )

Listview does not scroll


Unable to push a npm artifact to Gitlab

Question: I'm trying to push a npm artifact and have it stored in Gitlab in its "Package Repositories" but am having trouble. Have tried a few different ways, with all giving the same error.
The below mostly comes from How do I publish a private npm package with gitlab ci?, and this is my .gitlab-ci.yml file
image: node:latest

stages:
  - deploy

deploy:
  stage: deploy
  script:
    - npm install
    - npm config set //gitlab.com//api/v4/projects/700/packages/npm/:_authToken ${CI_JOB_TOKEN}
    - npm publish

But then gets an error of need auth You need to authorize this machine using npm adduser
I then run npm adduser, but that gets an error of Username: npm ERR! cb() never called!
And here is part of the package.json:
  "publishConfig": {
    "@foo:registry":"https://gitlab.com/api/v4/projects/700/packages/npm"
  },

Another method I try is doing not doing the npm config set..., but instead doing echo "//gitlab.com//api/v4/projects/700/packages/npm/:_authToken=${CI_JOB_TOKEN}">.npmrc, but that gets the same errors.
And yet another method I try is not having npm config nor echo //..., but having this in the .npmrc:
@foo:registry=https://gitlab.com/api/v4/projects/700/packages/npm
//gitlab.com/api/v4/projects/700/packages/:_authToken=${CI_JOB_TOKEN}

But even that has the same errors.
Comment: the `cb() never called!` error is a known bug. try reinstalling npm.
Comment: In your example `.npmrc` file, you are missing `/npm/` before `:_authToken`. It should be `//gitlab.com/api/v4/projects/700/packages/npm/:_authToken=${CI_JOB_TOKEN}`
Answer: Go through the troubleshooting steps that Gitlab provides.
https://docs.gitlab.com/ee/user/packages/npm_registry/index.html#publish-an-npm-package-by-using-cicd
Comment: Please don't leave "RTFM" answers, even if you're legitimately pointing someone towards the docs. This type of answer is so minimally helpful that it's best just left as a comment.
Comment: Which of these steps are for the `npm adduser` issue outlined in the question?
Comment: @ParkerD You don't actually need `npm adduser` as the message says. If you're using an auth token, you can use that directly.
Comment: Agreed, the GitLab docs didn't provide troubleshooting steps for this. I submitted a ticket after I figured it out. https://stackoverflow.com/a/77339968/1042151

Unable to push a npm artifact to Gitlab


eSocial: Assinatura Inválida .NET Framework

Question: Assinatura Inválida
Boa tarde, estou com um problema na transmissão das informações para os webservices do eSocial.
O processo que eu faço é o seguinte: os XMLs dos eventos são gerados no servidor, devolvidos para uma aplicação que roda na máquina do cliente e é responsável por assinar os eventos e realizar a transmissão ao eSocial.
A transmissão do lote é feita e recebida com sucesso, porém ao consultar as informações pelo webservice de consulta ( pelo recibo de entrega devolvido no envio ) é retornado um erro de código 142 e descrição Assinatura do evento inválida. Ações Sugeridas: Verificar se houve alteração do evento após a assinatura. Verificar a validade da assinatura.
Vou colocar abaixo o arquivo enviado e o método responsável por assinar.
(Os dados do certificados foram omitidos)
<eSocial xmlns="http://www.esocial.gov.br/schema/lote/eventos/envio/v1_1_1">
  <envioLoteEventos grupo="2">
    <ideEmpregador>
      <tpInsc>1</tpInsc>
      <nrInsc>95784204000177</nrInsc>
    </ideEmpregador>
    <ideTransmissor>
      <tpInsc>1</tpInsc>
      <nrInsc>05964161000119</nrInsc>
    </ideTransmissor>
    <eventos>
      <evento Id="ID1957842040001772018121314262900000">
        <eSocial xmlns="http://www.esocial.gov.br/schema/evt/evtToxic/v02_05_00">
          <evtToxic Id="ID1957842040001772018120609352600000">
            <ideEvento>
              <indRetif>0</indRetif>
              <tpAmb>2</tpAmb>
              <procEmi>1</procEmi>
              <verProc>01.00.00</verProc>
            </ideEvento>
            <ideEmpregador>
              <tpInsc>1</tpInsc>
              <nrInsc>95784204000177</nrInsc>
            </ideEmpregador>
            <ideVinculo>
              <cpfTrab>56641686094</cpfTrab>
              <nisTrab>12025177234</nisTrab>
              <matricula>621</matricula>
            </ideVinculo>
            <toxicologico>
              <dtExame>2018-12-06</dtExame>
              <indRecusa>S</indRecusa>
            </toxicologico>
          </evtToxic>
          <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
            <SignedInfo>
              <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" />
              <SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" />
              <Reference URI="">
                <Transforms>
                  <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
                  <Transform Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" />
                </Transforms>
                <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256" />
                <DigestValue>...</DigestValue>
              </Reference>
            </SignedInfo>
            <SignatureValue>...</SignatureValue>
            <KeyInfo>
              <X509Data>
                <X509Certificate>...</X509Certificate>
              </X509Data>
            </KeyInfo>
          </Signature>
        </eSocial>
      </evento>
    </eventos>
  </envioLoteEventos>
</eSocial>

Aqui o código responsável pela criação do lote e a seguir da assinatura do evento.
var cl = new WSEsocial.Envio.ServicoEnviarLoteEventosClient();

cl.ClientCredentials.ClientCertificate.SetCertificate(cert.SubjectName.Name, System.Security.Cryptography.X509Certificates.StoreLocation.CurrentUser, System.Security.Cryptography.X509Certificates.StoreName.My);

XmlElement _getXmlElement(string xml)
{
  var d = new XmlDocument();
  d.LoadXml(_serviceSign.Sign(cert, xml).OuterXml);
  return d.DocumentElement;
}

var lote = new eSocial()
{
  envioLoteEventos = new eSocialEnvioLoteEventos()
  {
    grupo = (sbyte)model.GrupoEvento,
    ideEmpregador = new TIdeEmpregador()
    {
      nrInsc = model.UnidadeInsc,
      tpInsc = (sbyte)model.TipoInsc
    },
    ideTransmissor = new TIdeTransmissor()
    {
      nrInsc = cnpj,
      tpInsc = (sbyte)1
    },
    eventos = new eSocialEnvioLoteEventosEventos()
    {
      evento = model.Eventos.Select(x => new TArquivoEsocial
      {
        Id = x.Id,
        Any = _getXmlElement(x.XmlString)
      }).ToList()
    }
  }
};

public XmlDocument Sign(System.Security.Cryptography.X509Certificates.X509Certificate2 cert, string xmlString)
    {
      var doc = new XmlDocument();
      doc.PreserveWhitespace = false;
      doc.LoadXml(xmlString);

      Reference referenc = new Reference
      {
        Uri = string.Empty,
        DigestMethod = "http://www.w3.org/2001/04/xmlenc#sha256",
      };
      referenc.AddTransform(new XmlDsigEnvelopedSignatureTransform());
      referenc.AddTransform(new XmlDsigC14NTransform());
      referenc.DigestMethod = XmlSignatureExtensions.SHA256DIGEST;

      var kInfo = new KeyInfo();
      kInfo.AddClause(new KeyInfoX509Data(cert));

      var privKey   = (RSACryptoServiceProvider)cert.PrivateKey;
      var enhCsp    = new RSACryptoServiceProvider().CspKeyContainerInfo;

      var cspparams = new CspParameters(enhCsp.ProviderType, enhCsp.ProviderName, privKey.CspKeyContainerInfo.KeyContainerName);
      privKey       = new RSACryptoServiceProvider(cspparams);

      var signDoc = new SignedXml(doc)
      {
        KeyInfo = kInfo,
        SigningKey = privKey
      };
      signDoc.SignedInfo.SignatureMethod = "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256";
      signDoc.SignedInfo.CanonicalizationMethod = SignedXml.XmlDsigC14NTransformUrl;
      signDoc.AddReference(referenc);

      signDoc.ComputeSignature();

      doc.DocumentElement.AppendChild(doc.ImportNode(signDoc.GetXml(), true));

      return doc;
    }

Agradeço muito se alguém conseguir me dar uma luz sobre.
Obrigado
Answer: Eu chuto que o problema deve estar na forma como você está obtendo a chave privada do certificado, para assinar o XML. Veja que você está pegando as propriedades ProviderType e ProviderName do objeto RSACryptoServiceProvider recém-criado, e não da chave privada do certificado:
var privKey   = (RSACryptoServiceProvider)cert.PrivateKey;
var enhCsp    = new RSACryptoServiceProvider().CspKeyContainerInfo;

var cspparams = new CspParameters(enhCsp.ProviderType,
                                  enhCsp.ProviderName,
                                  privKey.CspKeyContainerInfo.KeyContainerName);
privKey       = new RSACryptoServiceProvider(cspparams);

Esse pode até não ser o seu problema atual, mas pode ser um problema no futuro, porque quando você instancia um novo objeto da classe RSACryptoServiceProvider, a chave padrão é usada (documentação), e a chave padrão pode não ser a chave que você quer usar para a assinatura.
Veja essa minha outra resposta no Stack Overflow sobre o assunto:

E-Social. Assinatura do evento inválida - Stack Overflow em Português

Lá há um exemplo de rotina usada para assinar o XML e também algumas dicas de motivos que podem causar esse erro de assinatura inválida.
Veja também essa outra resposta, mais recente, que é sobre EFD-Reinf, mas, é muito similar ao eSocial (a diferença é que no EFD-Reinf o atributo reference.Uri deve ter conteúdo e no eSocial deve ser vazio):

EFD-Reinf: Erro Assinatura - Um Contexto XmlDocument é necessário para transformações envelopadas

Outra coisa, o evento S-2221 ("Exame Toxicológico do Motorista Profissional") que você está tentando enviar no exemplo, faz parte dos eventos de Saúde e Segurança do Trabalhador – SST, que só passarão a ser enviados a partir de julho/2019 para as empresas do Grupo 1 (cronograma), e esse evento especificamente foi incluído na versão 2.5 dos leiautes do eSocial, que foi implantada ontem no Ambiente de Produção Restrita, mas só será implantada no dia 21/01/2019 no Ambiente de Produção (notícia).
Se você está começando agora no eSocial, o melhor é sempre pegar o evento S-1000 ("Informações do Empregador/Contribuinte/Órgão Público") primeiro, que é o início de tudo.
Comment: Obrigado pelo retorno, estou ciente desta questão do SST, pelo fato de nosso software ser responsável apenas por estes eventos.
Comment: Vou verificar os dois link que encaminhou, muito obrigado
Comment: Entendi @VicenzoMartinelli. Mas de qualquer forma comece testando com um S-1000, porque eu creio que o S-2221 não esteja nem disponível para testar ainda. Depois me diga se conseguiu resolver o erro de assinatura.
Comment: Já efetuei o teste Pedro, realmente o problema estava na forma que estava pegando as informações da chave privada, mudei um pouco o código e funcionou ok.
Comment: Que bom @VicenzoMartinelli! Se puder depois marca a resposta como aceita, então! ;-)  Veja: [Alguém me respondeu](https://pt.stackoverflow.com/help/someone-answers) e [Por que votar?](https://pt.stackoverflow.com/help/why-vote).
Answer: Esse problema estava ocorrendo comigo na versão anterior ao .NET 4.6.2!
Estranhamente, para o certificado A1 funciona perfeitamente.
Mas para o certificado A3, o mesmo código dá erro na assinatura.
Seguindo o código do Pedro Gaspar nesse post, funcionou corretamente:

E-Social. Assinatura do evento inválida - Stack Overflow em Português


Translação, rotação e escala de um retangulo

Question: Imaginem os seguinte retângulo A e losango B:

O retângulo A tem as dimensões 4200 por 3000.
O losango B tem as dimensões máxima de 167 por 167, seus lados tem 98 por 98.
O retângulo A é equivalente ao losango B, por isso, gostaria de saber (uma formula genérica) como passar um ponto qualquer de A, ex: P(500, 400) para B.
Levando em consideração a translação, rotação e escala entre A e B. Se possível usando JavaSript na formula;
Para a escala estou usando uma regra de 3 simples.
Eu achei essa formula para a rotação:
<code>function rotacionar(x, y, radianos) {
    var graus  = radianos * Math.PI / 180;
    var seno   = Math.sin(graus);
    var coseno = Math.cos(graus);

    x += 98;

    return {
        x: parseInt(x * coseno + y * seno),
        y: parseInt(y * coseno - x * seno)
    }
}
</code>
Só que não sei mais como juntar tudo e dar o ponto exato.
Obs.: É pra usar no Phaser.IO para que o usuário ande pelo mapa (A = pixels e B = locais onde ele pode andar)
Comment: Acho que essa pergunta nao faz parte do [escopo](/help/on-topic) do site.
Comment: Exato. Do jeito que a pergunta está, ela não faz parte do escopo porque é essencialmente um problema de matemática. Procure ao menos fornecer exemplo de código Javascript que vc já fez e tentou, que aí ela talvez entre mais facilmente no escopo do site.
Comment: Assim está melhor?
Comment: A tag correta adicional para sua pergunta seria computação gráfica.
Comment: Não achei essa TAG para adicionar
Answer: Se eu me lembro bem dos livros de computação gráfica que li você tem que transladar o centro do retângulo para a origem (0,0) e tudo que estiver ligado a ele, rotacionar ao angulo desejado e transladar para o ponto antigo.
Se você aplicar a formula de rotação da forma como esta, o retângulo vai rotacionar em  volta da origem e não do seu centro como desejas.
Então para rotacionar algo deve-se transladar o eixo de rotação para a origem, rotacionar e voltar ao local.
A fórmula fica dessa forma:
xf = (xo - xr) * cos (@) - (yo - yr) * sin (@) + xr
yf = (yo - yr) * cos (@) + (xo - xr) * sin (@) + yr
Onde:

(xo,yo) = Ponto que você deseja rotacionar
(xr,yr) = Ponto em que você vai rotacionar o ponto acima (no seu
caso o centro  do retangulo)
(xf,yf) = O novo local do    ponto rotacionado
@ = Angulo de rotação

Este site tem o que precisa
Mas fica minha sugestão para não perder seu tempo programando na mão toda esta matemática porque poderás acabar por encontrar os Quaternions. Você poderia usar o excelente Box2D. Eu uso ele em C++ e funciona perfeitamente.
Comment: Como se chama isso na matemática? Se você lembrar posso correr atras de como fazer.
Comment: Ficamos negativos mas o que importa é o trabalho fluir. Valeu por ter me feito lembrar deste assunto.
Comment: Como vc usaria o Quaternions? nunca usei ele
Comment: Nem eu. Sempre usei alguma 3D/2D engine para evitar isto.
Comment: Cara, essa formula ai (mesmo eu achando q está certa) não deu o resulta q eu esperava, Ex.: se pegar no retangulo A o ponto (0,0) e tentar passar para um ponto do losango B, em nenhum momento tenho algo parecido como um ponto (98,0) ou (0,98), sabe pq?
Comment: A única falha e já corrigida é que ela não traz de volta ao ponto de origem. Então é só somar o ponto de rotação novamente ao resultado. Fiz um notebook explicando e mostrando isto:  https://github.com/kernill/Notebooks/blob/master/Rotação.ipynb


display a node id within a specific content type

Question: I'm looking for a way to display the node id somewhere on all displayed instances of a given content type.  It doesn't need to happen in any particular way, but it would sure help my users to be able to see the node id, as we use it to track physical objects.
I would imagine that there are a couple of ways to do this, so if someone could outline the options that would also help.
I've tried something like
<code><?php
$node_id = arg(1);
echo($node_id);
?>
</code>
To no avail, as I can only put this in each individual node instead of all nodes for a given content type.
Answer: If you're trying to do this for a certain content type, and have a custom theme you're using with your site, I'd suggest modifying the template file for that content type to print out the node id somewhere below or above the content.
However, if you're not very good with theming a really creative way to do it is by creating a view block and enabling it for that content type.  Then the node id can be displayed on the side of the content.  It can be nothing more than a line... doesn't have to be obtrusive.
To do the above:

Create a new view and call it something like 'Node id display' (uncheck the page version of the view and only create a block version).  Restrict it to the content type you want here, and make sure it's displaying fields, rather than titles, summaries or whatever is the default
Under the view's 'Fields' section, remove all fields that appear by default and only dislay the 'Content: Nid' field.  You can change the label to be something like "Node id" instead of 'Nid' if you prefer.
Under the 'Advanced' section of the view, add a contextual filter of 'Content: Nid', and in the configurations, select for it to 'Provide a default value' if a filter is not present.  From the drop-down, select 'Content ID from URL'
Either through a context or from the block's admin page, enable the new block for your specific content type... place the block on a sidebar, above the content, below the content... anywhere you want.

:-)
Hope this helps!
Comment: I tried most of this.  I should mention that I'm on drupal 5.8. I'm sure that this was way different back then, no?  I've followed your steps as much as I can, but I am not seeing the results I'd expect.
Comment: Oh no!  That's why it's so important to share your Drupal version.  I've used Drupal 6, and contextual filters are similar there, but have never used Drupal 5.  Is that the part you couldn't set up?  Setting up the contextual filter is crucial to get it to show up for the current node.  I'll go ahead and tag your question with the Drupal version too, in case there are any Drupal 5 gurus keeping up with that queue.
Comment: Unlikely, eh?  :)
Comment: So is it step #3 you're having trouble completing?
Comment: Many of the steps are different in D5.  Not just #3.


import certificate using command line in Win XP Home

Question: I need to install/import a CA cert (.cer) into the Trusted Root Cert Authorities store of a Win XP Home using command line. Anyone has any idea which command i can use?
Comment: because i'm creating a script that doesn't require user interaction
Comment: I don't mind installing other packages...
Comment: Why the command line particularly? Why can't you import using certmgr.msc?
Answer: Certutil.exe should do the trick, more details here in this Microsoft KB article. I've only used certutil on Server 2003 and XP Pro but according to this Microsoft KB XP Home is not listed so you may have issues.
If Certutil doesn't work for you you may be able to use a third party tool. Here's an example of a project that appears to do this but if there are issues with Certutil working I suspect there may be issues with this too, although you might be lucky.
Comment: I ended up installing Admin pack on a Win XP pro box and copying the following files to the Win XP home box: certadm.dll, certcli.dll, certreq.exe, certutil.exe
Answer: You need to use the version of certutil.exe from the Windows Server 2003 Administration Tools Pack.  Also QFE Q329357 is required.
Answer: Be careful using -f (force) or you may end up creating duplicate certificate stores.
Answer: Using Certutil you can run the following syntax:
<code>certutil -f -addstore -enterprise root Certificate_Name.cer
</code>


Why does this program NOT segfault?

Question: Usually, this question is probably phrased in a positive way, becoming the next member in the club of duplicate questions - this one hopefully isn't. I have written a simple program to reverse a string in C. Here it is:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void) {
    char arr[4] = "TEST";
    char rev[2];

    int j = 0;
    for(int i = 0; i < 4; i++) {
        j = 4 - i - 1;
        rev[j] = arr[i];
    }

    printf("%s\n",rev);
}

When I define char arr and char rev to be of size 4, everything works fine. When I leave arr size out I get unexpected repeat output like "TSTTST". When I define rev to be an array of 2 chars, I do not get a segfault, yet in the loop I am trying to access its third and fourth element. As far as my relatively limited understanding tells me, accessing the third element in an array of length two should segfault, right? So why doesn't it?
EDIT:
Interestingly enough, when I leave the loop out like so
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void) {
    char arr[4] = "TEST";
    char rev[2] = "00";

    printf("%s\n",rev);
}

it prints "00TEST". What happened here? Some kind of overflow? I even restarted the terminal, recompiled and ran again.
EDIT 2:
I have been made aware that this is indeed a duplicate. However, most of the suggested duplicates referred to C++, which this isn't. I think this is a good question for new C programmers to learn about and understand undefined behavior. I, for one, didn't know that accessing an array out of bounds does not always cause a SEGFAULT. Also, I learned that I have to terminate string literals myself, which I falsely believed was done automatically. This is partly wrong: it is added automatically - the C99 Standard (TC3) says in 6.4.5  String literals that terminating nulls are added in translation phase 7. As per this answer and the answers for this question, char arrays are also null-terminated, but this is only safe if the array has the correct length (string length + 1 for null-terminator).
Comment: About your edit: the `printf` format specifier `%s` expects a null-terminated string. Your "strings" aren't null-terminated so you have yet another variation of undefined behavior.
Comment: [Why don't I get a segmentation fault when I write beyond the end of an array?](https://stackoverflow.com/q/12410016/995714), [No segmentation fault for accessing out of bound memory](https://stackoverflow.com/q/27730460/995714)
Comment: `%s` will print until `\0`. Which you lack
Comment: dupes in C: [No out of bounds error](https://stackoverflow.com/q/9137157/995714), [C - Off by one error, but no segmentation fault?](https://stackoverflow.com/q/34521292/995714), [Array allows out of bounds access in C? (duplicate)](https://stackoverflow.com/q/38416792/995714)
Answer: char rev[2] assigns a memory of size 2*sizeof(char) with variable/pointer rev. You are accessing memory not allocated to the pointer. It may or may not cause errors.
It might appear to work fine, but it isn't very safe at all. By writing data outside the allocated block of memory you are overwriting some data you shouldn't. This is one of the greatest causes of segfaults and other memory errors, and what you're observing with it appearing to work in this short program is what makes it so difficult to hunt down the root cause.
When you do rev[2] or rev[3] you are accessing rev + 2 and rev + 3 addresses which are not allocated to rev pointer. Since its a small program and there is nothing there, it's not causing any errors.
In respect to edit:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void) {
    char arr[4] = "TEST";
    char rev[2] = "00";

    printf("%s\n",rev);
}

%s prints till null is encountered, the size you have assigned of the arr and rev doesn't allow for null to be there, try changing values as follow:
    char arr[5] = "TEST";
    char rev[3] = "00";

The program will work as intended as in arr there will be TEST\0 and rev will be rev\0 where \0 is null character in C.
Give this article a read, it'll solve most of your queries.

Why does this program NOT segfault?


Corporate Finance experiment, non-parametric test

Question: I have an numerical, continous variable as my dependent variable, namely CDS. My independent variable is categorical, namely different time intervals after a downgrade.
Unfortunately my data does not have a normal distribution nor are the variances homogenous. Therefore I have to use a non-parametric test.
I want to show that the mean of my dependent variable for each category is significantly different from 0. So in the end I get more or less results like the ones in the simple table below.
<code>        Interval 1     Interval 2     Interval 3     Interval 4     Interval 5

 CDS             5*             6***           7**            5*             6
</code>
Which test do I have to use?
Comment: Do you have any idea what the distribution looks like, and if you could make it look more 'normal' through a transform like a logarithm?  That would be one approach.  Lots of data is not truly normal, but we go with close enough.
Comment: I tried it with logarithm as well as the square root, which did not really help. Both Shapiro-Wilk-Test and Levene Test showed that my data is not homogenous in its variances and not normally distributed. Even after said transformations.
Comment: CDS = credit default swap or something else? If it is credit default swap, what is the variable measuring -- some price of a CDS or some other aspect of it? WHat are the values in those intervals? Are they *counts*?
Comment: CDS stands for CDS-bond basis. It is a numerical value and stands for the change of the CDS-bond basis. The intervals are time intervals: "[-30,-1]", "[1,30]", "[31,60]", "[61,90]", "[1,90]", "[-1,1]", "[1,10]", where for example "[-1,1]" stands for the time interval from 1 day before and one day after the downgrade. "[1,90]" stands for the time interval from 1 day after to 90 days after the downgrade. Each time interval is a category. i would like to show the mean change of the CDS-bond-basis for each interval and whether it is statistically significant different from 0.
Answer: What claims (even basic ones) can you make about the possible range of values or distribution?
If, for example, they can take on negative as well as positive values, and zero is sort of the 'expected' value (used a little loosely), then you probably already have it: the odds of five observations all lying on the same side of zero in that case is 1:32 which at least gives you 97% confidence they are all statistically above zero in a simple non-parametric test.
Comment: Ok I guess you're right. But don't you know a specific test I could do? I need to show my work in my paper.

    Min.      1st Qu.   Median     Mean  3rd Qu.     Max.
-277.778  -13.433    1.970    4.903   19.476  235.460
Comment: Take a look at the sign test and related non-parametrics: https://en.wikipedia.org/wiki/Sign_test


Exim address rewrite is not working

Question: after I could not get my Postfix server to do a proper rewrite, I tried to use an Exim server instead, but I am failing again on a simple rewrite.
I have used a default configuration, added my variables into it and created a new router for delivery to my both local mail servers on my private network. This router is the first in the processing line and configured this way:
manualroute_domains:
  driver = manualroute
  domains = lsearch;/etc/exim/relay_hosts
  route_data = ${lookup{$domain}lsearch{/etc/exim/relay_hosts}}
  transport = remote_smtp
  verify_recipient = false
  verify_sender    = false

A Testmail from domain1 to domain2 was successfully delivered.
Now I started to create my first rewriting rule at the "begin rewrite" block:
firstname.lastname@example.com email@example.com SFfrs

A Test wirh exim -brw firstname.lastname@example.com results in exactly the way I want my domain being rewritten, but when I try to send a mail from domain1, with the gateway as my smarthost, it always is received with the domain1.com ending and the new address is nowhere found in the header of the mail.
Is it possible, that I need to declare the use of rewriting in my router somewhere? I want to add conditions on my rewriting, so the use of header_rewrite on the router is no option for me.
Thank you all in advance for any idea or thought regarding my configuration!
With best regards
TooCloudy
Comment: What conditions do you intend to add?  You have the `S` flag set which means this rule fires at SMTP time before exim even decides what `$local_part` and `$domain` might be, which would likely make it hard to set any conditions.  See http://www.exim.org/exim-html-current/doc/html/spec_html/ch-address_rewriting.html#SECTrewriteS
Comment: Good to know, thank you! I want to do a conditional rewrite in the following way: If dom1 to dom2 or dom2 to dom1 don't do a rewrite. If mime is encrypted or signed dont do a rewrite. If both prior rules were not applied, do a rewrite incoming and outgoing on a fixed filetable list.
Answer: You are rewriting at the wrong time. The S rewrite option specifies rewriting incoming addresses as they received.  The pattern must match the whole address received in the MAIL FROM command.  You likely want to rewrite when the headers are read during data processing.
Please be aware that the rewritten addresses may trigger SPAM avoidance measures.  In particular, the SPF records for the domain are likely not to include your host as a permitted sender.
Try using a rewrite rule something like:
*@example.com      $email@example.com       Ffrs

If you want to rewrite individual addresses use /etc/email-aliases which will use one of these rewrite rules.
*@+local_domains "${lookup{${local_part}}lsearch{/etc/email-addresses}\
               {$value}fail}" Ffrs
*@ETC_MAILNAME "${lookup{${local_part}}lsearch{/etc/email-addresses}\
               {$value}fail}" Ffrs

You may need to create the /etc/email-addresses.  Although it is commonly supported, it is rarely used.  The above rules will work if your senders are all in the local domain(s) and should be rewritten for all local domains.
If you need to rewrite depended on the domain as well as the local_part, you need to configure for virtual domains.  If you have a defined a domain list virtual_domains the following rewrite rule should rewrite based on email-alias format files in /etc/exim4/virtual with names like email-example.com.
*@+virtual_domains "${lookup{${local_part}}lsearch{/etc/exim4/virtual/email-${domain}}\
               {$value}fail}" Ffrs

Alternatively, you could match domain qualified addresses like email@example.com in the /etc/aliases file using a rule like:
*@+local_domains "${lookup{${local_part}@${domain}}lsearch{/etc/email-addresses}\
               {$value}fail}" Ffrs

You can test your rewrite rules as documented in the Exim4 Specification rewrite chapter with a command like:
exim -brw email@example.com

Both alias and etc-aliases should contain lines like:
local_user:  email@example.com

Comment: That is strange, when I use the wildcard, the rewrite is applied without any problems. Unfortunately I need to rewrite the domain on a sender dependent list and cannot use wildcards. :( A file named `/etc/email-aliases` cannot be found on my system (CentOS 7.3), but I do have a router "system_aliases" with a lookup into the direction `data = ${lookup{$local_part}lsearch{/etc/aliases}}`. Is this the correct file?
Comment: @TooCloudy No, /etc/aliases is not the correct file. It is for addresses being delivered to.  It alters the list of addresses being delivered too without a rewrite.  This is standard email functionality.  I've updated my answer with more details on /etc/email-aliases. `man etc-alias` may provide a man page for aliases and email-aliases.

Exim address rewrite is not working


Insert data from sql file when call hook_schema on drupal 7

Question: I developed a module in drupal 7 that need a database table like "example_table". And i create "example_table" table using example_schema() function. Now I need to insert some default data.
Here, I can use drupal_write_record('example_table', $data); function to insert data into example_install() function.
But, I want to insert data from example_db.sql file.
How can I do it? Any Suggestions?
Answer: You can try this, hope it helps.
$db = new PDO($dsn, $user, $password);
$sql = file_get_contents('file.sql');
$qr = $db->exec($sql);

Otherwise try this, but i am also not aware of this correctly,
function my_module_install() {
  // Get the contents of the .sql file.
  $sql = file_get_contents(drupal_get_path('module', 'my_module') . '/module.sql');
  // Split into individual statements based on new lines.
  $statements = preg_split('/[\n\r]+/', $sql);
  foreach ($statements as $query) {
    db_query($query);
  }
}

Comment: Thank you Reddy, your first solution working... :), I update my question with answer, for first commer...
Comment: Hi @A Ajay Kumar Reddy, sorry for removing ans mark. I m working with drupal, maybe drupal has their own system without declaring username and password. So, i just need insert data without declaring username and password.


What does LayoutInflater in Android do?

Question: What is the use of LayoutInflater in Android?
Comment: Check if [this](https://stackoverflow.com/questions/51729036/what-is-a-layoutinflater-and-how-do-i-use-it-properly/51729037#51729037) one helps.
Answer: The LayoutInflater class is used to instantiate the contents of layout XML files into their corresponding View objects.
In other words, it takes an XML file as input and builds the View objects from it.
Comment: What i was looking for is the "in other words" part since the top part is already in the API docs
Comment: This still seems pretty abstract to me. So, let's say I have a `single_list_item.xml` file for a each line of a `ListView`. Is the use of that XML file sort of like an inflater?
Comment: in fact, the API docs say it all. Instantiating the contents of XML is the precise description. Sometimes we read too deep into the text and miss the obvious
Answer: What does LayoutInflator do?
When I first started Android programming, I was really confused by LayoutInflater and findViewById. Sometimes we used one and sometimes the other.

LayoutInflater is used to create a new View (or Layout) object from one of your xml layouts.
findViewById just gives you a reference to a view than has already been created. You might think that you haven't created any views yet, but whenever you call setContentView in onCreate, the activity's layout along with its subviews gets inflated (created) behind the scenes.

So if the view already exists, then use findViewById. If not, then create it with a LayoutInflater.
Example
Here is a mini project I made that shows both LayoutInflater and findViewById in action. With no special code, the layout looks like this.

The blue square is a custom layout inserted into the main layout with include (see here for more). It was inflated automatically because it is part of the content view. As you can see, there is nothing special about the code.
public class MainActivity extends AppCompatActivity {
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
    }
}

Now let's inflate (create) another copy of our custom layout and add it in.

LayoutInflater inflater = getLayoutInflater();
View myLayout = inflater.inflate(R.layout.my_layout, mainLayout, false);

To inflate the new view layout, all I did was tell the inflater the name of my xml file (my_layout), the parent layout that I want to add it to (mainLayout), and that I don't actually want to add it yet (false). (I could also set the parent to null, but then the layout parameters of my custom layout's root view would be ignored.)
Here it is again in context.
public class MainActivity extends AppCompatActivity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        // inflate the main layout for the activity
        setContentView(R.layout.activity_main);

        // get a reference to the already created main layout
        LinearLayout mainLayout = (LinearLayout) findViewById(R.id.activity_main_layout);

        // inflate (create) another copy of our custom layout
        LayoutInflater inflater = getLayoutInflater();
        View myLayout = inflater.inflate(R.layout.my_layout, mainLayout, false);

        // make changes to our custom layout and its subviews
        myLayout.setBackgroundColor(ContextCompat.getColor(this, R.color.colorAccent));
        TextView textView = (TextView) myLayout.findViewById(R.id.textView);
        textView.setText("New Layout");

        // add our custom layout to the main layout
        mainLayout.addView(myLayout);
    }
}

Notice how findViewById is used only after a layout has already been inflated.
Supplemental Code
Here is the xml for the example above.
activity_main.xml
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/activity_main_layout"
    android:orientation="horizontal"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:padding="16dp">

    <include layout="@layout/my_layout"/>

</LinearLayout>

my_layout.xml
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="100dp"
    android:layout_height="100dp"
    android:background="@color/colorPrimary">

    <TextView
        android:id="@+id/textView"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerInParent="true"
        android:padding="5dp"
        android:textColor="@android:color/white"
        android:text="My Layout"/>

</RelativeLayout>

When do you need LayoutInflater

The most common time most people use it is in a RecyclerView. (See these RecyclerView examples for a list or a grid.) You have to inflate a new layout for every single visible item in the list or grid.
You also can use a layout inflater if you have a complex layout that you want to add programmatically (like we did in our example). You could do it all in code, but it is much easier to define it in xml first and then just inflate it.
Comment: Hello sir, If I have more then one view within main_activity.xml , then how can i set the external.xml view in main_activity with center layout_gravity.
Comment: Is mylayout a View type or LinearLayout type?
Comment: @HSSingh, `myLayout` is a View, though I could have inflated it as a RelativeLayout (see my_layout.xml). After it is inflated I add it as a subview of the LinearLayout parent (that is, `mainLayout`).
Answer: When you use a custom view in a ListView you must define the row layout.
You create an xml where you place android widgets and then in the adapter's code you have to do something like this:
public MyAdapter(Context context, List<MyObject> objects) extends ArrayAdapter {
  super(context, 1, objects);
  /* We get the inflator in the constructor */
  mInflater = (LayoutInflater) context.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
}

@Override
public View getView(int position, View convertView, ViewGroup parent) {
  View view;
  /* We inflate the xml which gives us a view */
  view = mInflater.inflate(R.layout.my_list_custom_row, parent, false);

  /* Get the item in the adapter */
  MyObject myObject = getItem(position);

  /* Get the widget with id name which is defined in the xml of the row */
  TextView name = (TextView) view.findViewById(R.id.name);

  /* Populate the row's xml with info from the item */
  name.setText(myObject.getName());

  /* Return the generated view */
  return view;
}

Read more in the official documentation.
Comment: Ideally you should first test convertView to see if you can recycle a resource, so `View view = convertView; if (view == null) { view = mInflater.... }`
Comment: I don't think this answer really explains what a `LayoutInflater` is, although it explains where to use it. The answer 1-down is better.
Comment: This does not explain what a LayoutInflater is. It explains how to use it.
Comment: I'm looking for the explanation of LayoutInflater.
Comment: @Player1 You can check [this](https://stackoverflow.com/questions/51729036/what-is-layoutinflater-and-how-do-i-use-it-properly/51729037#51729037) post
Answer: LayoutInflater.inflate() provides a means to convert a res/layout/*.xml file defining a view into an actual View object usable in your application source code.
basic two steps: get the inflater and then inflate the resource
How do you get the inflater?
LayoutInflater inflater = (LayoutInflater) context
            .getSystemService(Context.LAYOUT_INFLATER_SERVICE);

How do you get the view assuming the xml file is "list_item.xml"?
View view = inflater.inflate(R.layout.list_item, parent, false);

Answer: Here is another example similar to the previous one, but extended to further demonstrate inflate parameters and dynamic behavior it can provide.
Suppose your ListView row layout can have variable number of TextViews. So first you inflate the base item View (just like the previous example), and then loop dynamically adding TextViews at run-time. Using android:layout_weight additionally aligns everything perfectly.
Here are the Layouts resources:
list_layout.xml
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:orientation="horizontal" >
    <TextView
        android:id="@+id/field1"
        android:layout_width="0dp"
        android:layout_height="wrap_content"
        android:layout_weight="2"/>
    <TextView
        android:id="@+id/field2"
        android:layout_width="0dp"
        android:layout_height="wrap_content"
        android:layout_weight="1"
/>
</LinearLayout>

schedule_layout.xml
<?xml version="1.0" encoding="utf-8"?>
   <TextView xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="0dp"
    android:layout_height="wrap_content"
    android:layout_weight="1"/>

Override getView method in extension of BaseAdapter class
@Override
public View getView(int position, View convertView, ViewGroup parent) {
    LayoutInflater inflater = activity.getLayoutInflater();
    View lst_item_view = inflater.inflate(R.layout.list_layout, null);
    TextView t1 = (TextView) lst_item_view.findViewById(R.id.field1);
    TextView t2 = (TextView) lst_item_view.findViewById(R.id.field2);
    t1.setText("some value");
    t2.setText("another value");

    // dinamically add TextViews for each item in ArrayList list_schedule
    for(int i = 0; i < list_schedule.size(); i++){
        View schedule_view = inflater.inflate(R.layout.schedule_layout, (ViewGroup) lst_item_view, false);
        ((TextView)schedule_view).setText(list_schedule.get(i));
        ((ViewGroup) lst_item_view).addView(schedule_view);
    }
    return lst_item_view;
}

Note different inflate method calls:
inflater.inflate(R.layout.list_layout, null); // no parent
inflater.inflate(R.layout.schedule_layout, (ViewGroup) lst_item_view, false); // with parent preserving LayoutParams

Answer: This class is used to instantiate layout XML file into its corresponding View objects. It is never be used directly -- use getLayoutInflater() or getSystemService(String) to retrieve a standard LayoutInflater instance that is already hooked up to the current context and correctly configured for the device you are running on. For example:
LayoutInflater inflater = (LayoutInflater)context.getSystemService
      (Context.LAYOUT_INFLATER_SERVICE);

Reference: http://developer.android.com/reference/android/view/LayoutInflater.html
Comment: This may be true but does not answer the question.
Answer: LayoutInflater is a class used to instantiate layout XML file into its corresponding view objects which can be used in Java programs.
In simple terms, there are two ways to create UI in android. One is a static way and another is dynamic or programmatically.
Suppose we have a simple layout main.xml  having one textview and one edittext as follows.
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:id="@+id/layout1"
    >
<TextView
        android:id="@+id/namelabel"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Enter your name"
        android:textAppearance="?android:attr/textAppearanceLarge" >
    </TextView>
    <EditText
        android:id="@+id/name"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_alignParentLeft="true"
        android:layout_marginTop="14dp"
        android:ems="10">
    </EditText>
</LinearLayout>

We can display this layout in static way by
public void onCreate(Bundle savedInstanceState) {
  super.onCreate(savedInstanceState);
  setContentView(R.layout.main);

A dynamic way of creating a view means the view is not mentioned in our main.xml but we want to show with this in run time. For example, we have another XML in layout folder as  footer.xml
<?xml version="1.0" encoding="utf-8"?>
<TextView xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/TextView1"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:gravity="center_horizontal"
    android:text="Add your record"
    android:textSize="24sp" >
 </TextView>

We want to show this textbox in run time within our main UI. So here we will inflate text.xml. See how:
public void onCreate(Bundle savedInstanceState) {

  super.onCreate(savedInstanceState);
  setContentView(R.layout.main);
  final LayoutInflater  inflater = (LayoutInflater)getSystemService(Context.LAYOUT_INFLATER_SERVICE);
  TextView t = (TextView)inflater.inflate(R.layout.footer,null);

  lLayout = (LinearLayout)findViewById(R.id.layout1);
  lLayout.addView(t);

Here I have used getSystemService (String) to retrieve a  LayoutInflater instance. I can use getLayoutInflator() too to inflate instead of using getSystemService (String) like below:
LayoutInflator inflater = getLayoutInflater();
TextView t = (TextView) inflater.inflate(R.layout.footer, null);
lLayout.addView(t);

Answer: Inflating means reading the XML file that describes a layout (or GUI element) and to create the actual objects that correspond to it, and thus make the object visible within an Android app.
final Dialog mDateTimeDialog = new Dialog(MainActivity.this);

// Inflate the root layout
final RelativeLayout mDateTimeDialogView = (RelativeLayout) getLayoutInflater().inflate(R.layout.date_time_dialog, null);

// Grab widget instance
final DateTimePicker mDateTimePicker = (DateTimePicker) mDateTimeDialogView.findViewById(R.id.DateTimePicker);

This file could saved as date_time_dialog.xml:
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/DateTimeDialog" android:layout_width="100px"
    android:layout_height="wrap_content">
    <com.dt.datetimepicker.DateTimePicker
            android:id="@+id/DateTimePicker" android:layout_width="fill_parent"
            android:layout_height="wrap_content" />
    <LinearLayout android:id="@+id/ControlButtons"
            android:layout_width="fill_parent" android:layout_height="wrap_content"
            android:layout_below="@+id/DateTimePicker"
            android:padding="5dip">
            <Button android:id="@+id/SetDateTime" android:layout_width="0dip"
                    android:text="@android:string/ok" android:layout_weight="1"
                    android:layout_height="wrap_content"
                   />
            <Button android:id="@+id/ResetDateTime" android:layout_width="0dip"
                    android:text="Reset" android:layout_weight="1"
                    android:layout_height="wrap_content"
                    />
            <Button android:id="@+id/CancelDialog" android:layout_width="0dip"
                    android:text="@android:string/cancel" android:layout_weight="1"
                    android:layout_height="wrap_content"
                     />
    </LinearLayout>

This file could saved as date_time_picker.xml:
 <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent" android:layout_height="wrap_content" `enter code here`
    android:padding="5dip" android:id="@+id/DateTimePicker">
<LinearLayout
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:baselineAligned="true"
android:orientation="horizontal">

    <LinearLayout
    android:id="@+id/month_container"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_marginLeft="1dp"
    android:layout_marginTop="5dp"
    android:layout_marginRight="5dp"
    android:layout_marginBottom="5dp"
    android:gravity="center"
    android:orientation="vertical">
    <Button
        android:id="@+id/month_plus"
        android:layout_width="45dp"
        android:layout_height="45dp"
        android:background="@drawable/image_button_up_final"/>
    <EditText
        android:id="@+id/month_display"
        android:layout_width="45dp"
        android:layout_height="35dp"
        android:background="@drawable/picker_middle"
        android:focusable="false"
        android:gravity="center"
        android:singleLine="true"
        android:textColor="#000000">
    </EditText>
    <Button
        android:id="@+id/month_minus"
        android:layout_width="45dp"
        android:layout_height="45dp"
        android:background="@drawable/image_button_down_final"/>
</LinearLayout>
<LinearLayout
    android:id="@+id/date_container"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_marginLeft="0.5dp"
    android:layout_marginTop="5dp"
    android:layout_marginRight="5dp"
    android:layout_marginBottom="5dp"
    android:gravity="center"
    android:orientation="vertical">
    <Button
        android:id="@+id/date_plus"
        android:layout_width="45dp"
        android:layout_height="45dp"
        android:background="@drawable/image_button_up_final"/>
    <EditText
        android:id="@+id/date_display"
        android:layout_width="45dp"
        android:layout_height="35dp"
        android:background="@drawable/picker_middle"
        android:gravity="center"
        android:focusable="false"
        android:inputType="number"
        android:textColor="#000000"
        android:singleLine="true"/>
    <Button
        android:id="@+id/date_minus"
        android:layout_width="45dp"
        android:layout_height="45dp"
        android:background="@drawable/image_button_down_final"/>
</LinearLayout>
<LinearLayout
    android:id="@+id/year_container"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_marginLeft="0.5dp"
    android:layout_marginTop="5dp"
    android:layout_marginRight="5dp"
    android:layout_marginBottom="5dp"
    android:gravity="center"
    android:orientation="vertical">
    <Button
        android:id="@+id/year_plus"
        android:layout_width="45dp"
        android:layout_height="45dp"
            android:background="@drawable/image_button_up_final"/>
    <EditText
        android:id="@+id/year_display"
        android:layout_width="45dp"
        android:layout_height="35dp"
        android:background="@drawable/picker_middle"
        android:gravity="center"
        android:focusable="false"
        android:inputType="number"
        android:textColor="#000000"
        android:singleLine="true"/>
    <Button
        android:id="@+id/year_minus"
        android:layout_width="45dp"
        android:layout_height="45dp"
        android:background="@drawable/image_button_down_final"/>
</LinearLayout>
<LinearLayout
        android:id="@+id/hour_container"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginLeft="5dp"
        android:layout_marginRight="5dp"
        android:layout_marginTop="5dp"
        android:layout_marginBottom="5dp"
        android:gravity="center"
        android:orientation="vertical">
        <Button
            android:id="@+id/hour_plus"
            android:layout_width="45dp"
            android:layout_height="45dp"
            android:background="@drawable/image_button_up_final"/>
        <EditText
            android:id="@+id/hour_display"
            android:layout_width="45dp"
            android:layout_height="35dp"
            android:background="@drawable/picker_middle"
            android:gravity="center"
            android:focusable="false"
            android:inputType="number"
            android:textColor="#000000"
            android:singleLine="true">
        </EditText>
        <Button
            android:id="@+id/hour_minus"
            android:layout_width="45dp"
            android:layout_height="45dp"
            android:background="@drawable/image_button_down_final"/>
    </LinearLayout>
    <LinearLayout
        android:id="@+id/min_container"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginLeft="0.35dp"
        android:layout_marginTop="5dp"
        android:layout_marginRight="5dp"
        android:layout_marginBottom="5dp"
        android:gravity="center"
        android:orientation="vertical">
        <Button
            android:id="@+id/min_plus"
            android:layout_width="45dp"
            android:layout_height="45dp"
            android:background="@drawable/image_button_up_final"/>
        <EditText
            android:id="@+id/min_display"
            android:layout_width="45dp"
            android:layout_height="35dp"
            android:background="@drawable/picker_middle"
            android:gravity="center"
            android:focusable="false"
            android:inputType="number"
            android:textColor="#000000"
            android:singleLine="true"/>
        <Button
            android:id="@+id/min_minus"
            android:layout_width="45dp"
            android:layout_height="45dp"
            android:background="@drawable/image_button_down_final"/>
    </LinearLayout>

    <LinearLayout
        android:id="@+id/meridiem_container"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginLeft="0.35dp"
        android:layout_marginTop="5dp"
        android:layout_marginRight="5dp"
        android:layout_marginBottom="5dp"
        android:gravity="center"
        android:orientation="vertical">
        <ToggleButton
            android:id="@+id/toggle_display"
            style="@style/SpecialToggleButton"
            android:layout_width="40dp"
            android:layout_height="32dp"
            android:layout_marginLeft="5dp"
            android:layout_marginTop="45dp"
            android:layout_marginRight="5dp"
            android:layout_marginBottom="5dp"
            android:padding="5dp"
            android:gravity="center"
            android:textOn="@string/meridiem_AM"
            android:textOff="@string/meridiem_PM"
            android:checked="true"/>

    </LinearLayout>
</LinearLayout>
</RelativeLayout>

The MainActivity class saved as MainActivity.java:
public class MainActivity extends Activity {
    EditText editText;
    Button button_click;
    public static Activity me = null;
    String meridiem;
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        editText = (EditText)findViewById(R.id.edittext1);
        button_click = (Button)findViewById(R.id.button1);
        button_click.setOnClickListener(new OnClickListener() {
            @Override
            public void onClick(View view){
                final Dialog mDateTimeDialog = new Dialog(MainActivity.this);
                final RelativeLayout mDateTimeDialogView = (RelativeLayout)   getLayoutInflater().inflate(R.layout.date_time_dialog, null);
                final DateTimePicker mDateTimePicker = (DateTimePicker) mDateTimeDialogView.findViewById(R.id.DateTimePicker);
                // mDateTimePicker.setDateChangedListener();
                ((Button) mDateTimeDialogView.findViewById(R.id.SetDateTime)).setOnClickListener(new OnClickListener() {
                    public void onClick(View v) {
                        mDateTimePicker.clearFocus();
                        int hour = mDateTimePicker.getHour();
                        String result_string = mDateTimePicker.getMonth() +" "+   String.valueOf(mDateTimePicker.getDay()) + ", " + String.valueOf(mDateTimePicker.getYear())
                        + "  " +(mDateTimePicker.getHour()<=9? String.valueOf("0"+mDateTimePicker.getHour()) : String.valueOf(mDateTimePicker.getHour())) + ":" + (mDateTimePicker.getMinute()<=9?String.valueOf("0"+mDateTimePicker.getMinute()):String.valueOf(mDateTimePicker.getMinute()))+" "+mDateTimePicker.getMeridiem();
                        editText.setText(result_string);
                        mDateTimeDialog.dismiss();
                    }
                });
                // Cancel the dialog when the "Cancel" button is clicked
                ((Button) mDateTimeDialogView.findViewById(R.id.CancelDialog)).setOnClickListener(new OnClickListener() {
                    public void onClick(View v) {
                        // TODO Auto-generated method stub
                        mDateTimeDialog.cancel();
                    }
                });
                // Reset Date and Time pickers when the "Reset" button is clicked
                ((Button) mDateTimeDialogView.findViewById(R.id.ResetDateTime)).setOnClickListener(new OnClickListener() {
                    public void onClick(View v) {
                        // TODO Auto-generated method stub
                        mDateTimePicker.reset();
                    }
                });

                // Setup TimePicker
                // No title on the dialog window
                mDateTimeDialog.requestWindowFeature(Window.FEATURE_NO_TITLE);
                // Set the dialog content view
                mDateTimeDialog.setContentView(mDateTimeDialogView);
                // Display the dialog
                mDateTimeDialog.show();
            }
        });
    }
}

Answer: What inflater does
It takes a xml layout as input (say) and converts it to View object.
Why needed
Let us think a scenario where we need to create a custom listview. Now each row should be custom. But how can we do it. Its not possible to assign a xml layout to a row of listview. So, we create a View object. Thus we can access the elements in it (textview,imageview etc) and also assign the object as row of listview
So, whenever we need to assign view type object somewhere and we have our custom xml design we just convert it to object by inflater and use it.
Comment: So, Mister Zahan, is it like the DOM in Javascript? •o•
Answer: here is an example for geting a refrence for the root View of a layout ,
inflating it and using it with setContentView(View view)
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    LayoutInflater li=getLayoutInflater();
    View rootView=li.inflate(R.layout.activity_main,null);
    setContentView(rootView);

}

Answer: Layout inflater is a class that reads the xml appearance description and convert them into java based View objects.
Answer: LayoutInflater creates View objects based on layouts defined in XML. There are several different ways to use LayoutInflater, including creating custom Views, inflating Fragment views into Activity views, creating Dialogs, or simply inflating a layout file View into an Activity.
There are a lot of misconceptions about how the inflation process works. I think this comes from poor of the documentation for the inflate() method. If you want to learn about the inflate() method in detail, I wrote a blog post about it here:
https://www.bignerdranch.com/blog/understanding-androids-layoutinflater-inflate/
Answer: my customize list hope it illustrate concept
public class second extends ListActivity {

@Override
protected void onCreate(Bundle savedInstanceState) {
    // TODO Auto-generated method stub
    super.onCreate(savedInstanceState);
    setContentView(R.layout.second);
//  TextView textview=(TextView)findViewById(R.id.textView1);
//  textview.setText(getIntent().getExtras().getString("value"));

    setListAdapter(new MyAdapter(this,R.layout.list_item,R.id.textView1, getResources().getStringArray(R.array.counteries)));
}

private class MyAdapter extends ArrayAdapter<String>{

    public MyAdapter(Context context, int resource, int textViewResourceId,
            String[] objects) {
        super(context, resource, textViewResourceId, objects);
        // TODO Auto-generated constructor stub
    }

    @Override
    public View getView(int position, View convertView, ViewGroup parent) {

        LayoutInflater inflater=(LayoutInflater) getSystemService(Context.LAYOUT_INFLATER_SERVICE);
        View row=inflater.inflate(R.layout.list_item,parent,false);
        String[]items=getResources().getStringArray(R.array.counteries);

        ImageView iv=(ImageView) row.findViewById(R.id.imageView1);
        TextView tv=(TextView) row.findViewById(R.id.textView1);
        tv.setText(items[position]);

        if(items[position].equals("unitedstates")){
            iv.setImageResource(R.drawable.usa);
        }else   if(items[position].equals("Russia")){
            iv.setImageResource(R.drawable.russia);
        }else   if(items[position].equals("Japan")){
            iv.setImageResource(R.drawable.japan);
        }
        // TODO Auto-generated method stub
        return row;
    }

}

}

Answer: LayoutInflater is a fundamental component in Android. You must use it all the time to turn xml files into view hierarchies.
Answer: Inflater actually some sort of convert to data,  views, instances, to visible UI representation.. ..thus it make use of data feed into from maybe adapters,  etc. programmatically. then integrating it with an xml you defined, that tells it how the data should be represented in UI

What does LayoutInflater in Android do?


How come regularization knows which coefficients to decrease and which coefficients to increase/leave as they are?

Question: I am just learning about regularization and I haven't found a satisfying answer to this question. When I perform regularization, how come the right coefficients get decreased or turned into $0$?
I just learned that for ridge regression we add the term:
$$\lambda \sum_{j=1}^p \beta_j^2$$
to the cost function. $\lambda$ is the regularization parameter, $p$ is the number of predictors and $\beta_j$ are the coefficients of the model's parameters.
I learned that in this type of regression the coefficients can be decreased, but they are never quite $0$.
In the case of lasso regression we add the term:
$$\lambda \sum_{j=1}^p |\beta_j|$$
to the cost function. Unlike ridge regression, here the coefficients can be turned to $0$.
So far so good. What really perplexes me, seems like magic, is how come the right coefficients are decreased/turned into $0$, and the right coefficients are increased/left out as they are? How come regression doesn't decrease a coefficient that would actually improve the model and that we would actually consider to be important? How come it doesn't make a mistake like that? I mean, we just added a term to the cost function. It seems too simple to work, to do the right thing. How does regularization know which coefficients to mess with and which coefficients not to? I don't see anything in the math of regression that tells this algorithm which coefficients to decrease. How does it know?
Comment: The answer is in the rest of the cost function. You're still trying to fit a function to the data by minimizing the sum of the errors (typically) of some sort. So the regularization keeps the coefficients from getting too big (and hence overfitting), while the sum-of-errors term tries to match the data more closely.
Comment: I would argue that it doesn't get it right. By regularizing, you bias the parameter estimates. However, it gets it wrong in a useful way by constraining the estimated values and keeping them from fluctuating too wildly, exchanging unbiasedness somewhat for a large reduction in variance. (Or at least we hope it gets it wrong in a useful way.)
Answer: It basically depends on how you're updating the value of your parameters. Let's first clarify what is happening without regularization. Suppose you want to minimize a function (objective function) that depends on a parameter $p$ and a variable $x$ as follows
\begin{align}
f(x,p)
\end{align}
and that at each step you're updating the value of your parameter $b$ with the next formula
\begin{align*}
p = p - \gamma \nabla_{p}f(x,p) \:\:\: (1)
\end{align*}
where $\gamma$ is normally known as the learning rate and $\nabla$ denotes the gradient of the function with respect to $p$. Now, if add a term that depends on the parameter $p$ to the objective function it would affect how I update my parameter $p$. For instance,
\begin{align}
f(x,p) + \lambda p^2
\end{align}
where we're considering $\lambda > 0$. Thus, we reformulate our updating parameter formula as
\begin{align}
p = p - \gamma [\nabla_{p}f(x,p) + 2\lambda p]
\end{align}
which is equivalent to
\begin{align}
p = (1) -2\gamma\lambda p  \:\:\: (2)
\end{align}
It is clear to see that (2) < (1). Thus, that's how regularization shrinks coefficients.
Answer:
When you are adding the regularization term in the loss function, you are essentially penalizing the coefficients which are causing the loss to be high.
Now through the gradient descent, the model is incentivized to penalize all these coefficients to lower values which in turn reduces the loss.

Hope this helps building some intuition!
Answer: Basketball players cost money.  To save money, it's best for a team not to have any players. However, we also want to win games.
A penalty is like penalizing the money we need to spend on players.  Winning games is like maximizing likelihood (or minimizing cost).
If we want to both win games and also minimize our spending, we only keep the best players.
A penalty (which is analogous to a penalty on how much we spend) is put on a likelihood  (aka cost, which is negative log likelihood usually, and is analogous to winning games)
$$Likelihood + weight*penalty$$
Define "important" as important to maximize cost/likelihood. Any penalization of the coefficients on the important covariates decreases likelihood; the answer is that with a big enough weight on the penalty, it will penalize the coefficients for the important covariates.  If weight is small enough, it will try to still maximize likelihood, which can only be achieved by penalizing the non-important coefficients and leaving the important ones alone.


How to loop through a Pandas dataframe

Question: I am trying to loop through a Pandas dataframe.  The list L includes values that are used to specify what row X or Y should begin from i.e., (1:, 2:, 3:).
list = [1,2,3]

for L in list:
    X = data.ix[L:, 'X':]
    Y = data.ix[L:, 'Y']
    regressor = LinearRegression()
    regressor.fit(X, Y)
    prediction = regressor.predict([[Variable]])

The error when attempting the above is:
TypeError: 'type' object is not iterable

Comment: don't you mean `for l in L:`? You're trying to iterate over the `list` type, also why are you recreating the regression model on each iteration? just define it outside once and refer to it
Comment: Also all you're doing is just slicing the df incrementally, you can achieve the same by doing `for df in data.iloc[::1]: X = df['X'] Y = df['Y'] e.t.c.`
Comment: Ed, I am recreating the regression model on each iteration because each subset of X/Y is a different model with different results.  I'm essentially trying to do a regression with a rolling window of model inputs.
Comment: Ed, could you please lay out how exactly you would implement your solution.  I don't quite follow.  Thank you so much.
Answer: IIUC you can do something like the following:
l = [1,2,3]
results = []

for idx in l:
    X = data.ix[idx:, 'X':]
    Y = data.ix[idx:, 'Y']
    regressor = LinearRegression()
    regressor.fit(X, Y)
    results.append(regressor.predict([[Variable]]))

However, I don't know what Variable is here, you could also just do the following:
for df in data.iloc[::1]:
    regressor = LinearRegression()
    regressor.fit(df['X'], df['Y'])
    results.append(regressor.predict([[Variable]]))

Comment: Hi EdChum - The first example you posted is similar to the solution I gave.  I get the error: ValueError: shapes (1,1) and (4,) not aligned: 1 (dim 1) != 4 (dim 0).  For your second example, it is unclear to me how you are looping through the list of 1,2,3.
Comment: In the second example I don't loop through the list, it just shows how to iteratively slice the df, for your new error it supposed to be `X = data.ix[idx:, 'X']` instead?
Comment: I am getting the same error with:     for idx in list:X = data.ix[idx:, 'X':]
    Y = data.ix[idx:, 'Y']
Answer: You should try iterrrows,[http://pandas-docs.github.io/pandas-docs-travis/basics.html#iterrows]
>>> df = pd.DataFrame([[1, 1.5]], columns=['int', 'float'])
>>> row = next(df.iterrows())[1]
>>> row
int      1.0
float    1.5
Name: 0, dtype: float64

Answer: You can also convert it to dict or list in before and than use as you know:
list_from_df = df.to_list()
for item in list_from_df:
   print(item)

Or as a dict:
df.to_dict()
for key, value in list_from_df.items():
   print(key) # index
   print(value)

How to loop through a Pandas dataframe


Klein bottle homology by Mayer Vietoris

Question: I'm still applying Mayer Vietoris, this time to the Klein bottle. I'm using the decomposition as on Wikipedia here and I've calculated $H_0$ and $H_n$ for $n \geq 2$ correctly. Now I'm struggling with $H_1$.
My sequence:
$$ 0 \xrightarrow{} H_1( S^1) \xrightarrow{(i,j)} H_1(M) \oplus H_1(M^\prime )\xrightarrow{k-l} H_1(K) \xrightarrow{\partial_1} \tilde{H_0} = 0 $$
Wikipedia writes "The central map $(i,j)$ sends $1$ to $(2, −2)$". Does it matter whether it's sent to $(2,-2)$ or $(2,2)$ ? I think $(i,j)$ maps $1$ to $(2,2)$.
Using this, I get
(i) $im((i,j)) = 2 \mathbb{Z} \oplus 2 \mathbb{Z} = ker (k-l)$
And using the first isomorphism theorem I get
(ii) $H_1(K) / ker (\partial_1) = im(\partial_1) = \tilde{H_0} = 0$
and therefore $H_1(K) \cong \mathbb{Z}$ because $ker(\partial_1) = \mathbb{Z}$ which is clearly wrong but I don't see where the mistake is.
(iii) I also know $k-l$ is surjective so $im (k-l) = H_1(K) \cong \mathbb{Z} \oplus \mathbb{Z} / ker (k-l)$
But if $ker (k-l) = 2 \mathbb{Z} \oplus 2 \mathbb{Z} $ then I'd get $H_1(K) =  \mathbb{Z}/2 \oplus  \mathbb{Z}/2 $
which is also wrong.
What am I doing wrong? Many thanks for your help!
Comment: Why is $A\cap B\simeq \mathbb{S}^1$? According to the wikipedia picture we should have $A\cap B\simeq \mathbb{S}^1\sqcup\mathbb{S}^1$, shouldn't we?
Comment: I am doing my first course in algebraic (or in fact any) topology, and I often come across your questions on this site. By now it seems to me that you are like an expert in this field... Having done other topics in maths, I find this area unusually painful, and so I was just wondering how you found this journey? Any tips on resources besides Hatcher?
Answer: Where $(i,j)$ sends $1$ depends on how your inclusion maps look and which orientation you pick, that is which isomorphisms $H_1(S^1) \cong \mathbb Z$, $H_1(M) \cong \mathbb Z$ and $H_1(M') \cong \mathbb Z$ you pick. It is therefore ok to assume $(i,j)1 = (2,2)$.
Now you get
(1) $im(i,j) = \ker (k-l) = 2\mathbb Z(1,1) \not = 2\mathbb Z \times 2 \mathbb Z$.
(2) $\partial_1 = 0$ and therefore $k-l$ is surjective. We now have $H_1(K) \cong [\mathbb Z \times \mathbb Z] / [2\mathbb Z(1,1)]$.
(3) Use that $\mathbb Z \times \mathbb Z = \mathbb Z (1,0) \oplus \mathbb Z(1,1)$ to conclude $H_1(K) \cong \mathbb Z \times \mathbb Z/2\mathbb Z$.
Comment: You should read this with some linear algebra in mind. $2\mathbb Z (1,1)$ is the subgroup of $\mathbb Z \times \mathbb Z$ that consists of the elements $(n,n)$ for some $n \in 2\mathbb Z$.
Comment: Thank you! But what is $2 \mathbb{Z} (1,1)$? First I thought it's the free abelian group over the basis consisting of one element, $(1,1)$ but then that would be $\mathbb{Z}$ and I need it to be $\mathbb{Z} \oplus 2\mathbb{Z}$ but I don't see how $2 \mathbb{Z} (1,1) = \mathbb{Z} \oplus 2\mathbb{Z}$...

Klein bottle homology by Mayer Vietoris


Characterization of elements of $X^*$ via the Radon-Nikodym theorem

Question: I am reading Lindenstrauss' Classical Banach Spaces II and I am having trouble with the following characterization  of integrals. First a couple of preliminary definitions:

Let $(\Omega, \Sigma, \mu)$ a complete $\sigma$-finite space. A Banach
  space $X$ consisting of equivalence classes, modulo equality almost
  everywhere, of locally integrable real valued functions on $\Sigma$ is
  called a Köhte function space if the following conditions hold:

If $\vert f (\omega) \vert \le \vert g(\omega) \vert $ a.e. on $\Omega$ with $f$ measurable and $g \in X$, then $f \in X$ and $\Vert
> f \Vert \le \Vert g \Vert$.
For every $\sigma \in \Sigma$ with $\mu ( \sigma ) < \infty$ the characteristic function $\chi_\sigma$ of $\sigma$ belongs to X.

Also we have that:

Every measurable function $g$ on $\Omega$ such that $gf \in L_1(\mu)$ for every $f \in X$
  defines an element $x_g^* \in X^*$ by $x_g^*(f) = \int_\Omega
> f(\omega) g(\omega) d\mu$. Any functional of the form $x_g^*$ is
  called an integral.

Then the book states (this is the part that I haven't been able to work around):

It is an immediate consequence of the Radon-Nikodym theorem that a
  functional $x^* \in X^*$ is an integral is and only if, for every
  sequence $\{ f_n \} \in X$ with $f_n ( \omega) \downarrow 0$ a.e. we
  have that $\vert x^* \vert (f_n) \to 0$.

I don't understand how exactly this follows from the Radon-Nikodym theorem. Could anyone clarify it to me?
Comment: When you say $gf\in L^1$ you mean that $gf\in L^1$ for all $f\in X$, right?
Comment: @DavidC.Ullrich Yes, I forgot to add that. I have edited the post to reflect it.
Comment: Also what does $|x^*|(f)$ mean? Is that $|x^*(f)|$ or something else?
Comment: The reason I suspect it may be something else is that if $\nu$ is a complex measure then $|\nu|(E)\ne|\nu(E)|$. Also if it just meant $|x^*(f)|$ it's hard to see why he'd include the absolute value sign when he says $|x^*|(f_n)\to0$. So: What _is_ $|x^*|(f)$?
Answer: I haven't worked out the details - also I suspect we still haven't been given all the relevant definitions. But in case it helps, here's how the argument "must" go in outline, if it's by R-N:
Somehow we reduce to the case $\mu(\Omega)<\infty$. Define a complex measure $\nu$ by $$\nu(E)=x^*(\chi_E).$$Detail: Something shows somehow that $\nu$ is in fact countably additive. Now $$x^*\phi=\int\phi\,d\nu$$for integrable simple functions $\phi$. Detail: Somehow, maybe simple functions are dense, this extends to $\phi\in X$.
And now somehow that last condition shows somehow that $\nu<<\mu$.