<commit_before>#!/usr/bin/python

"""Xcode project file generator.

This module is both an Xcode project file generator and a documentation of the
Xcode project file format.  Knowledge of the project file format was gained
based on extensive experience with Xcode, and by making changes to projects in
Xcode.app and observing the resultant changes in the associated project files.

XCODE PROJECT FILES

The generator targets the file format as written by Xcode 3.1 (specifically,
3.1.2), but past experience has taught that the format has not changed
significantly in the past several years, and future versions of Xcode are able
to read older project files.

Xcode project files are "bundled": the project "file" from an end-user's
perspective is actually a directory with an ".xcodeproj" extension.  The
project file from this module's perspective is actually a file inside this
directory, always named "project.pbxproj".  This file contains a complete
description of the project and is all that is needed to use the xcodeproj.
Other files contained in the xcodeproj directory are simply used to store
per-user settings, such as the state of various UI elements in the Xcode
application.

The project.pbxproj file is a property list, stored in a format almost
identical to the NeXTstep property list format.  The file is able to carry
Unicode data, and is encoded in UTF-8.  The root element in the property list
is a dictionary that contains several properties of minimal interest, and two
properties of immense interest.  The most important property is a dictionary
named "objects".  The entire structure of the project is represented by the
children of this property.  The objects dictionary is keyed by unique 96-bit
values represented by 24 uppercase hexadecimal characters.  Each value in the
objects dictionary is itself a dictionary, describing an individual object.

Each object in the dictionary is a member of a class, which is identified by
the "isa" property of each object.  A variety of classes are represented in a
project file.  Objects can refer to other objects by ID, using the 24-character
hexadecimal object key.  A project's objects form a tree, with a root object
of class PBXProject at the root.  As an example, the PBXProject object serves
as parent to an XCConfigurationList object defining the build configurations
used in the project, a PBXGroup object serving as a container for all files
referenced in the project, and a list of target objects, each of which defines
a target in the project.  There are several different types of target object,
such as PBXNativeTarget and PBXAggregateTarget.  In this module, this
relationship is expressed by having each target type derive from an abstract
base named XCTarget.

The project.pbxproj file's root dictionary also contains a property, sibling to
the "objects" dictionary, named "rootObject".  The value of rootObject is a
24-character object key referring to the root PBXProject object in the
objects dictionary.

In Xcode, every file used as input to a target or produced as a final product
of a target must appear somewhere in the hierarchy rooted at the PBXGroup
object referenced by the PBXProject's mainGroup property.  A PBXGroup is
generally represented as a folder in the Xcode application.  PBXGroups can
contain other PBXGroups as well as PBXFileReferences, which are pointers to
actual files.

Each XCTarget contains a list of build phases, represented in this module by
the abstract base XCBuildPhase.  Examples of concrete XCBuildPhase derivations
are PBXSourcesBuildPhase and PBXFrameworksBuildPhase, which correspond to the
"Compile Sources" and "Link Binary With Libraries" phases displayed in the
Xcode application.  Files used as input to these phases (for example, source
files in the former case and libraries and frameworks in the latter) are
represented by PBXBuildFile objects, referenced by elements of "files" lists
in XCTarget objects.  Each PBXBuildFile object refers to a PBXBuildFile
object as a "weak" reference: it does not "own" the PBXBuildFile, which is
owned by the root object's mainGroup or a descendant group.  In most cases, the
layer of indirection between an XCBuildPhase and a PBXFileReference via a
PBXBuildFile appears extraneous, but there's actually one reason for this:
file-specific compiler flags are added to the PBXBuildFile object so as to
allow a single file to be a member of multiple targets while having distinct
compiler flags for each.  These flags can be modified in the Xcode applciation
in the "Build" tab of a File Info window.

When a project is open in the Xcode application, Xcode will rewrite it.  As
such, this module is careful to adhere to the formatting used by Xcode, to
avoid insignificant changes appearing in the file when it is used in the
Xcode application.  This will keep version control repositories happy, and
makes it possible to compare a project file used in Xcode to one generated by
this module to determine if any significant changes were made in the
application.

Xcode has its own way of assigning 24-character identifiers to each object,
which is not duplicated here.  Because the identifier only is only generated
once, when an object is created, and is then left unchanged, there is no need
to attempt to duplicate Xcode's behavior in this area.  The generator is free
to select any identifier, even at random, to refer to the objects it creates,
and Xcode will retain those identifiers and use them when subsequently
rewriting the project file.  However, the generator would choose new random
identifiers each time the project files are generated, leading to difficulties
comparing "used" project files to "pristine" ones produced by this module,
and causing the appearance of changes as every object identifier is changed
when updated projects are checked in to a version control repository.  To
mitigate this problem, this module chooses identifiers in a more deterministic
way, by hashing a description of each object as well as its parent and ancestor
objects.  This strategy should result in minimal "shift" in IDs as successive
generations of project files are produced.

THIS MODULE

This module introduces several classes, all derived from the XCObject class.
Nearly all of the "brains" are built into the XCObject class, which understands
how to create and modify objects, maintain the proper tree structure, compute
identifiers, and print objects.  For the most part, classes derived from
XCObject need only provide a _schema class object, a dictionary that
expresses what properties objects of the class may contain.

Given this structure, it's possible to build a minimal project file by creating
objects of the appropriate types and making the proper connections:

  config_list = XCConfigurationList()
  group = PBXGroup()
  project = PBXProject({'buildConfigurationList': config_list,
                        'mainGroup': group})

With the project object set up, it can be added to an XCProjectFile object.
XCProjectFile is a pseudo-class in the sense that it is a concrete XCObject
subclass that does not actually correspond to a class type found in a project
file.  Rather, it is used to represent the project file's root dictionary.
Printing an XCProjectFile will print the entire project file, including the
full "objects" dictionary.

  project_file = XCProjectFile({'rootObject': project})
  project_file.ComputeIDs()
  project_file.Print()

Xcode project files are always encoded in UTF-8.  This module will accept
strings of either the str class or the unicode class.  Strings of class str
are assumed to already be encoded in UTF-8.  Obviously, if you're just using
ASCII, you won't encounter difficulties because ASCII is a UTF-8 subset.
Strings of class unicode are handled properly and encoded in UTF-8 when
a project file is output.
"""

import hashlib
import os.path
import re
import string
import sys


# See XCObject._EncodeString.  This pattern is used to determine when a string
# can be printed unquoted.  Strings that match this pattern may be printed
# unquoted.  Strings that do not match must be quoted and may be further
# transformed to be properly encoded.  Note that this expression matches the
# characters listed with "+", for 1 or more occurrences: if a string is empty,
# it must not match this pattern, because it needs to be encoded as "".
unquoted = re.compile('^[' + re.escape('ABCDEFGHIJKLMNOPQRSTUVWXYZ' +
                                       'abcdefghijklmnopqrstuvwxyz' +
                                       '0123456789$./_') + ']+$')


class XCObject(object):
  """The abstract base of all class types used in Xcode project files.

  Class variables:
    _schema: A dictionary defining the properties of this class.  The keys to
             _schema are string property keys as used in project files.  Values
             are a list of four or five elements:
             [ is_list, property_type, is_strong, is_required, default ]
             is_list: True if the property described is a list, as opposed
                      to a single element.
             property_type: The type to use as the value of the property,
                            or if is_list is True, the type to use for each
                            element of the value's list.  property_type must
                            be an XCObject subclass, or one of the built-in
                            types str, int, or dict.
             is_strong: If property_type is an XCObject subclass, is_strong
                        is True to assert that this class "owns," or serves
                        as parent, to the property value (or, if is_list is
                        True, values).  is_strong must be False if<commit_msg>Fix RE from r122.<commit_after>#!/usr/bin/python

"""Xcode project file generator.

This module is both an Xcode project file generator and a documentation of the
Xcode project file format.  Knowledge of the project file format was gained
based on extensive experience with Xcode, and by making changes to projects in
Xcode.app and observing the resultant changes in the associated project files.

XCODE PROJECT FILES

The generator targets the file format as written by Xcode 3.1 (specifically,
3.1.2), but past experience has taught that the format has not changed
significantly in the past several years, and future versions of Xcode are able
to read older project files.

Xcode project files are "bundled": the project "file" from an end-user's
perspective is actually a directory with an ".xcodeproj" extension.  The
project file from this module's perspective is actually a file inside this
directory, always named "project.pbxproj".  This file contains a complete
description of the project and is all that is needed to use the xcodeproj.
Other files contained in the xcodeproj directory are simply used to store
per-user settings, such as the state of various UI elements in the Xcode
application.

The project.pbxproj file is a property list, stored in a format almost
identical to the NeXTstep property list format.  The file is able to carry
Unicode data, and is encoded in UTF-8.  The root element in the property list
is a dictionary that contains several properties of minimal interest, and two
properties of immense interest.  The most important property is a dictionary
named "objects".  The entire structure of the project is represented by the
children of this property.  The objects dictionary is keyed by unique 96-bit
values represented by 24 uppercase hexadecimal characters.  Each value in the
objects dictionary is itself a dictionary, describing an individual object.

Each object in the dictionary is a member of a class, which is identified by
the "isa" property of each object.  A variety of classes are represented in a
project file.  Objects can refer to other objects by ID, using the 24-character
hexadecimal object key.  A project's objects form a tree, with a root object
of class PBXProject at the root.  As an example, the PBXProject object serves
as parent to an XCConfigurationList object defining the build configurations
used in the project, a PBXGroup object serving as a container for all files
referenced in the project, and a list of target objects, each of which defines
a target in the project.  There are several different types of target object,
such as PBXNativeTarget and PBXAggregateTarget.  In this module, this
relationship is expressed by having each target type derive from an abstract
base named XCTarget.

The project.pbxproj file's root dictionary also contains a property, sibling to
the "objects" dictionary, named "rootObject".  The value of rootObject is a
24-character object key referring to the root PBXProject object in the
objects dictionary.

In Xcode, every file used as input to a target or produced as a final product
of a target must appear somewhere in the hierarchy rooted at the PBXGroup
object referenced by the PBXProject's mainGroup property.  A PBXGroup is
generally represented as a folder in the Xcode application.  PBXGroups can
contain other PBXGroups as well as PBXFileReferences, which are pointers to
actual files.

Each XCTarget contains a list of build phases, represented in this module by
the abstract base XCBuildPhase.  Examples of concrete XCBuildPhase derivations
are PBXSourcesBuildPhase and PBXFrameworksBuildPhase, which correspond to the
"Compile Sources" and "Link Binary With Libraries" phases displayed in the
Xcode application.  Files used as input to these phases (for example, source
files in the former case and libraries and frameworks in the latter) are
represented by PBXBuildFile objects, referenced by elements of "files" lists
in XCTarget objects.  Each PBXBuildFile object refers to a PBXBuildFile
object as a "weak" reference: it does not "own" the PBXBuildFile, which is
owned by the root object's mainGroup or a descendant group.  In most cases, the
layer of indirection between an XCBuildPhase and a PBXFileReference via a
PBXBuildFile appears extraneous, but there's actually one reason for this:
file-specific compiler flags are added to the PBXBuildFile object so as to
allow a single file to be a member of multiple targets while having distinct
compiler flags for each.  These flags can be modified in the Xcode applciation
in the "Build" tab of a File Info window.

When a project is open in the Xcode application, Xcode will rewrite it.  As
such, this module is careful to adhere to the formatting used by Xcode, to
avoid insignificant changes appearing in the file when it is used in the
Xcode application.  This will keep version control repositories happy, and
makes it possible to compare a project file used in Xcode to one generated by
this module to determine if any significant changes were made in the
application.

Xcode has its own way of assigning 24-character identifiers to each object,
which is not duplicated here.  Because the identifier only is only generated
once, when an object is created, and is then left unchanged, there is no need
to attempt to duplicate Xcode's behavior in this area.  The generator is free
to select any identifier, even at random, to refer to the objects it creates,
and Xcode will retain those identifiers and use them when subsequently
rewriting the project file.  However, the generator would choose new random
identifiers each time the project files are generated, leading to difficulties
comparing "used" project files to "pristine" ones produced by this module,
and causing the appearance of changes as every object identifier is changed
when updated projects are checked in to a version control repository.  To
mitigate this problem, this module chooses identifiers in a more deterministic
way, by hashing a description of each object as well as its parent and ancestor
objects.  This strategy should result in minimal "shift" in IDs as successive
generations of project files are produced.

THIS MODULE

This module introduces several classes, all derived from the XCObject class.
Nearly all of the "brains" are built into the XCObject class, which understands
how to create and modify objects, maintain the proper tree structure, compute
identifiers, and print objects.  For the most part, classes derived from
XCObject need only provide a _schema class object, a dictionary that
expresses what properties objects of the class may contain.

Given this structure, it's possible to build a minimal project file by creating
objects of the appropriate types and making the proper connections:

  config_list = XCConfigurationList()
  group = PBXGroup()
  project = PBXProject({'buildConfigurationList': config_list,
                        'mainGroup': group})

With the project object set up, it can be added to an XCProjectFile object.
XCProjectFile is a pseudo-class in the sense that it is a concrete XCObject
subclass that does not actually correspond to a class type found in a project
file.  Rather, it is used to represent the project file's root dictionary.
Printing an XCProjectFile will print the entire project file, including the
full "objects" dictionary.

  project_file = XCProjectFile({'rootObject': project})
  project_file.ComputeIDs()
  project_file.Print()

Xcode project files are always encoded in UTF-8.  This module will accept
strings of either the str class or the unicode class.  Strings of class str
are assumed to already be encoded in UTF-8.  Obviously, if you're just using
ASCII, you won't encounter difficulties because ASCII is a UTF-8 subset.
Strings of class unicode are handled properly and encoded in UTF-8 when
a project file is output.
"""

import hashlib
import os.path
import re
import string
import sys


# See XCObject._EncodeString.  This pattern is used to determine when a string
# can be printed unquoted.  Strings that match this pattern may be printed
# unquoted.  Strings that do not match must be quoted and may be further
# transformed to be properly encoded.  Note that this expression matches the
# characters listed with "+", for 1 or more occurrences: if a string is empty,
# it must not match this pattern, because it needs to be encoded as "".
unquoted = re.compile('^[A-Za-z0-9$./_]+$')


class XCObject(object):
  """The abstract base of all class types used in Xcode project files.

  Class variables:
    _schema: A dictionary defining the properties of this class.  The keys to
             _schema are string property keys as used in project files.  Values
             are a list of four or five elements:
             [ is_list, property_type, is_strong, is_required, default ]
             is_list: True if the property described is a list, as opposed
                      to a single element.
             property_type: The type to use as the value of the property,
                            or if is_list is True, the type to use for each
                            element of the value's list.  property_type must
                            be an XCObject subclass, or one of the built-in
                            types str, int, or dict.
             is_strong: If property_type is an XCObject subclass, is_strong
                        is True to assert that this class "owns," or serves
                        as parent, to the property value (or, if is_list is
                        True, values).  is_strong must be False if
{'fraction_non_alphanumeric': 0.03367473618870267, 'fraction_numerical': 0.0029484792054624456, 'mean_word_length': 3.8441493360060135, 'pattern_counts': {'":': 2, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2855949', 'n_tokens_mistral': 4805, 'n_tokens_neox': 4514, 'n_words': 2883}