c8d: Just enough Windows support to run the test suite
<!--
Please make sure you've read and understood our contributing guidelines;
https://github.com/moby/moby/blob/master/CONTRIBUTING.md

** Make sure all your commits include a signature generated with `git commit -s` **

For additional information on our contributing process, read our contributing
guide https://docs.docker.com/opensource/code/

If this is a bug fix, make sure your description includes "fixes #xxxx", or
"closes #xxxx"

Please provide the following information:
-->

**- What I did**

Building upon ~~#46402~~ #46572 to bring the containerd image-store support on Windows up to something that is worth having tests enabled for.

**- How I did it**

Firstly, cherry-picked #46402 onto the mainline branch, once containerd 1.7.6 had been vendored by #45966. Update: Currently based directly on #46402 until it merges, unless something important lands in mainline first.

Secondly, got the snapshotter tests functional on Windows matching the Linux version. (Two-liner, most of the work was done by @rumpl in #46402).

~~Violently beat my head against~~ Iteratively worked through failures observed in the integration test suite setup, building the busybox image. As it happens, this step does most of the things we seem to care about, so it was a good test-case.

Rebased the results of the above after #46572 landed, which integrated the outcomes of the second paragraph already.

Remaining TODOs to undraft this PR:

- [x] Label this PR with `containerd-integration` (someone needs to do that for me) and drop the last (0d5edcdca184c8bd3c50b8b25d75544fa9e7d190) commit
- [x] Rebase once ~~#46402~~ #46572 is merged, dropping the first couple of commits
- [x] Make and implement a decision about hack 1 (53777db68e5172cc50568a21959dff704565110d)
- [x] Make and implement a decision about hack 2 (https://github.com/moby/moby/pull/46539/commits/6e48f48df593fb689bb26894a36949632f61bd24 or [`b099f42`](https://github.com/moby/moby/pull/46539/files/53777db68e5172cc50568a21959dff704565110d..b099f42423c30eeaa26c77de32b0e169103c9e82))

----

*- Hack 1 (53777db68e5172cc50568a21959dff704565110d)*

builder-next specifies the `host` network provider, but [that fails on Windows](https://github.com/moby/buildkit/blob/v0.12.2/util/network/netproviders/network_windows.go#L11-L13), blocking daemon startup even though we don't currently support builder-next.

My current hack is to use `auto`, since that means [`cni` if configured](https://github.com/moby/buildkit/blob/v0.12.2/util/network/netproviders/network.go#L36-L46) and if not, [on Windows it falls back to `null`](https://github.com/moby/buildkit/blob/v0.12.2/util/network/netproviders/network_windows.go#L15-L18), and [on Linux it falls back to `host`](https://github.com/moby/buildkit/blob/v0.12.2/util/network/netproviders/network_unix.go#L15-L18).

I've currently only done this for Windows with a `runtime.GOOS` test, but I'm inclined to think we should use `auto` always, since: it does the right thing on Linux, and if we ever do need a more-complex network setup, CNI is the obvious path. If we ever do get builder-next working for Windows, it'll need to use CNI anyway, even for the equivalent of host networking.

----

*- Hack 2 (https://github.com/moby/moby/pull/46539/commits/6e48f48df593fb689bb26894a36949632f61bd24 or [`b099f42`](https://github.com/moby/moby/pull/46539/files/53777db68e5172cc50568a21959dff704565110d..b099f42423c30eeaa26c77de32b0e169103c9e82))*

The windowfilter graphdriver is a bit weird: When you "mount" an image from it, it actually [only does the volume setup (Prepare and Activate in HCS terms)](https://github.com/moby/moby/blob/51f0e7b0e49c709729d67559c9c07616c1da019b/daemon/graphdriver/windows/windows.go#L385-L407) and then [returns the `\\?\Volume{GUID}\` formatted volume name](https://github.com/moby/moby/blob/51f0e7b0e49c709729d67559c9c07616c1da019b/daemon/graphdriver/windows/windows.go#L414-L416), where other graphdrivers (and all the containerd snapshotters) actually [mount the image to a host directory](https://github.com/moby/moby/blob/51f0e7b0e49c709729d67559c9c07616c1da019b/daemon/graphdriver/overlay2/overlay.go#L596-L598) and [return that directory](https://github.com/moby/moby/blob/51f0e7b0e49c709729d67559c9c07616c1da019b/daemon/graphdriver/overlay2/overlay.go#L608).

However, starting a container as we do now [requires us to pass that `\\?\Volume{GUID}\`-formatted volume name in the OCI spec as `Root.Path`](https://github.com/opencontainers/runtime-spec/blob/main/config.md#root).

Normally, containerd doesn't expose that volume name to callers, but we can use the underlying Windows API to get the volume name from the mount point, which relies on [internal knowledge of the containerd/mount implementation](https://github.com/moby/moby/pull/46539/files#diff-6688f4342adf127b206582942bc147a0efab01c2e376b8a1a81e62c4bfee3ce1R296-R337) ([earlier implementation of the same idea that I'm less confident in](https://github.com/moby/moby/pull/46539/files/6e48f48df593fb689bb26894a36949632f61bd24#diff-6688f4342adf127b206582942bc147a0efab01c2e376b8a1a81e62c4bfee3ce1R296-R322)).

The BindFilter approach was pretty messy though, and I didn't like it very much. (There's probably a bunch of overlooked edge cases there too. I have no idea why I see four copies of the bind-mount there, which is another vote against the older BindFilter version.) The `GetLayerMountPath` isn't too bad, it's still poking at internal details, but they're details that containerd's mount package could trivially provide an API to expose, if we go this way.

Beyond cleaning it up, an alternative approach would be to not populate `Windows.LayerFolders` nor `Root.Path` at all (removing the need for 1ad566e21dee685a0c10b01054bd40c5de7a0e5b as well) and instead use [containerd's `WithSnapshot` and `WithSnapshotter` config-functor-things](https://github.com/containerd/containerd/blob/v1.7.6/container_opts.go#L178-L207) when setting up the container.

However, this would, at least on Windows, require us to unmount the directory before starting the container, as this flow makes hcsshim responsible for both [mounting of the root layer](https://github.com/microsoft/hcsshim/blob/v0.11.1/internal/hcsoci/resources_wcow.go#L60-L73) and [deriving `Windows.LayerFolders` from the snapshot](https://github.com/microsoft/hcsshim/blob/v0.11.1/cmd/containerd-shim-runhcs-v1/service_internal.go#L118-L158). Note that in the latter case, [there's a bug where hcsshim'll *append* to any `Windows.LayerFolders` passed in](https://github.com/opencontainers/runtime-spec/issues/1185#issuecomment-1474549056) ([0.12 onwards rejects this case explicitly instead](https://github.com/microsoft/hcsshim/blob/main/cmd/containerd-shim-runhcs-v1/rootfs.go#L15-L45)), so we *must not* populate it in this case. `nil` is fine, even though it's [out-of-spec](https://github.com/opencontainers/runtime-spec/issues/1185).

This should be _feasible_ as we already have the `conditionalMountOnStart` hook, but code flows after that is called need to be checked to ensure they are correctly handling the case of not mounting the container filesystem locally. For example, I noticed code on the Windows side that checks the same conditions as the current `conditionalMountOnStart` check, and if they match, does a temporary mount of the image; clearly there's some scope for refactoring that to not duplicate the condition around.

If it was just Windows, I'd have tried to go with this approach. However, I think it'd be cleaner if we use `WithSnapshot` for both Windows and Linux when executing containers. This would be a larger change, and probably has repercussions I'm unaware of at this time, since I haven't really worked to understand the overarching flow and its rationales. (Most other containerd-calling code [appears to use `WithNewSnapshot`](https://github.com/containerd/nerdctl/blob/v1.6.0/pkg/cmd/container/create.go#L316), but moby currently has a pile of stuff it wants to do to the image first, so I'm not considering that here.)

----

**- How to verify it**

CI.

This PR has the `containerd-integration` label applied, so the new snapshotter test suite introduced in ~~#46402~~ #46572 will run for this PR (and any PR with that label), giving us both Windows and Linux coverage. As long as the failures in the snapshotter test suite are consistent across platforms (and I didn't break anything), we're probably fine.

All of the changes made here (except an interface change while the compiler will validate) are only in the "use snapshotter" code-path, so verification should be low-risk.

If someone wants to take the time and try this out with the WASM runtime, that'd be awesome. I have no particular plans to do so, even though that's what started me on this piece of work, months ago.

**- Description for the changelog**
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

Containerd image store on Windows is slightly usable.

**- A picture of a cute animal (not mandatory but encouraged)**

![image](https://github.com/moby/moby/assets/138397/149b0bcc-f13b-4b02-be07-a49dcfbee82b)

daemon/oci_windows.go
package daemon // import "github.com/docker/docker/daemon"

import (
	"context"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
	"strings"

	coci "github.com/containerd/containerd/oci"
	"github.com/containerd/log"
	containertypes "github.com/docker/docker/api/types/container"
	imagetypes "github.com/docker/docker/api/types/image"
	"github.com/docker/docker/container"
	"github.com/docker/docker/daemon/config"
	"github.com/docker/docker/errdefs"
	"github.com/docker/docker/image"
	"github.com/docker/docker/oci"
	"github.com/docker/docker/pkg/sysinfo"
	"github.com/docker/docker/pkg/system"
	specs "github.com/opencontainers/runtime-spec/specs-go"
	"github.com/pkg/errors"
	"golang.org/x/sys/windows/registry"
)

const (
	credentialSpecRegistryLocation = `SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Containers\CredentialSpecs`
	credentialSpecFileLocation     = "CredentialSpecs"
)

func (daemon *Daemon) createSpec(ctx context.Context, daemonCfg *configStore, c *container.Container) (*specs.Spec, error) {
	img, err := daemon.imageService.GetImage(ctx, string(c.ImageID), imagetypes.GetImageOpts{})
	if err != nil {
		return nil, err
	}
	if err := image.CheckOS(img.OperatingSystem()); err != nil {
		return nil, err
	}

	s := oci.DefaultSpec()

	if err := coci.WithAnnotations(c.HostConfig.Annotations)(ctx, nil, nil, &s); err != nil {
		return nil, err
	}

	linkedEnv, err := daemon.setupLinkedContainers(c)
	if err != nil {
		return nil, err
	}

	// Note, unlike Unix, we do NOT call into SetupWorkingDirectory as
	// this is done in VMCompute. Further, we couldn't do it for Hyper-V
	// containers anyway.

	if err := daemon.setupSecretDir(c); err != nil {
		return nil, err
	}

	if err := daemon.setupConfigDir(c); err != nil {
		return nil, err
	}

	// In s.Mounts
	mounts, err := daemon.setupMounts(c)
	if err != nil {
		return nil, err
	}

	var isHyperV bool
	if c.HostConfig.Isolation.IsDefault() {
		// Container using default isolation, so take the default from the daemon configuration
		isHyperV = daemon.defaultIsolation.IsHyperV()
	} else {
		// Container may be requesting an explicit isolation mode.
		isHyperV = c.HostConfig.Isolation.IsHyperV()
	}

	if isHyperV {
		s.Windows.HyperV = &specs.WindowsHyperV{}
	}

	// If the container has not been started, and has configs or secrets
	// secrets, create symlinks to each config and secret. If it has been
	// started before, the symlinks should have already been created. Also, it
	// is important to not mount a Hyper-V  container that has been started
	// before, to protect the host from the container; for example, from
	// malicious mutation of NTFS data structures.
	if !c.HasBeenStartedBefore && (len(c.SecretReferences) > 0 || len(c.ConfigReferences) > 0) {
		// The container file system is mounted before this function is called,
		// except for Hyper-V containers, so mount it here in that case.
		if isHyperV {
			if err := daemon.Mount(c); err != nil {
				return nil, err
			}
			defer daemon.Unmount(c)
		}
		if err := c.CreateSecretSymlinks(); err != nil {
			return nil, err
		}
		if err := c.CreateConfigSymlinks(); err != nil {
			return nil, err
		}
	}

	secretMounts, err := c.SecretMounts()
	if err != nil {
		return nil, err
	}
	if secretMounts != nil {
		mounts = append(mounts, secretMounts...)
	}

	if configMounts := c.ConfigMounts(); configMounts != nil {
		mounts = append(mounts, configMounts...)
	}

	for _, mount := range mounts {
		m := specs.Mount{
			Source:      mount.Source,
			Destination: mount.Destination,
		}
		if !mount.Writable {
			m.Options = append(m.Options, "ro")
		}
		s.Mounts = append(s.Mounts, m)
	}

	// In s.Process
	s.Process.Cwd = c.Config.WorkingDir
	s.Process.Env = c.CreateDaemonEnvironment(c.Config.Tty, linkedEnv)
	s.Process.Terminal = c.Config.Tty

	if c.Config.Tty {
		s.Process.ConsoleSize = &specs.Box{
			Height: c.HostConfig.ConsoleSize[0],
			Width:  c.HostConfig.ConsoleSize[1],
		}
	}
	s.Process.User.Username = c.Config.User
	s.Windows.LayerFolders, err = daemon.imageService.GetLayerFolders(img, c.RWLayer)
	if err != nil {
		return nil, errors.Wrapf(err, "container %s", c.ID)
	}

	// Get endpoints for the libnetwork allocated networks to the container
	var epList []string
	AllowUnqualifiedDNSQuery := false
	gwHNSID := ""
	if c.NetworkSettings != nil {
		for n := range c.NetworkSettings.Networks {
			sn, err := daemon.FindNetwork(n)
			if err != nil {
				continue
			}

			ep, err := getEndpointInNetwork(c.Name, sn)
			if err != nil {
				continue
			}

			data, err := ep.DriverInfo()
			if err != nil {
				continue
			}

			if data["GW_INFO"] != nil {
				gwInfo := data["GW_INFO"].(map[string]interface{})
				if gwInfo["hnsid"] != nil {
					gwHNSID = gwInfo["hnsid"].(string)
				}
			}

			if data["hnsid"] != nil {
				epList = append(epList, data["hnsid"].(string))
			}

			if data["AllowUnqualifiedDNSQuery"] != nil {
				AllowUnqualifiedDNSQuery = true
			}
		}
	}

	var networkSharedContainerID string
	if c.HostConfig.NetworkMode.IsContainer() {
		networkSharedContainerID = c.NetworkSharedContainerID
		for _, ep := range c.SharedEndpointList {
			epList = append(epList, ep)
		}
	}

	if gwHNSID != "" {
		epList = append(epList, gwHNSID)
	}

	var dnsSearch []string
	if len(c.HostConfig.DNSSearch) > 0 {
		dnsSearch = c.HostConfig.DNSSearch
	} else if len(daemonCfg.DNSSearch) > 0 {
		dnsSearch = daemonCfg.DNSSearch
	}

	s.Windows.Network = &specs.WindowsNetwork{
		AllowUnqualifiedDNSQuery:   AllowUnqualifiedDNSQuery,
		DNSSearchList:              dnsSearch,
		EndpointList:               epList,
		NetworkSharedContainerName: networkSharedContainerID,
	}

	if err := daemon.createSpecWindowsFields(c, &s, isHyperV); err != nil {
		return nil, err
	}

	if log.G(ctx).Level >= log.DebugLevel {
		if b, err := json.Marshal(&s); err == nil {
			log.G(ctx).Debugf("Generated spec: %s", string(b))
		}
	}

	return &s, nil
}

// Sets the Windows-specific fields of the OCI spec
func (daemon *Daemon) createSpecWindowsFields(c *container.Container, s *specs.Spec, isHyperV bool) error {
	s.Hostname = c.FullHostname()

	if len(s.Process.Cwd) == 0 {
		// We default to C:\ to workaround the oddity of the case that the
		// default directory for cmd running as LocalSystem (or
		// ContainerAdministrator) is c:\windows\system32. Hence docker run
		// <image> cmd will by default end in c:\windows\system32, rather
		// than 'root' (/) on Linux. The oddity is that if you have a dockerfile
		// which has no WORKDIR and has a COPY file ., . will be interpreted
		// as c:\. Hence, setting it to default of c:\ makes for consistency.
		s.Process.Cwd = `C:\`
	}

	if c.Config.ArgsEscaped {
		s.Process.CommandLine = c.Path
		if len(c.Args) > 0 {
			s.Process.CommandLine += " " + system.EscapeArgs(c.Args)
		}
	} else {
		s.Process.Args = append([]string{c.Path}, c.Args...)
	}
	s.Root.Readonly = false // Windows does not support a read-only root filesystem
	if !isHyperV {
		if c.BaseFS == "" {
			return errors.New("createSpecWindowsFields: BaseFS of container " + c.ID + " is unexpectedly empty")
		}

		s.Root.Path = c.BaseFS // This is not set for Hyper-V containers
		if !strings.HasSuffix(s.Root.Path, `\`) {
			s.Root.Path = s.Root.Path + `\` // Ensure a correctly formatted volume GUID path \\?\Volume{GUID}\
		}
	}

	// First boot optimization
	s.Windows.IgnoreFlushesDuringBoot = !c.HasBeenStartedBefore

	setResourcesInSpec(c, s, isHyperV)

	// Read and add credentials from the security options if a credential spec has been provided.
	if err := daemon.setWindowsCredentialSpec(c, s); err != nil {
		return err
	}

	devices, err := setupWindowsDevices(c.HostConfig.Devices)
	if err != nil {
		return err
	}

	s.Windows.Devices = append(s.Windows.Devices, devices...)

	return nil
}

var errInvalidCredentialSpecSecOpt = errdefs.InvalidParameter(fmt.Errorf("invalid credential spec security option - value must be prefixed by 'file://', 'registry://', or 'raw://' followed by a non-empty value"))

// setWindowsCredentialSpec sets the spec's `Windows.CredentialSpec`
// field if relevant
func (daemon *Daemon) setWindowsCredentialSpec(c *container.Container, s *specs.Spec) error {
	if c.HostConfig == nil || c.HostConfig.SecurityOpt == nil {
		return nil
	}

	// TODO (jrouge/wk8): if provided with several security options, we silently ignore
	// all but the last one (provided they're all valid, otherwise we do return an error);
	// this doesn't seem like a great idea?
	credentialSpec := ""

	// TODO(thaJeztah): extract validating and parsing SecurityOpt to a reusable function.
	for _, secOpt := range c.HostConfig.SecurityOpt {
		k, v, ok := strings.Cut(secOpt, "=")
		if !ok {
			return errdefs.InvalidParameter(fmt.Errorf("invalid security option: no equals sign in supplied value %s", secOpt))
		}
		// FIXME(thaJeztah): options should not be case-insensitive
		if !strings.EqualFold(k, "credentialspec") {
			return errdefs.InvalidParameter(fmt.Errorf("security option not supported: %s", k))
		}

		scheme, value, ok := strings.Cut(v, "://")
		if !ok || value == "" {
			return errInvalidCredentialSpecSecOpt
		}
		var err error
		switch strings.ToLower(scheme) {
		case "file":
			credentialSpec, err = readCredentialSpecFile(c.ID, daemon.root, filepath.Clean(value))
			if err != nil {
				return errdefs.InvalidParameter(err)
			}
		case "registry":
			credentialSpec, err = readCredentialSpecRegistry(c.ID, value)
			if err != nil {
				return errdefs.InvalidParameter(err)
			}
		case "config":
			// if the container does not have a DependencyStore, then it
			// isn't swarmkit managed. In order to avoid creating any
			// impression that `config://` is a valid API, return the same
			// error as if you'd passed any other random word.
			if c.DependencyStore == nil {
				return errInvalidCredentialSpecSecOpt
			}

			csConfig, err := c.DependencyStore.Configs().Get(value)
			if err != nil {
				return errdefs.System(errors.Wrap(err, "error getting value from config store"))
			}
			// stuff the resulting secret data into a string to use as the
			// CredentialSpec
			credentialSpec = string(csConfig.Spec.Data)
		case "raw":
			credentialSpec = value
		default:
			return errInvalidCredentialSpecSecOpt
		}
	}

	if credentialSpec != "" {
		if s.Windows == nil {
			s.Windows = &specs.Windows{}
		}
		s.Windows.CredentialSpec = credentialSpec
	}

	return nil
}

func setResourcesInSpec(c *container.Container, s *specs.Spec, isHyperV bool) {
	// In s.Windows.Resources
	cpuShares := uint16(c.HostConfig.CPUShares)
	cpuMaximum := uint16(c.HostConfig.CPUPercent) * 100
	cpuCount := uint64(c.HostConfig.CPUCount)
	if c.HostConfig.NanoCPUs > 0 {
		if isHyperV {
			cpuCount = uint64(c.HostConfig.NanoCPUs / 1e9)
			leftoverNanoCPUs := c.HostConfig.NanoCPUs % 1e9
			if leftoverNanoCPUs != 0 {
				cpuCount++
				cpuMaximum = uint16(c.HostConfig.NanoCPUs / int64(cpuCount) / (1e9 / 10000))
				if cpuMaximum < 1 {
					// The requested NanoCPUs is so small that we rounded to 0, use 1 instead
					cpuMaximum = 1
				}
			}
		} else {
			cpuMaximum = uint16(c.HostConfig.NanoCPUs / int64(sysinfo.NumCPU()) / (1e9 / 10000))
			if cpuMaximum < 1 {
				// The requested NanoCPUs is so small that we rounded to 0, use 1 instead
				cpuMaximum = 1
			}
		}
	}

	if cpuMaximum != 0 || cpuShares != 0 || cpuCount != 0 {
		if s.Windows.Resources == nil {
			s.Windows.Resources = &specs.WindowsResources{}
		}
		s.Windows.Resources.CPU = &specs.WindowsCPUResources{
			Maximum: &cpuMaximum,
			Shares:  &cpuShares,
			Count:   &cpuCount,
		}
	}

	memoryLimit := uint64(c.HostConfig.Memory)
	if memoryLimit != 0 {
		if s.Windows.Resources == nil {
			s.Windows.Resources = &specs.WindowsResources{}
		}
		s.Windows.Resources.Memory = &specs.WindowsMemoryResources{
			Limit: &memoryLimit,
		}
	}

	if c.HostConfig.IOMaximumBandwidth != 0 || c.HostConfig.IOMaximumIOps != 0 {
		if s.Windows.Resources == nil {
			s.Windows.Resources = &specs.WindowsResources{}
		}
		s.Windows.Resources.Storage = &specs.WindowsStorageResources{
			Bps:  &c.HostConfig.IOMaximumBandwidth,
			Iops: &c.HostConfig.IOMaximumIOps,
		}
	}
}

// mergeUlimits merge the Ulimits from HostConfig with daemon defaults, and update HostConfig
// It will do nothing on non-Linux platform
func (daemon *Daemon) mergeUlimits(c *containertypes.HostConfig, daemonCfg *config.Config) {
	return
}

// registryKey is an interface wrapper around `registry.Key`,
// listing only the methods we care about here.
// It's mainly useful to easily allow mocking the registry in tests.
type registryKey interface {
	GetStringValue(name string) (val string, valtype uint32, err error)
	Close() error
}

var registryOpenKeyFunc = func(baseKey registry.Key, path string, access uint32) (registryKey, error) {
	return registry.OpenKey(baseKey, path, access)
}

// readCredentialSpecRegistry is a helper function to read a credential spec from
// the registry. If not found, we return an empty string and warn in the log.
// This allows for staging on machines which do not have the necessary components.
func readCredentialSpecRegistry(id, name string) (string, error) {
	key, err := registryOpenKeyFunc(registry.LOCAL_MACHINE, credentialSpecRegistryLocation, registry.QUERY_VALUE)
	if err != nil {
		return "", errors.Wrapf(err, "failed handling spec %q for container %s - registry key %s could not be opened", name, id, credentialSpecRegistryLocation)
	}
	defer key.Close()

	value, _, err := key.GetStringValue(name)
	if err != nil {
		if err == registry.ErrNotExist {
			return "", fmt.Errorf("registry credential spec %q for container %s was not found", name, id)
		}
		return "", errors.Wrapf(err, "error reading credential spec %q from registry for container %s", name, id)
	}

	return value, nil
}

// readCredentialSpecFile is a helper function to read a credential spec from
// a file. If not found, we return an empty string and warn in the log.
// This allows for staging on machines which do not have the necessary components.
func readCredentialSpecFile(id, root, location string) (string, error) {
	if filepath.IsAbs(location) {
		return "", fmt.Errorf("invalid credential spec: file:// path cannot be absolute")
	}
	base := filepath.Join(root, credentialSpecFileLocation)
	full := filepath.Join(base, location)
	if !strings.HasPrefix(full, base) {
		return "", fmt.Errorf("invalid credential spec: file:// path must be under %s", base)
	}
	bcontents, err := os.ReadFile(full)
	if err != nil {
		return "", errors.Wrapf(err, "failed to load credential spec for container %s", id)
	}
	return string(bcontents[:]), nil
}

func setupWindowsDevices(devices []containertypes.DeviceMapping) (specDevices []specs.WindowsDevice, err error) {
	for _, deviceMapping := range devices {
		if strings.HasPrefix(deviceMapping.PathOnHost, "class/") {
			specDevices = append(specDevices, specs.WindowsDevice{
				ID:     strings.TrimPrefix(deviceMapping.PathOnHost, "class/"),
				IDType: "class",
			})
		} else {
			idType, id, ok := strings.Cut(deviceMapping.PathOnHost, "://")
			if !ok {
				return nil, errors.Errorf("invalid device assignment path: '%s', must be 'class/ID' or 'IDType://ID'", deviceMapping.PathOnHost)
			}
			if idType == "" {
				return nil, errors.Errorf("invalid device assignment path: '%s', IDType cannot be empty", deviceMapping.PathOnHost)
			}
			specDevices = append(specDevices, specs.WindowsDevice{
				ID:     id,
				IDType: idType,
			})
		}
	}

	return specDevices, nil
}

package daemon // import "github.com/docker/docker/daemon"

import (
	"context"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
	"strings"

	"github.com/Microsoft/hcsshim"
	coci "github.com/containerd/containerd/oci"
	"github.com/containerd/log"
	containertypes "github.com/docker/docker/api/types/container"
	imagetypes "github.com/docker/docker/api/types/image"
	"github.com/docker/docker/container"
	"github.com/docker/docker/daemon/config"
	"github.com/docker/docker/errdefs"
	"github.com/docker/docker/image"
	"github.com/docker/docker/oci"
	"github.com/docker/docker/pkg/sysinfo"
	"github.com/docker/docker/pkg/system"
	specs "github.com/opencontainers/runtime-spec/specs-go"
	"github.com/pkg/errors"
	"golang.org/x/sys/windows/registry"
)

const (
	credentialSpecRegistryLocation = `SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Containers\CredentialSpecs`
	credentialSpecFileLocation     = "CredentialSpecs"
)

func (daemon *Daemon) createSpec(ctx context.Context, daemonCfg *configStore, c *container.Container) (*specs.Spec, error) {
	img, err := daemon.imageService.GetImage(ctx, string(c.ImageID), imagetypes.GetImageOpts{})
	if err != nil {
		return nil, err
	}
	if err := image.CheckOS(img.OperatingSystem()); err != nil {
		return nil, err
	}

	s := oci.DefaultSpec()

	if err := coci.WithAnnotations(c.HostConfig.Annotations)(ctx, nil, nil, &s); err != nil {
		return nil, err
	}

	linkedEnv, err := daemon.setupLinkedContainers(c)
	if err != nil {
		return nil, err
	}

	// Note, unlike Unix, we do NOT call into SetupWorkingDirectory as
	// this is done in VMCompute. Further, we couldn't do it for Hyper-V
	// containers anyway.

	if err := daemon.setupSecretDir(c); err != nil {
		return nil, err
	}

	if err := daemon.setupConfigDir(c); err != nil {
		return nil, err
	}

	// In s.Mounts
	mounts, err := daemon.setupMounts(c)
	if err != nil {
		return nil, err
	}

	var isHyperV bool
	if c.HostConfig.Isolation.IsDefault() {
		// Container using default isolation, so take the default from the daemon configuration
		isHyperV = daemon.defaultIsolation.IsHyperV()
	} else {
		// Container may be requesting an explicit isolation mode.
		isHyperV = c.HostConfig.Isolation.IsHyperV()
	}

	if isHyperV {
		s.Windows.HyperV = &specs.WindowsHyperV{}
	}

	// If the container has not been started, and has configs or secrets
	// secrets, create symlinks to each config and secret. If it has been
	// started before, the symlinks should have already been created. Also, it
	// is important to not mount a Hyper-V  container that has been started
	// before, to protect the host from the container; for example, from
	// malicious mutation of NTFS data structures.
	if !c.HasBeenStartedBefore && (len(c.SecretReferences) > 0 || len(c.ConfigReferences) > 0) {
		// The container file system is mounted before this function is called,
		// except for Hyper-V containers, so mount it here in that case.
		if isHyperV {
			if err := daemon.Mount(c); err != nil {
				return nil, err
			}
			defer daemon.Unmount(c)
		}
		if err := c.CreateSecretSymlinks(); err != nil {
			return nil, err
		}
		if err := c.CreateConfigSymlinks(); err != nil {
			return nil, err
		}
	}

	secretMounts, err := c.SecretMounts()
	if err != nil {
		return nil, err
	}
	if secretMounts != nil {
		mounts = append(mounts, secretMounts...)
	}

	if configMounts := c.ConfigMounts(); configMounts != nil {
		mounts = append(mounts, configMounts...)
	}

	for _, mount := range mounts {
		m := specs.Mount{
			Source:      mount.Source,
			Destination: mount.Destination,
		}
		if !mount.Writable {
			m.Options = append(m.Options, "ro")
		}
		s.Mounts = append(s.Mounts, m)
	}

	// In s.Process
	s.Process.Cwd = c.Config.WorkingDir
	s.Process.Env = c.CreateDaemonEnvironment(c.Config.Tty, linkedEnv)
	s.Process.Terminal = c.Config.Tty

	if c.Config.Tty {
		s.Process.ConsoleSize = &specs.Box{
			Height: c.HostConfig.ConsoleSize[0],
			Width:  c.HostConfig.ConsoleSize[1],
		}
	}
	s.Process.User.Username = c.Config.User
	s.Windows.LayerFolders, err = daemon.imageService.GetLayerFolders(img, c.RWLayer, c.ID)
	if err != nil {
		return nil, errors.Wrapf(err, "GetLayerFolders failed: container %s", c.ID)
	}

	// Get endpoints for the libnetwork allocated networks to the container
	var epList []string
	AllowUnqualifiedDNSQuery := false
	gwHNSID := ""
	if c.NetworkSettings != nil {
		for n := range c.NetworkSettings.Networks {
			sn, err := daemon.FindNetwork(n)
			if err != nil {
				continue
			}

			ep, err := getEndpointInNetwork(c.Name, sn)
			if err != nil {
				continue
			}

			data, err := ep.DriverInfo()
			if err != nil {
				continue
			}

			if data["GW_INFO"] != nil {
				gwInfo := data["GW_INFO"].(map[string]interface{})
				if gwInfo["hnsid"] != nil {
					gwHNSID = gwInfo["hnsid"].(string)
				}
			}

			if data["hnsid"] != nil {
				epList = append(epList, data["hnsid"].(string))
			}

			if data["AllowUnqualifiedDNSQuery"] != nil {
				AllowUnqualifiedDNSQuery = true
			}
		}
	}

	var networkSharedContainerID string
	if c.HostConfig.NetworkMode.IsContainer() {
		networkSharedContainerID = c.NetworkSharedContainerID
		for _, ep := range c.SharedEndpointList {
			epList = append(epList, ep)
		}
	}

	if gwHNSID != "" {
		epList = append(epList, gwHNSID)
	}

	var dnsSearch []string
	if len(c.HostConfig.DNSSearch) > 0 {
		dnsSearch = c.HostConfig.DNSSearch
	} else if len(daemonCfg.DNSSearch) > 0 {
		dnsSearch = daemonCfg.DNSSearch
	}

	s.Windows.Network = &specs.WindowsNetwork{
		AllowUnqualifiedDNSQuery:   AllowUnqualifiedDNSQuery,
		DNSSearchList:              dnsSearch,
		EndpointList:               epList,
		NetworkSharedContainerName: networkSharedContainerID,
	}

	if err := daemon.createSpecWindowsFields(c, &s, isHyperV); err != nil {
		return nil, err
	}

	if log.G(ctx).Level >= log.DebugLevel {
		if b, err := json.Marshal(&s); err == nil {
			log.G(ctx).Debugf("Generated spec: %s", string(b))
		}
	}

	return &s, nil
}

// Sets the Windows-specific fields of the OCI spec
func (daemon *Daemon) createSpecWindowsFields(c *container.Container, s *specs.Spec, isHyperV bool) error {
	s.Hostname = c.FullHostname()

	if len(s.Process.Cwd) == 0 {
		// We default to C:\ to workaround the oddity of the case that the
		// default directory for cmd running as LocalSystem (or
		// ContainerAdministrator) is c:\windows\system32. Hence docker run
		// <image> cmd will by default end in c:\windows\system32, rather
		// than 'root' (/) on Linux. The oddity is that if you have a dockerfile
		// which has no WORKDIR and has a COPY file ., . will be interpreted
		// as c:\. Hence, setting it to default of c:\ makes for consistency.
		s.Process.Cwd = `C:\`
	}

	if c.Config.ArgsEscaped {
		s.Process.CommandLine = c.Path
		if len(c.Args) > 0 {
			s.Process.CommandLine += " " + system.EscapeArgs(c.Args)
		}
	} else {
		s.Process.Args = append([]string{c.Path}, c.Args...)
	}
	s.Root.Readonly = false // Windows does not support a read-only root filesystem
	if !isHyperV {
		if c.BaseFS == "" {
			return errors.New("createSpecWindowsFields: BaseFS of container " + c.ID + " is unexpectedly empty")
		}

		if daemon.UsesSnapshotter() {
			// daemon.Mount() for the snapshotters actually mounts the filesystem to the host
			// using containerd/mount.All and BaseFS is the directory where this is mounted.
			// This is consistent with Linux-based graphdriver implementations.
			// For the windowsfilter graphdriver, the underlying Get() call does not actually mount
			// the filesystem to a path, and BaseFS is the Volume GUID of the prepared/activated
			// filesystem.

			// The spec for Root.Path for Windows specifies that for Process-isolated containers,
			// it must be in the Volume GUID (\\?\\Volume{GUID} style), not a host-mounted directory.
			backingDevicePath, err := getBackingDeviceForContainerdMount(c.BaseFS)
			if err != nil {
				return errors.Wrapf(err, "createSpecWindowsFields: Failed to get backing device of BaseFS of container %s", c.ID)
			}
			s.Root.Path = backingDevicePath
		} else {
			s.Root.Path = c.BaseFS // This is not set for Hyper-V containers
		}
		if !strings.HasSuffix(s.Root.Path, `\`) {
			s.Root.Path = s.Root.Path + `\` // Ensure a correctly formatted volume GUID path \\?\Volume{GUID}\
		}
	}

	// First boot optimization
	s.Windows.IgnoreFlushesDuringBoot = !c.HasBeenStartedBefore

	setResourcesInSpec(c, s, isHyperV)

	// Read and add credentials from the security options if a credential spec has been provided.
	if err := daemon.setWindowsCredentialSpec(c, s); err != nil {
		return err
	}

	devices, err := setupWindowsDevices(c.HostConfig.Devices)
	if err != nil {
		return err
	}

	s.Windows.Devices = append(s.Windows.Devices, devices...)

	return nil
}

// getBackingDeviceForContainerdMount extracts the backing device or directory mounted at mountPoint
// by containerd's mount.Mount implementation for Windows.
func getBackingDeviceForContainerdMount(mountPoint string) (string, error) {
	// NOTE: This relies on details of the behaviour of containerd's mount implementation for Windows,
	// and so is somewhat fragile.
	// TODO: Upstream this into the mount package.
	// The implementation would be the same, but it'll be better-encapsulated.

	// See containerd/containerd/mount/mount_windows.go
	// This is mostly just copied from mount.Unmount

	const sourceStreamName = "containerd.io-source"

	mountPoint = filepath.Clean(mountPoint)
	adsFile := mountPoint + ":" + sourceStreamName
	var layerPath string

	if _, err := os.Lstat(adsFile); err == nil {
		layerPathb, err := os.ReadFile(mountPoint + ":" + sourceStreamName)
		if err != nil {
			return "", fmt.Errorf("failed to retrieve layer source for mount %s: %w", mountPoint, err)
		}
		layerPath = string(layerPathb)
	}

	if layerPath == "" {
		return "", fmt.Errorf("no layer source for mount %s", mountPoint)
	}

	home, layerID := filepath.Split(layerPath)
	di := hcsshim.DriverInfo{
		HomeDir: home,
	}

	backingDevice, err := hcsshim.GetLayerMountPath(di, layerID)
	if err != nil {
		return "", fmt.Errorf("failed to retrieve backing device for layer %s: %w", mountPoint, err)
	}

	return backingDevice, nil
}

var errInvalidCredentialSpecSecOpt = errdefs.InvalidParameter(fmt.Errorf("invalid credential spec security option - value must be prefixed by 'file://', 'registry://', or 'raw://' followed by a non-empty value"))

// setWindowsCredentialSpec sets the spec's `Windows.CredentialSpec`
// field if relevant
func (daemon *Daemon) setWindowsCredentialSpec(c *container.Container, s *specs.Spec) error {
	if c.HostConfig == nil || c.HostConfig.SecurityOpt == nil {
		return nil
	}

	// TODO (jrouge/wk8): if provided with several security options, we silently ignore
	// all but the last one (provided they're all valid, otherwise we do return an error);
	// this doesn't seem like a great idea?
	credentialSpec := ""

	// TODO(thaJeztah): extract validating and parsing SecurityOpt to a reusable function.
	for _, secOpt := range c.HostConfig.SecurityOpt {
		k, v, ok := strings.Cut(secOpt, "=")
		if !ok {
			return errdefs.InvalidParameter(fmt.Errorf("invalid security option: no equals sign in supplied value %s", secOpt))
		}
		// FIXME(thaJeztah): options should not be case-insensitive
		if !strings.EqualFold(k, "credentialspec") {
			return errdefs.InvalidParameter(fmt.Errorf("security option not supported: %s", k))
		}

		scheme, value, ok := strings.Cut(v, "://")
		if !ok || value == "" {
			return errInvalidCredentialSpecSecOpt
		}
		var err error
		switch strings.ToLower(scheme) {
		case "file":
			credentialSpec, err = readCredentialSpecFile(c.ID, daemon.root, filepath.Clean(value))
			if err != nil {
				return errdefs.InvalidParameter(err)
			}
		case "registry":
			credentialSpec, err = readCredentialSpecRegistry(c.ID, value)
			if err != nil {
				return errdefs.InvalidParameter(err)
			}
		case "config":
			// if the container does not have a DependencyStore, then it
			// isn't swarmkit managed. In order to avoid creating any
			// impression that `config://` is a valid API, return the same
			// error as if you'd passed any other random word.
			if c.DependencyStore == nil {
				return errInvalidCredentialSpecSecOpt
			}

			csConfig, err := c.DependencyStore.Configs().Get(value)
			if err != nil {
				return errdefs.System(errors.Wrap(err, "error getting value from config store"))
			}
			// stuff the resulting secret data into a string to use as the
			// CredentialSpec
			credentialSpec = string(csConfig.Spec.Data)
		case "raw":
			credentialSpec = value
		default:
			return errInvalidCredentialSpecSecOpt
		}
	}

	if credentialSpec != "" {
		if s.Windows == nil {
			s.Windows = &specs.Windows{}
		}
		s.Windows.CredentialSpec = credentialSpec
	}

	return nil
}

func setResourcesInSpec(c *container.Container, s *specs.Spec, isHyperV bool) {
	// In s.Windows.Resources
	cpuShares := uint16(c.HostConfig.CPUShares)
	cpuMaximum := uint16(c.HostConfig.CPUPercent) * 100
	cpuCount := uint64(c.HostConfig.CPUCount)
	if c.HostConfig.NanoCPUs > 0 {
		if isHyperV {
			cpuCount = uint64(c.HostConfig.NanoCPUs / 1e9)
			leftoverNanoCPUs := c.HostConfig.NanoCPUs % 1e9
			if leftoverNanoCPUs != 0 {
				cpuCount++
				cpuMaximum = uint16(c.HostConfig.NanoCPUs / int64(cpuCount) / (1e9 / 10000))
				if cpuMaximum < 1 {
					// The requested NanoCPUs is so small that we rounded to 0, use 1 instead
					cpuMaximum = 1
				}
			}
		} else {
			cpuMaximum = uint16(c.HostConfig.NanoCPUs / int64(sysinfo.NumCPU()) / (1e9 / 10000))
			if cpuMaximum < 1 {
				// The requested NanoCPUs is so small that we rounded to 0, use 1 instead
				cpuMaximum = 1
			}
		}
	}

	if cpuMaximum != 0 || cpuShares != 0 || cpuCount != 0 {
		if s.Windows.Resources == nil {
			s.Windows.Resources = &specs.WindowsResources{}
		}
		s.Windows.Resources.CPU = &specs.WindowsCPUResources{
			Maximum: &cpuMaximum,
			Shares:  &cpuShares,
			Count:   &cpuCount,
		}
	}

	memoryLimit := uint64(c.HostConfig.Memory)
	if memoryLimit != 0 {
		if s.Windows.Resources == nil {
			s.Windows.Resources = &specs.WindowsResources{}
		}
		s.Windows.Resources.Memory = &specs.WindowsMemoryResources{
			Limit: &memoryLimit,
		}
	}

	if c.HostConfig.IOMaximumBandwidth != 0 || c.HostConfig.IOMaximumIOps != 0 {
		if s.Windows.Resources == nil {
			s.Windows.Resources = &specs.WindowsResources{}
		}
		s.Windows.Resources.Storage = &specs.WindowsStorageResources{
			Bps:  &c.HostConfig.IOMaximumBandwidth,
			Iops: &c.HostConfig.IOMaximumIOps,
		}
	}
}

// mergeUlimits merge the Ulimits from HostConfig with daemon defaults, and update HostConfig
// It will do nothing on non-Linux platform
func (daemon *Daemon) mergeUlimits(c *containertypes.HostConfig, daemonCfg *config.Config) {
	return
}

// registryKey is an interface wrapper around `registry.Key`,
// listing only the methods we care about here.
// It's mainly useful to easily allow mocking the registry in tests.
type registryKey interface {
	GetStringValue(name string) (val string, valtype uint32, err error)
	Close() error
}

var registryOpenKeyFunc = func(baseKey registry.Key, path string, access uint32) (registryKey, error) {
	return registry.OpenKey(baseKey, path, access)
}

// readCredentialSpecRegistry is a helper function to read a credential spec from
// the registry. If not found, we return an empty string and warn in the log.
// This allows for staging on machines which do not have the necessary components.
func readCredentialSpecRegistry(id, name string) (string, error) {
	key, err := registryOpenKeyFunc(registry.LOCAL_MACHINE, credentialSpecRegistryLocation, registry.QUERY_VALUE)
	if err != nil {
		return "", errors.Wrapf(err, "failed handling spec %q for container %s - registry key %s could not be opened", name, id, credentialSpecRegistryLocation)
	}
	defer key.Close()

	value, _, err := key.GetStringValue(name)
	if err != nil {
		if err == registry.ErrNotExist {
			return "", fmt.Errorf("registry credential spec %q for container %s was not found", name, id)
		}
		return "", errors.Wrapf(err, "error reading credential spec %q from registry for container %s", name, id)
	}

	return value, nil
}

// readCredentialSpecFile is a helper function to read a credential spec from
// a file. If not found, we return an empty string and warn in the log.
// This allows for staging on machines which do not have the necessary components.
func readCredentialSpecFile(id, root, location string) (string, error) {
	if filepath.IsAbs(location) {
		return "", fmt.Errorf("invalid credential spec: file:// path cannot be absolute")
	}
	base := filepath.Join(root, credentialSpecFileLocation)
	full := filepath.Join(base, location)
	if !strings.HasPrefix(full, base) {
		return "", fmt.Errorf("invalid credential spec: file:// path must be under %s", base)
	}
	bcontents, err := os.ReadFile(full)
	if err != nil {
		return "", errors.Wrapf(err, "failed to load credential spec for container %s", id)
	}
	return string(bcontents[:]), nil
}

func setupWindowsDevices(devices []containertypes.DeviceMapping) (specDevices []specs.WindowsDevice, err error) {
	for _, deviceMapping := range devices {
		if strings.HasPrefix(deviceMapping.PathOnHost, "class/") {
			specDevices = append(specDevices, specs.WindowsDevice{
				ID:     strings.TrimPrefix(deviceMapping.PathOnHost, "class/"),
				IDType: "class",
			})
		} else {
			idType, id, ok := strings.Cut(deviceMapping.PathOnHost, "://")
			if !ok {
				return nil, errors.Errorf("invalid device assignment path: '%s', must be 'class/ID' or 'IDType://ID'", deviceMapping.PathOnHost)
			}
			if idType == "" {
				return nil, errors.Errorf("invalid device assignment path: '%s', IDType cannot be empty", deviceMapping.PathOnHost)
			}
			specDevices = append(specDevices, specs.WindowsDevice{
				ID:     id,
				IDType: idType,
			})
		}
	}

	return specDevices, nil
}

An advantage of the `hcsshim.GetLayerMountPath` approach is that it won't pick up random bindfilter mounts if you somehow give it the wrong directory.

On the other hand, if we do implement real bind-mounts on Windows for some reason, then `hcsshim.GetLayerMountPath` would be wrong. But if that happens, it could be dealt with at the time.

So yeah, I think my preferred solution to this is a new function that reads the ADS file and uses `hcsshim.GetLayerMountPath`, and then propose upstreaming that into containerd.

(Technically, my preferred solution is to shift to `WithSnapshot`, but that's a wider question)