Closes#16667
Uses the prefix "devicemapper:" for all the fmt and logrus error, debug, and info messages.
Signed-off-by: Chris Dituri <csdituri@gmail.com>
Upstream-commit: 9b584781ad9c810bc9060448f391a3ae6a7d7cbb
Component: engine
This commit adds a transfer manager which deduplicates and schedules
transfers, and also an upload manager and download manager that build on
top of the transfer manager to provide high-level interfaces for uploads
and downloads. The push and pull code is modified to use these building
blocks.
Some benefits of the changes:
- Simplification of push/pull code
- Pushes can upload layers concurrently
- Failed downloads and uploads are retried after backoff delays
- Cancellation is supported, but individual transfers will only be
cancelled if all pushes or pulls using them are cancelled.
- The distribution code is decoupled from Docker Engine packages and API
conventions (i.e. streamformatter), which will make it easier to split
out.
This commit also includes unit tests for the new distribution/xfer
package. The tests cover 87.8% of the statements in the package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 572ce802306a4e919802e5b77cbeca94acda7c0a
Component: engine
jm.ID is already checked in the outer "if",
so theres no reason to check it again here.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 7a6f5d9b3186de2116686d5e2c40eff673dae6ec
Component: engine
aufs kernel module creates whiteout files on upper layer delete (and
other situations) and those files already are 'translated' regarding
ownership in host terms (e.g. they are already "0:0" owned), so when
these layers are copied around with pkg/archive we don't want to try and
translate these files regarding ownership.
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
Upstream-commit: 1626c9dae0f285185e3ef4c1e2be55a807e8c0ed
Component: engine
Each plug-in operates as a separate service, and registers with Docker
through general (plug-ins API)
[https://blog.docker.com/2015/06/extending-docker-with-plugins/]. No
Docker daemon recompilation is required in order to add / remove an
authentication plug-in. Each plug-in is notified twice for each
operation: 1) before the operation is performed and, 2) before the
response is returned to the client. The plug-ins can modify the response
that is returned to the client.
The authorization depends on the authorization effort that takes place
in parallel [https://github.com/docker/docker/issues/13697].
This is the official issue of the authorization effort:
https://github.com/docker/docker/issues/14674
(Here)[https://github.com/rhatdan/docker-rbac] you can find an open
document that discusses a default RBAC plug-in for Docker.
Signed-off-by: Liron Levin <liron@twistlock.com>
Added container create flow test and extended the verification for ps
Upstream-commit: 75c353f0ad73bd83ed18e92857dd99a103bb47e3
Component: engine
In the existing code, "diff" has function scope and the value from the
previous iteration may be used if it is not reset. This appears to be an
oversight. This commit changes its scope to the for loop body.
One confusing point is that the cursor movement escape sequences appear
to be necessary even if the requested movement is 0. I haven't been able
to figure out why this makes a difference.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 59df2adc071e0186ccd0ba7ea9387142e168d735
Component: engine
When we handle a message that isn't tracked in the "line" map (for
example, one with no ID), clear the line map. This means we won't update
lines that were part of a previous, completed set of operations when
doing something like pull -a. It also has the beneficial side effect
of avoiding terminal glitching in these types of situations, since
messages that don't get tracked in the "line" map cause the count of the
number of lines to get out of sync.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: fc41d393946e6da59792da9ed6e5ab8ed6f58814
Component: engine
When user call the `Call()` method, they don't always want to sent
some args or get the return value, so they use `nil` when call `Call()`
method and this will casue an error. It's better to not trying to
encode or decode if it's nil.
Signed-off-by: Lei Jitang <leijitang@huawei.com>
Upstream-commit: a2d348968fe754df41d0a1b56c7ba29ebbaa6849
Component: engine
This function was starting a goroutine that modifies one of its return
values. The intent is for the goroutine to only influence the return
value when it's causing the function to return, but it's racy and can
also modify the return value when the function is returning due to the
timeout. Fix the goroutine to not modify return values directly.
Also, give the channel a buffer so that the goroutine doesn't block
forever after a timeout.
Fixes#18305
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 2704fd9156bfb0fb8dc16c42902bb18ea5aa94a9
Component: engine
I saw a failure of TestDockerCmdWithTimeout. This test starts a command
that produces output after 10 ms, but uses a 5 ms timeout, so normally
the command will be killed before the output. The time intervals are so
small that the timeout may not reliably trigger before the output, which
can cause the test to fail.
This commit changes the test to only fail if the process is still alive
after 10 seconds. This means the test will confirm that the timeouts are
happening, but not attempt to gauge that the timeouts are happening
within milliseconds of when they are expected (which can't be done
reliably).
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 13d768b8ee05d3158c62d761e4ebe657cd7a8c25
Component: engine
A TopicFunc is an interface to let the pubisher decide whether it needs
to send a message to a subscriber or not. It returns true if the
publisher must send the message and false otherwise.
Users of the pubsub package can create a subscriber with a topic
function by calling `pubsub.SubscribeTopic`.
Message delivery has also been modified to use concurrent channels per
subscriber. That way, topic verification and message delivery is not
o(N+M) anymore, based on the number of subscribers and topic verification
complexity.
Using pubsub topics, the API stops controlling the message delivery,
delegating that function to a topic generated with the filtering
provided by the user. The publisher sends every message to the
subscriber if there is no filter, but the api doesn't have to select
messages to return anymore.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Upstream-commit: 434d2e8745696255a204d9eefc6a2854ff74e5c2
Component: engine
Improves the current filtering implementation complixity.
Currently, the best case is O(N) and worst case O(N^2) for key-value filtering.
In the new implementation, the best case is O(1) and worst case O(N), again for key-value filtering.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Upstream-commit: 93d1dd8036d57f5cf1e5cbbbad875ae9a6fa6180
Component: engine
It will Tar up contents of child directory onto tmpfs if mounted over
This patch will use the new PreMount and PostMount hooks to "tar"
up the contents of the base image on top of tmpfs mount points.
Signed-off-by: Dan Walsh <dwalsh@redhat.com>
Upstream-commit: b3e527dfd242ad30c0297c8b257862116cf2c50e
Component: engine
It actually adds nothing to queuing requests.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
Upstream-commit: ca5795cef810c85f101eb0aa3efe3ec8d756490b
Component: engine
Moved a defer up to a better spot.
Fixed TestUntarPathWithInvalidDest to actually fail for the right reason
Closes#18170
Signed-off-by: Doug Davis <dug@us.ibm.com>
Upstream-commit: 7bb9fc415ac50abf5f754005264bfe152eeca7d6
Component: engine
Add distribution package for managing pulls and pushes. This is based on
the old code in the graph package, with major changes to work with the
new image/layer model.
Add v1 migration code.
Update registry, api/*, and daemon packages to use the reference
package's types where applicable.
Update daemon package to use image/layer/tag stores instead of the graph
package
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 4352da7803d182a6013a5238ce20a7c749db979a
Component: engine
Can't safely use uint32 for locker since we need to decrement the count,
which requires loading the unit and doing some math, which is inherintly
racey.
Instead use Int32 which we can safely use with atomic and AddInt32 with
`-1`
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 985175fd8f8f662d5067bd62a89330e9a437375c
Component: engine
Fixes#16555
Original docker `cp` always copy symbol link itself instead of target,
now we provide '-L' option to allow docker to follow symbol link to real
target.
Signed-off-by: Zhang Wei <zhangwei555@huawei.com>
Upstream-commit: 92600bdec1284f9031868751f61bef476d2e1dbd
Component: engine
