This is a place where docker is, IMO, over-flexible to the point of
confusion. 1 semantic is better than 2.

On Tue, Nov 11, 2014 at 7:19 PM, Clayton Coleman notifications@github.com
wrote:

To expand environment variables, the command for a pod must use sh - command:
["sh", "-c", "${MYVAR}"]. However, images with an entrypoint will not be
able to use this technique, because the command array is then appended to
the entry point.

Saying that images with entry points can't do variable expansion seems too
limiting. We have no technique to allow a consumer to solve that problem.
Assuming that entrypoint is here to stay and an image consumer may want to
either replace variables OR override the entrypoint, there are a few
options:

1. Allow a command to specify it should be expanded prior to
   execution. Won't allow environment in the image to take effect unless we
   load it by introspecting the image and emulating environment support. Will
   not catch ~/.bash_profile settings?
2. Allow a command to be specified as a shell call, and concatenate it
   into a string before invoking docker. Probably not safe, and breaks
   argument encapsulation (we'd have to safe shell quote).
3. When command is specified, bypass entrypoint totally. Breaks
   existing users who depend on arguments. Prevents image consumers from
   letting the image define its default behavior and specify arguments. a. Add
   an option to command to treat the command as an override to entrypoint.
   Backwards compatible. b. Add a fake environment var to be evaluated prior
   to container start that substitutes the images entrypoint (`command: ['sh',
   '-c', '${ENTRYPOINT}', ...]')
4. Add an option to command which prepends "sh" "-c". Not guaranteed
   to work, not all images have "sh" or may have nonstandard shell paths.

3 seems the most consistent (don't allow entry points) with the minor
inconvenience of ignoring the preferred path of the image author. 3a seems
crufty. 3b is very crufty. 1, 2, and 4 seem very crufty.

Reply to this email directly or view it on GitHub
#2316.

I don't expect content paths to change much, and if you specify your own cmd you're implicitly saying you know what arguments you're providing to a command and hence know the location.

If we do 3 we might introduce a flag in v1beta1/2 and then swap the behavior in 3 (no more entrypoint). Or we can just break entrypoints with commands now.

On Nov 11, 2014, at 11:17 PM, Tim Hockin notifications@github.com wrote:

This is a place where docker is, IMO, over-flexible to the point of
confusion. 1 semantic is better than 2.

On Tue, Nov 11, 2014 at 7:19 PM, Clayton Coleman notifications@github.com
wrote:

To expand environment variables, the command for a pod must use sh - command:
["sh", "-c", "${MYVAR}"]. However, images with an entrypoint will not be
able to use this technique, because the command array is then appended to
the entry point.

Saying that images with entry points can't do variable expansion seems too
limiting. We have no technique to allow a consumer to solve that problem.
Assuming that entrypoint is here to stay and an image consumer may want to
either replace variables OR override the entrypoint, there are a few
options:

1. Allow a command to specify it should be expanded prior to
   execution. Won't allow environment in the image to take effect unless we
   load it by introspecting the image and emulating environment support. Will
   not catch ~/.bash_profile settings?
2. Allow a command to be specified as a shell call, and concatenate it
   into a string before invoking docker. Probably not safe, and breaks
   argument encapsulation (we'd have to safe shell quote).
3. When command is specified, bypass entrypoint totally. Breaks
   existing users who depend on arguments. Prevents image consumers from
   letting the image define its default behavior and specify arguments. a. Add
   an option to command to treat the command as an override to entrypoint.
   Backwards compatible. b. Add a fake environment var to be evaluated prior
   to container start that substitutes the images entrypoint (`command: ['sh',
   '-c', '${ENTRYPOINT}', ...]')
4. Add an option to command which prepends "sh" "-c". Not guaranteed
   to work, not all images have "sh" or may have nonstandard shell paths.

3 seems the most consistent (don't allow entry points) with the minor
inconvenience of ignoring the preferred path of the image author. 3a seems
crufty. 3b is very crufty. 1, 2, and 4 seem very crufty.

Reply to this email directly or view it on GitHub
#2316.

—
Reply to this email directly or view it on GitHub.

See also #1710.

/cc @erictune

Related: Users find it confusing for command to be an array, or at least only an array.

Docker supports:

• Cmd - Command to run specified as a string or an array of strings.
• Entrypoint - Set the entrypoint for the container a a string or an array of strings

Do they really support two different json data types using the same field names? We shouldn't do that, but I do think we need to make this easier to use. I could imagine supporting argv (array), command (string), and overrideContainerCommand (bool that indicates the entrypoint string should be set instead of cmd).

If the user wants control over which shell to invoke, they always should have the option of /bin/sh -c ...

Kubelet should be able to inspect the container image to determine whether /bin/sh exists. If it doesn't, there are 2 options: (1) fail and generate an event that indicates permanent failure, or (2) bind-mount a shell into the container.

On Tue, Jan 13, 2015 at 12:15 PM, bgrant0607 notifications@github.com wrote:

Related: Users find it confusing for command to be an array, or at least only an array.

Docker supports:

Cmd - Command to run specified as a string or an array of strings.
Entrypoint - Set the entrypoint for the container a a string or an array of strings

Do they really support two different json data types using the same field names? We shouldn't do that, but I do think we need to make this easier to use. I could imagine supporting argv (array), command (string), and overrideContainerCommand (bool that indicates the entrypoint string should be set instead of cmd).

They really do support 3 different ways to specify commands, and no
really clear way to differentiate.

I can see us accepting []string or string for command (not that hard
to implement) since we pass it straight through to docker (though we
should look at what they actually do with it). This is, IMO, a bit of
an api disaster.

I don't see a way for us to override Docker's entrypoint without a
corresponding entrypoint field in our API. It's a mess, but there are
containers out there that set the entrypoint.

If the user wants control over which shell to invoke, they always should have the option of /bin/sh -c ...

IMO this should ALWAYS have been the API.

Kubelet should be able to inspect the container image to determine whether /bin/sh exists. If it doesn't, there are 2 options: (1) fail and generate an event that indicates permanent failure, or (2) bind-mount a shell into the container.

I think (2) is dubious, and (1) is unneeded. Just pass it in and fail
if there is no shell?

My interpretation is the following:

• Entrypoint can be an array or command string
• Cmd can be an array or command string
• If both are specified, entrypoint prefixes the command, at least if arrays are used
• Either or both can be overridden on the command line

The crossproduct is more than 3 ways.

I don't want to screw up our automatic json parsing, so if we support both the array and string, they should be separate fields.

I suppose we can deal with crashlooping in the same manner as a non-existent image. The only way to fix it would be to update the command or push a new image (and maybe explicitly update to the new image tag), but I suppose that's ok, provided we do have a way to notify the user and gracefully back off.

On Tue, Jan 13, 2015 at 3:30 PM, bgrant0607 notifications@github.com wrote:

My interpretation is the following:

Entrypoint can be an array or command string
Cmd can be an array or command string
If both are specified, entrypoint prefixes the command, at least if arrays are used
Either or both can be overridden on the command line

The crossproduct is more than 3 ways.

I decided to do an experiment to see what the cross-product really is.

https://gist.github.com/thockin/c44a88b84c2a1041a1bd

            | No ENTRYPOINT   | ENTRYPOINT entry                 |
ENTRYPOINT ["entry"]  |
------------+-----------------+------------------------------
---+-----------------------|
No CMD      | <error>         | /bin/sh -c entry                 |
entry                 |
CMD cmd     | /bin/sh -c cmd  | /bin/sh -c entry /bin/sh -c cmd  |
entry /bin/sh -c cmd  |
CMD ["cmd"] | cmd             | /bin/sh -c entry cmd             |
entry cmd             |
------------------------------------------------------------------------------------------

That's awful. It's not consistent with docker's docs and it is often
nonsensical.

I don't want to screw up our automatic json parsing, so if we support both the array and string, they should be separate fields.

I don't get this. It's relatively easy to write a receiving type that
accepts either string or array, and it is unambiguous.

I suppose we can deal with crashlooping in the same manner as a non-existent image. The only way to fix it would be to update the command or push a new image (and maybe explicitly update to the new image tag), but I suppose that's ok, provided we do have a way to notify the user and gracefully back off.

Reply to this email directly or view it on GitHub.

On Jan 13, 2015, at 7:24 PM, Tim Hockin notifications@github.com wrote:

On Tue, Jan 13, 2015 at 3:30 PM, bgrant0607 notifications@github.com wrote:

My interpretation is the following:

Entrypoint can be an array or command string
Cmd can be an array or command string
If both are specified, entrypoint prefixes the command, at least if arrays are used
Either or both can be overridden on the command line

The crossproduct is more than 3 ways.

I decided to do an experiment to see what the cross-product really is.

| No ENTRYPOINT | ENTRYPOINT entry | 
ENTRYPOINT ["entry"] | 
------------+-----------------+------------------------------ 
---+-----------------------| 
No CMD | <error> | /bin/sh -c entry | 
entry | 
CMD cmd | /bin/sh -c cmd | /bin/sh -c entry /bin/sh -c cmd | 
entry /bin/sh -c cmd | 
CMD ["cmd"] | cmd | /bin/sh -c entry cmd | 
entry cmd | 
------------------------------------------------------------------------------------------ 

That's awful. It's not consistent with docker's docs and it is often
nonsensical.

I don't want to screw up our automatic json parsing, so if we support both the array and string, they should be separate fields.

I don't get this. It's relatively easy to write a receiving type that
accepts either string or array, and it is unambiguous.

Harder to write clients for api fields that can change type, especially if we don't normalize it when we send it in a response (be strict in what you output, generous in what you accept etc).

Entry point I think makes a lot of sense for most users - it's "container as user experience" or "container as binary". We found the Openshift image works better that way (because client and server are in the same binary). I'd really like to preserve the idea of the default executable (which is really the shell). Maybe we should assume every image has an entry point, and split command into "executable" and "arguments"

I suppose we can deal with crashlooping in the same manner as a non-existent image. The only way to fix it would be to update the command or push a new image (and maybe explicitly update to the new image tag), but I suppose that's ok, provided we do have a way to notify the user and gracefully back off.

Reply to this email directly or view it on GitHub.
—
Reply to this email directly or view it on GitHub.

The issue I have with entrypoint is not that it doesn't make sense, but
it's totally non-obvious what the final result will be when you start
mixing entrypoint and cmd.

If we split command[] into argv0 and args, I guess it's a bit clearer, but
for containers without an ENTRYPOINT, args will be interpretted as argv0 by
docker - still a messy UX.

On Tue, Jan 13, 2015 at 4:50 PM, Clayton Coleman notifications@github.com
wrote:

On Jan 13, 2015, at 7:24 PM, Tim Hockin notifications@github.com
wrote:

On Tue, Jan 13, 2015 at 3:30 PM, bgrant0607 notifications@github.com
wrote:

My interpretation is the following:

Entrypoint can be an array or command string
Cmd can be an array or command string
If both are specified, entrypoint prefixes the command, at least if
arrays are used
Either or both can be overridden on the command line

The crossproduct is more than 3 ways.

I decided to do an experiment to see what the cross-product really is.

| No ENTRYPOINT | ENTRYPOINT entry |
ENTRYPOINT ["entry"] |
------------+-----------------+------------------------------
---+-----------------------|
No CMD | <error> | /bin/sh -c entry |
entry |
CMD cmd | /bin/sh -c cmd | /bin/sh -c entry /bin/sh -c cmd |
entry /bin/sh -c cmd |
CMD ["cmd"] | cmd | /bin/sh -c entry cmd |
entry cmd |

------------------------------------------------------------------------------------------

That's awful. It's not consistent with docker's docs and it is often
nonsensical.

I don't want to screw up our automatic json parsing, so if we support
both the array and string, they should be separate fields.

I don't get this. It's relatively easy to write a receiving type that
accepts either string or array, and it is unambiguous.

Harder to write clients for api fields that can change type, especially if
we don't normalize it when we send it in a response (be strict in what you
output, generous in what you accept etc).

Entry point I think makes a lot of sense for most users - it's "container
as user experience" or "container as binary". We found the Openshift image
works better that way (because client and server are in the same binary).
I'd really like to preserve the idea of the default executable (which is
really the shell). Maybe we should assume every image has an entry point,
and split command into "executable" and "arguments"

I suppose we can deal with crashlooping in the same manner as a
non-existent image. The only way to fix it would be to update the command
or push a new image (and maybe explicitly update to the new image tag), but
I suppose that's ok, provided we do have a way to notify the user and
gracefully back off.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

On Jan 13, 2015, at 7:55 PM, Tim Hockin notifications@github.com wrote:

The issue I have with entrypoint is not that it doesn't make sense, but
it's totally non-obvious what the final result will be when you start
mixing entrypoint and cmd.

If we split command[] into argv0 and args, I guess it's a bit clearer, but
for containers without an ENTRYPOINT, args will be interpretted as argv0 by
docker - still a messy UX.

The Kubelet can hide that though when the container is invoked (given the image must be pulled). I guess I'm proposing there only be one option, and we enforce "what users get".

On Tue, Jan 13, 2015 at 4:50 PM, Clayton Coleman notifications@github.com
wrote:

On Jan 13, 2015, at 7:24 PM, Tim Hockin notifications@github.com
wrote:

On Tue, Jan 13, 2015 at 3:30 PM, bgrant0607 notifications@github.com
wrote:

My interpretation is the following:

Entrypoint can be an array or command string
Cmd can be an array or command string
If both are specified, entrypoint prefixes the command, at least if
arrays are used
Either or both can be overridden on the command line

The crossproduct is more than 3 ways.

I decided to do an experiment to see what the cross-product really is.

| No ENTRYPOINT | ENTRYPOINT entry |
ENTRYPOINT ["entry"] |
------------+-----------------+------------------------------
---+-----------------------|
No CMD | <error> | /bin/sh -c entry |
entry |
CMD cmd | /bin/sh -c cmd | /bin/sh -c entry /bin/sh -c cmd |
entry /bin/sh -c cmd |
CMD ["cmd"] | cmd | /bin/sh -c entry cmd |
entry cmd |

------------------------------------------------------------------------------------------

That's awful. It's not consistent with docker's docs and it is often
nonsensical.

I don't want to screw up our automatic json parsing, so if we support
both the array and string, they should be separate fields.

I don't get this. It's relatively easy to write a receiving type that
accepts either string or array, and it is unambiguous.

Harder to write clients for api fields that can change type, especially if
we don't normalize it when we send it in a response (be strict in what you
output, generous in what you accept etc).

Entry point I think makes a lot of sense for most users - it's "container
as user experience" or "container as binary". We found the Openshift image
works better that way (because client and server are in the same binary).
I'd really like to preserve the idea of the default executable (which is
really the shell). Maybe we should assume every image has an entry point,
and split command into "executable" and "arguments"

I suppose we can deal with crashlooping in the same manner as a
non-existent image. The only way to fix it would be to update the command
or push a new image (and maybe explicitly update to the new image tag), but
I suppose that's ok, provided we do have a way to notify the user and
gracefully back off.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

—
Reply to this email directly or view it on GitHub.

Among other things, Swagger has trouble with custom types: #2971.

What do you mean "hide that"? If I have a container with no entrypoint and
I pass a pod with argv0="" and args=["foo", "bar"], what gets exec()ed?

On Tue, Jan 13, 2015 at 5:11 PM, Clayton Coleman notifications@github.com
wrote:

On Jan 13, 2015, at 7:55 PM, Tim Hockin notifications@github.com
wrote:

The issue I have with entrypoint is not that it doesn't make sense, but
it's totally non-obvious what the final result will be when you start
mixing entrypoint and cmd.

If we split command[] into argv0 and args, I guess it's a bit clearer,
but
for containers without an ENTRYPOINT, args will be interpretted as argv0
by
docker - still a messy UX.

The Kubelet can hide that though when the container is invoked (given the
image must be pulled). I guess I'm proposing there only be one option, and
we enforce "what users get".

On Tue, Jan 13, 2015 at 4:50 PM, Clayton Coleman <
notifications@github.com>
wrote:

On Jan 13, 2015, at 7:24 PM, Tim Hockin notifications@github.com
wrote:

On Tue, Jan 13, 2015 at 3:30 PM, bgrant0607 <
notifications@github.com>
wrote:

My interpretation is the following:

Entrypoint can be an array or command string
Cmd can be an array or command string
If both are specified, entrypoint prefixes the command, at least
if
arrays are used
Either or both can be overridden on the command line

The crossproduct is more than 3 ways.

I decided to do an experiment to see what the cross-product really
is.

| No ENTRYPOINT | ENTRYPOINT entry |
ENTRYPOINT ["entry"] |
------------+-----------------+------------------------------
---+-----------------------|
No CMD | <error> | /bin/sh -c entry |
entry |
CMD cmd | /bin/sh -c cmd | /bin/sh -c entry /bin/sh -c cmd |
entry /bin/sh -c cmd |
CMD ["cmd"] | cmd | /bin/sh -c entry cmd |
entry cmd |

---

That's awful. It's not consistent with docker's docs and it is often
nonsensical.

> I don't want to screw up our automatic json parsing, so if we
support
both the array and string, they should be separate fields.

I don't get this. It's relatively easy to write a receiving type
that
accepts either string or array, and it is unambiguous.

Harder to write clients for api fields that can change type,
especially if
we don't normalize it when we send it in a response (be strict in what
you
output, generous in what you accept etc).

Entry point I think makes a lot of sense for most users - it's
"container
as user experience" or "container as binary". We found the Openshift
image
works better that way (because client and server are in the same
binary).
I'd really like to preserve the idea of the default executable (which
is
really the shell). Maybe we should assume every image has an entry
point,
and split command into "executable" and "arguments"

I suppose we can deal with crashlooping in the same manner as a
non-existent image. The only way to fix it would be to update the
command
or push a new image (and maybe explicitly update to the new image
tag), but
I suppose that's ok, provided we do have a way to notify the user and
gracefully back off.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/2316#issuecomment-69850996>

.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

We should not limit our UX by shortcomings in tools.

On Tue, Jan 13, 2015 at 5:14 PM, bgrant0607 notifications@github.com
wrote:

Among other things, Swagger has trouble with custom types: #2971
#2971.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

Personally, I'd prefer separate argv and command fields. Overloading a single field is confusing and error-prone.

I'm all for trying to fix it, but if we can't make the swagger work, it will degrade the UX in other ways, through incorrect documentation, validation, and configuration transformations.

Let's figure out what we thing sane semantics are, then debate how to
achieve that :)

As it is, we either do what docker does - two fields with
bizarrely-intersecting meaning - or we restrict what can be done. Or ???

On Tue, Jan 13, 2015 at 5:21 PM, bgrant0607 notifications@github.com
wrote:

Personally, I'd prefer separate argv and command fields. Overloading a
single field is confusing and error-prone.

I'm all for trying to fix it, but if we can't make the swagger work, it
will degrade the UX in other ways, through incorrect documentation,
validation, and configuration transformations.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

We should at least return what command was executed in container status so users can debug what went wrong.

I'm ok with some restrictions, especially if there's an escape hatch. But we do need to make containers with entrypoints work. I've used them and agree they are convenient.

One option could be to do podex-like expansion in the client and build the command there. Kubelet could always clear entrypoint (empty array?). Would kind of suck for direct API usage, though.

now THAT I can get behind :)

On Tue, Jan 13, 2015 at 5:25 PM, bgrant0607 notifications@github.com
wrote:

We should at least return what command was executed in container status so
users can debug what went wrong.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

Maybe it wouldn't be so bad to simply say we have 2 fields, argv0 and
args. If you specify argv0 we do the equivalent of docker's ENTRYPOINT.
If you specify args, we do the equivalent of docker's CMD. If you do not
spec argv0 and the container has no ENTRYPOINT, args[0] gets promoted to
argv0. This is, in effect what happens today. There's still a problem, in
that docker allows arrays for ENTRYPOINT.

Then we have to answer the question of bare-string (not array) commands.

On Tue, Jan 13, 2015 at 5:27 PM, bgrant0607 notifications@github.com
wrote:

I'm ok with some restrictions, especially if there's an escape hatch. But
we do need to make containers with entrypoints work. I've used them and
agree they are convenient.

One option could be to do podex-like expansion in the client and build the
command there. Kubelet could always clear entrypoint (empty array?). Would
kind of suck for direct API usage, though.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

/bin/sh "foo" "bar". If you have entry point, entrypoint "foo" "bar"

On Jan 13, 2015, at 8:16 PM, Tim Hockin notifications@github.com wrote:

What do you mean "hide that"? If I have a container with no entrypoint and
I pass a pod with argv0="" and args=["foo", "bar"], what gets exec()ed?

On Tue, Jan 13, 2015 at 5:11 PM, Clayton Coleman notifications@github.com
wrote:

On Jan 13, 2015, at 7:55 PM, Tim Hockin notifications@github.com
wrote:

The issue I have with entrypoint is not that it doesn't make sense, but
it's totally non-obvious what the final result will be when you start
mixing entrypoint and cmd.

If we split command[] into argv0 and args, I guess it's a bit clearer,
but
for containers without an ENTRYPOINT, args will be interpretted as argv0
by
docker - still a messy UX.

The Kubelet can hide that though when the container is invoked (given the
image must be pulled). I guess I'm proposing there only be one option, and
we enforce "what users get".

On Tue, Jan 13, 2015 at 4:50 PM, Clayton Coleman <
notifications@github.com>
wrote:

On Jan 13, 2015, at 7:24 PM, Tim Hockin notifications@github.com
wrote:

On Tue, Jan 13, 2015 at 3:30 PM, bgrant0607 <
notifications@github.com>
wrote:

My interpretation is the following:

Entrypoint can be an array or command string
Cmd can be an array or command string
If both are specified, entrypoint prefixes the command, at least
if
arrays are used
Either or both can be overridden on the command line

The crossproduct is more than 3 ways.

I decided to do an experiment to see what the cross-product really
is.

| No ENTRYPOINT | ENTRYPOINT entry |
ENTRYPOINT ["entry"] |
------------+-----------------+------------------------------
---+-----------------------|
No CMD | <error> | /bin/sh -c entry |
entry |
CMD cmd | /bin/sh -c cmd | /bin/sh -c entry /bin/sh -c cmd |
entry /bin/sh -c cmd |
CMD ["cmd"] | cmd | /bin/sh -c entry cmd |
entry cmd |

---

That's awful. It's not consistent with docker's docs and it is often
nonsensical.

> I don't want to screw up our automatic json parsing, so if we
support
both the array and string, they should be separate fields.

I don't get this. It's relatively easy to write a receiving type
that
accepts either string or array, and it is unambiguous.

Harder to write clients for api fields that can change type,
especially if
we don't normalize it when we send it in a response (be strict in what
you
output, generous in what you accept etc).

Entry point I think makes a lot of sense for most users - it's
"container
as user experience" or "container as binary". We found the Openshift
image
works better that way (because client and server are in the same
binary).
I'd really like to preserve the idea of the default executable (which
is
really the shell). Maybe we should assume every image has an entry
point,
and split command into "executable" and "arguments"

I suppose we can deal with crashlooping in the same manner as a
non-existent image. The only way to fix it would be to update the
command
or push a new image (and maybe explicitly update to the new image
tag), but
I suppose that's ok, provided we do have a way to notify the user and
gracefully back off.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/2316#issuecomment-69850996>

.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

—
Reply to this email directly or view it on GitHub.

Why is sh involved? If command is an array, there should be no shell at
all.

My main gripe is that I do not see a semantic that a) makes some semblance
of sense and b) retains full compat with docker.

On Tue, Jan 13, 2015 at 5:45 PM, Clayton Coleman notifications@github.com
wrote:

/bin/sh "foo" "bar". If you have entry point, entrypoint "foo" "bar"

On Jan 13, 2015, at 8:16 PM, Tim Hockin notifications@github.com
wrote:

What do you mean "hide that"? If I have a container with no entrypoint
and
I pass a pod with argv0="" and args=["foo", "bar"], what gets exec()ed?

On Tue, Jan 13, 2015 at 5:11 PM, Clayton Coleman <
notifications@github.com>
wrote:

On Jan 13, 2015, at 7:55 PM, Tim Hockin notifications@github.com
wrote:

The issue I have with entrypoint is not that it doesn't make sense,
but
it's totally non-obvious what the final result will be when you
start
mixing entrypoint and cmd.

If we split command[] into argv0 and args, I guess it's a bit
clearer,
but
for containers without an ENTRYPOINT, args will be interpretted as
argv0
by
docker - still a messy UX.

The Kubelet can hide that though when the container is invoked (given
the
image must be pulled). I guess I'm proposing there only be one option,
and
we enforce "what users get".

On Tue, Jan 13, 2015 at 4:50 PM, Clayton Coleman <
notifications@github.com>
wrote:

On Jan 13, 2015, at 7:24 PM, Tim Hockin <
notifications@github.com>
wrote:

On Tue, Jan 13, 2015 at 3:30 PM, bgrant0607 <
notifications@github.com>
wrote:

My interpretation is the following:

Entrypoint can be an array or command string
Cmd can be an array or command string
If both are specified, entrypoint prefixes the command, at
least
if
arrays are used
Either or both can be overridden on the command line

The crossproduct is more than 3 ways.

I decided to do an experiment to see what the cross-product
really
is.

| No ENTRYPOINT | ENTRYPOINT entry |
ENTRYPOINT ["entry"] |
------------+-----------------+------------------------------
---+-----------------------|
No CMD | <error> | /bin/sh -c entry |
entry |
CMD cmd | /bin/sh -c cmd | /bin/sh -c entry /bin/sh -c cmd |
entry /bin/sh -c cmd |
CMD ["cmd"] | cmd | /bin/sh -c entry cmd |
entry cmd |

---

That's awful. It's not consistent with docker's docs and it is
often
nonsensical.

> I don't want to screw up our automatic json parsing, so if we
support
both the array and string, they should be separate fields.

I don't get this. It's relatively easy to write a receiving type
that
accepts either string or array, and it is unambiguous.

Harder to write clients for api fields that can change type,
especially if
we don't normalize it when we send it in a response (be strict in
what
you
output, generous in what you accept etc).

Entry point I think makes a lot of sense for most users - it's
"container
as user experience" or "container as binary". We found the
Openshift
image
works better that way (because client and server are in the same
binary).
I'd really like to preserve the idea of the default executable
(which
is
really the shell). Maybe we should assume every image has an entry
point,
and split command into "executable" and "arguments"

I suppose we can deal with crashlooping in the same manner as
a
non-existent image. The only way to fix it would be to update the
command
or push a new image (and maybe explicitly update to the new image
tag), but
I suppose that's ok, provided we do have a way to notify the user
and
gracefully back off.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
<

https://github.com/GoogleCloudPlatform/kubernetes/issues/2316#issuecomment-69850996>

.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/2316#issuecomment-69852949>

.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#2316 (comment)
.

@thockin My bad, 'docker entrypoint/cmd' is the entrypoint and container of the actual docker container created

Thoughts on matrix:

1. make sure to cover entrypoint/command being multi-element arrays

2. do we want to cover Dockerfile's plain-string (non-array)mode for ENTRYPOINT
   and CMD? It's busted, IMO.

3. Setting Command in our API and absorbing CMD from dockerfile is weird.
   Maybe we should say: if you set Command in our API, but not Args, we will
   blank-out CMD to Docker. A quick test convinces me this is what docker does,
   though I am not sure if that is in the CLI (pass Cmd=[] to API) or in the
   daemon. In short, if you set Command in k8s, all of Dockerfile is
   overridden. If you set Args in k8s, Dockerfile.CMD can show through.

4. your 8th and 9th rows seem backwards in final result

@thockin

do we want to cover Dockerfile's plain-string (non-array)mode for ENTRYPOINT
and CMD? It's busted, IMO.

Please god no

theoretically there's a collision with a valid JSONPath thing you would want to store in an env var and what our syntax would be) as the only way to do expansion, and just use the normal Value field.

That's why I (slightly) prefer a different API block - no escaping rules

I don't see us ever changing env to a map. It's not worth it.

Whether a separate field like Generate or a different API block, I don't see why we need escaping. I don't think we should support arbitrary expansion, just straight copying of the specified field.

As for Command/Args in v1beta1/2 -- maybe introduce a new Entrypoint field and leave Command as is? Hopefully these versions will be gone soon, anyway. They just need to not lose information when roundtripping.

If we overlap the same API block, we need escaping. what if I want
literally FOO="$.pod.metadata.name" - don't interpret it. I need to a) be
able to escape the $ and b) know that I need to and c) know how to.

On Fri, Mar 27, 2015 at 9:31 AM, Brian Grant notifications@github.com
wrote:

I don't see us ever changing env to a map. It's not worth it.

Whether a separate field like Generate or a different API block, I don't
see why we need escaping. I don't think we should support arbitrary
expansion, just straight copying of the specified field.

As for Command/Args in v1beta1/2 -- maybe introduce a new Entrypoint field
and leave Command as is? Hopefully these versions will be gone soon,
anyway. They just need to not lose information when roundtripping.

—
Reply to this email directly or view it on GitHub
#2316 (comment)
.

I think that's a good thing, because order always matters in ENV resolution.

----- Original Message -----

This model precludes changing env to a map, ever.

On Fri, Mar 27, 2015 at 6:32 AM, Clayton Coleman notifications@github.com
wrote:

On Mar 27, 2015, at 2:58 AM, Brian Grant notifications@github.com
wrote:

New block in the API.
I kind of like the "alternate value" field on the env var struct. Having
to explain two orderings to users seems excessive:

[
{
Name: "FOO",
Value: "",
Generated: "$.pod.blah.baz",
}
]

One minor point in its favor is that a naive client knows "FOO" is set,
and can read it as empty if they don't understand generated, and the output
would have FOO= which is exactly what you would get running the pod outside
of Kube. But it's not a strong opinion.

For the downward API, just json paths. Not really "magic", since they'd
be in a separate api block.
Just whole values. Fancier expansion would have to be done using
standard envvar expansions.
The goal would be to make it consistent with other parts of the API,
such as field selection and perhaps templating in kubectl and elsewhere.
The references would at least use actual API field names and paths would be
composed via a principled method as opposed to ${K8S_POD_NAME}, which is
100% ad hoc.
—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#2316 (comment)
.

---

Reply to this email directly or view it on GitHub:
#2316 (comment)

My proposal though was there were two fields in the EnvVar, one for normal, unqualified escaping (handled by docker), and one for generated / downward api fields. The two were exclusive, and non-overlapping.

----- Original Message -----

If we overlap the same API block, we need escaping. what if I want
literally FOO="$.pod.metadata.name" - don't interpret it. I need to a) be
able to escape the $ and b) know that I need to and c) know how to.

On Fri, Mar 27, 2015 at 9:31 AM, Brian Grant notifications@github.com
wrote:

I don't see us ever changing env to a map. It's not worth it.

Whether a separate field like Generate or a different API block, I don't
see why we need escaping. I don't think we should support arbitrary
expansion, just straight copying of the specified field.

As for Command/Args in v1beta1/2 -- maybe introduce a new Entrypoint field
and leave Command as is? Hopefully these versions will be gone soon,
anyway. They just need to not lose information when roundtripping.

—
Reply to this email directly or view it on GitHub
#2316 (comment)
.

---

Reply to this email directly or view it on GitHub:
#2316 (comment)

...which is just a different factoring than being in a different API block,
but precludes ever doing a map. It annoys me that our env is not a map. I
know Brian thinks we'll never convert, but I am more hopeful.

I'm fine either way - we can always rev the API later :)

On Fri, Mar 27, 2015 at 9:44 AM, Clayton Coleman notifications@github.com
wrote:

My proposal though was there were two fields in the EnvVar, one for
normal, unqualified escaping (handled by docker), and one for generated /
downward api fields. The two were exclusive, and non-overlapping.

----- Original Message -----

If we overlap the same API block, we need escaping. what if I want
literally FOO="$.pod.metadata.name" - don't interpret it. I need to a)
be
able to escape the $ and b) know that I need to and c) know how to.

On Fri, Mar 27, 2015 at 9:31 AM, Brian Grant notifications@github.com
wrote:

I don't see us ever changing env to a map. It's not worth it.

Whether a separate field like Generate or a different API block, I
don't
see why we need escaping. I don't think we should support arbitrary
expansion, just straight copying of the specified field.

As for Command/Args in v1beta1/2 -- maybe introduce a new Entrypoint
field
and leave Command as is? Hopefully these versions will be gone soon,
anyway. They just need to not lose information when roundtripping.

—
Reply to this email directly or view it on GitHub
<
#2316 (comment)

.

---

Reply to this email directly or view it on GitHub:

#2316 (comment)

—
Reply to this email directly or view it on GitHub
#2316 (comment)
.

@thockin I am surprised given your general distaste for env vars that would would want to make that big of an API change around them :)

@smarterclayton has a good point about ordering.

One of the things that annoys me more than env vars is awkward APIs :)

I think that's a good thing, because order always matters in ENV resolution.

Why does order matter? We don't do cross-env resolution (AFAIK)

@bgrant0607

As for Command/Args in v1beta1/2 -- maybe introduce a new Entrypoint field and leave Command as is? Hopefully these versions will be gone soon, anyway. They just need to not lose information when roundtripping.

That is a good idea.

@yifan-gu I see that you moved RunContainer into the dockertools package recently; I like the rest of what you did but I feel pretty strongly that RunContainer ought to be a kubelet function, since it defines the bridge between the kubernetes API and the docker API. I like the other methods moving into the dockertools package. Another reason I think it should stay is that lot of kubelet internals get injected into it. Any thoughts?

Order matters when Docker resolves things. Or if someone else implements an ordered algorithm. If you don't have that clear ordering then you don't have consistent resolution.

On Mar 27, 2015, at 1:54 PM, Tim Hockin notifications@github.com wrote:

I think that's a good thing, because order always matters in ENV resolution.

Why does order matter? We don't do cross-env resolution (AFAIK)

—
Reply to this email directly or view it on GitHub.

@pmorie Thanks for the notification. I think there is still a runContainer in kubelet, which can bridge kubelet and the container runtime. The kubelet will describe a container to the container runtime and let the runtime launch the container.
By kubelet internals do you mean ref, recorder, etc? If so, I agree that ideally they should not be injected to the container runtime. I am thinking of ways to get around with them

FYI, here is a related discussion:
#6068 (comment)

@yifan-gu Check out my wip at #6100

@smarterclayton @bgrant0607 Can this issue be closed now that #6100 has landed?

Yes.

The network test failure made me wonder: do we need to distinguish between unspecified command/args and empty command/args? How would I clear the CMD specified in the image?

@bgrant0607, as in, you want to run the default entrypoint without any
arguments, when the image has default arguments specified?t

On Wed, Apr 1, 2015 at 1:02 PM, Brian Grant notifications@github.com
wrote:

The network test failure made me wonder: do we need to distinguish between
unspecified command/args and empty command/args? How would I clear the CMD
specified in the image?

—
Reply to this email directly or view it on GitHub
#2316 (comment)
.

I'm not saying I want to do that, but that's the scenario I was thinking about, yes.

@bgrant0607 The way you would have to do it currently is manually specify
the entrypoint to be the same one as in the image. I think it would be the
same with just docker, as well.

On Wed, Apr 1, 2015 at 1:27 PM, Brian Grant notifications@github.com
wrote:

I'm not saying I want to do that, but that's the scenario I was thinking
about, yes.

—
Reply to this email directly or view it on GitHub
#2316 (comment)
.