How Go's Standard Library Implements --
The standalone -- is often described as meaning “stop parsing options.”
That description is useful, but it can make -- sound like a special instruction interpreted by the shell or operating system. It is neither.
The shell passes -- to the program as an ordinary argument. The program—or a command-line parsing library used by the program—must recognize it and decide what it means.
Go's standard-library flag package provides a particularly clear implementation. Its handling of -- is only a few lines long.
By following the code from os.Args into the parser, we can see exactly what happens:
- The operating system supplies an argument vector.
- Go exposes that vector as
os.Args. flag.Parse()removes the program name and passes the remaining strings to aFlagSet.FlagSet.Parse()repeatedly asksparseOne()to parse the next flag.parseOne()recognizes the exact string--.- It removes
--from the remaining arguments. - It reports that there are no more flags to parse.
- Everything after
--remains available as positional arguments.
There is no hidden shell behavior. It is ordinary string processing inside the program.
A Small Example
Consider this program:
package main
import (
"flag"
"fmt"
)
func main() {
verbose := flag.Bool("verbose", false, "enable verbose output")
output := flag.String("output", "", "output file")
flag.Parse()
fmt.Printf("verbose: %v\n", *verbose)
fmt.Printf("output: %q\n", *output)
fmt.Printf("arguments: %#v\n", flag.Args())
}
Run it with flags followed by ordinary arguments:
$ go run . -verbose -output report.txt alpha beta
verbose: true
output: "report.txt"
arguments: []string{"alpha", "beta"}
Now place -- before something that looks like a flag:
$ go run . -verbose -- -output report.txt
verbose: true
output: ""
arguments: []string{"-output", "report.txt"}
Before --, -verbose is interpreted as a flag.
After --, -output is not interpreted as a flag. It is returned as an ordinary argument, followed by report.txt.
The resulting argument boundary is:
flags positional arguments
----------------------- --------------------------
-verbose -output report.txt
--
^
removed by the parser
The -- itself does not appear in flag.Args(). It marks the boundary and is then discarded.
Start at os.Args
A Go program receives its command line through os.Args.
For a command such as:
$ ./example -verbose -- -output report.txt
the slice will conceptually contain:
[]string{
"./example",
"-verbose",
"--",
"-output",
"report.txt",
}
The first element is normally the name or path used to invoke the program:
os.Args[0] == "./example"
The remaining elements are the arguments supplied to it:
os.Args[1:] == []string{
"-verbose",
"--",
"-output",
"report.txt",
}
Nothing has interpreted -- yet. At this point it is simply the string "--" at index 2.
This is important because it establishes where the behavior does not occur:
- The shell did not remove
--. - The operating system did not classify it as an option terminator.
os.Argsdoes not give it any special type.- Go does not treat it specially merely because it appears in the argument vector.
Its meaning appears later, inside the flag package.
flag.Parse() Passes os.Args[1:] to CommandLine
The package-level flag.Parse() function is a small wrapper:
func Parse() {
// Ignore errors; CommandLine is set for ExitOnError.
CommandLine.Parse(os.Args[1:])
}
The complete command line is not passed to the parser. It passes:
os.Args[1:]
That excludes os.Args[0], the command name.
The remaining argument slice is sent to CommandLine.Parse().
CommandLine is the default package-level FlagSet. Calls such as:
flag.Bool("verbose", false, "enable verbose output")
flag.String("output", "", "output file")
register flags with this default set.
Therefore:
flag.Parse()
is effectively shorthand for:
flag.CommandLine.Parse(os.Args[1:])
You can also create a separate FlagSet and pass it any slice of strings:
fs := flag.NewFlagSet("example", flag.ContinueOnError)
err := fs.Parse([]string{"-verbose", "--", "-output"})
This reinforces the fact that the parser is not intrinsically tied to a shell or even to the process's actual command line. It operates on a []string.
FlagSet.Parse() Stores the Remaining Arguments
The FlagSet.Parse() method begins by storing the supplied slice:
func (f *FlagSet) Parse(arguments []string) error {
f.parsed = true
f.args = arguments
for {
seen, err := f.parseOne()
if seen {
continue
}
if err == nil {
break
}
// Error handling omitted.
}
return nil
}
The important assignment is:
f.args = arguments
f.args represents the portion of the argument list that has not yet been consumed.
Suppose parsing begins with:
f.args = []string{
"-verbose",
"--",
"-output",
"report.txt",
}
Parse() then repeatedly calls:
f.parseOne()
The return values mean, approximately:
seen == true
A flag was successfully parsed. Call parseOne() again.
seen == false && err == nil
The next argument is not another flag, or the parser encountered --. Stop parsing normally.
err != nil
Something looked like a flag but was invalid.
This loop is the mechanism that turns the result from parseOne() into a parsing boundary.
parseOne() Examines Only the Next Argument
The terminator is recognized inside FlagSet.parseOne():
func (f *FlagSet) parseOne() (bool, error) {
if len(f.args) == 0 {
return false, nil
}
s := f.args[0]
if len(s) < 2 || s[0] != '-' {
return false, nil
}
numMinuses := 1
if s[1] == '-' {
numMinuses++
if len(s) == 2 { // "--" terminates the flags
f.args = f.args[1:]
return false, nil
}
}
// Normal flag parsing continues.
}
This code makes several decisions in sequence.
First, it checks whether any arguments remain:
if len(f.args) == 0 {
return false, nil
}
If not, parsing is complete.
It then looks only at the first remaining argument:
s := f.args[0]
If that string is too short to be a flag or does not begin with -, parsing stops:
if len(s) < 2 || s[0] != '-' {
return false, nil
}
This is why Go's standard flag package also stops at the first positional argument. It does not continue searching later in the slice for additional flags.
If the argument starts with one dash, the parser initially assumes one leading dash:
numMinuses := 1
It then checks for a second:
if s[1] == '-' {
numMinuses++
At that point, the parser knows that the argument begins with --.
But an argument that merely begins with -- is not necessarily the terminator. For example:
--verbose
is a regular flag in Go's flag package.
The terminator requires one additional condition:
if len(s) == 2
The string must contain exactly two characters.
Consequently:
-- terminator
--verbose flag named verbose
---verbose invalid flag syntax
The exact standalone string -- enters this branch:
if len(s) == 2 { // "--" terminates the flags
f.args = f.args[1:]
return false, nil
}
Those two statements implement the behavior.
Removing the Terminator
The first statement advances the remaining argument slice by one element:
f.args = f.args[1:]
Suppose the current state is:
f.args = []string{
"--",
"-output",
"report.txt",
}
After the assignment, it becomes:
f.args = []string{
"-output",
"report.txt",
}
No strings after -- are modified.
The parser does not remove their dashes, copy them into a different representation, or mark them individually as positional. It simply removes the first element—--—and leaves the rest of the slice in place.
This is why the terminator itself is absent from flag.Args().
Before:
["--", "-output", "report.txt"]
After:
["-output", "report.txt"]
Telling the Parse Loop to Stop
The second statement returns:
return false, nil
This tells FlagSet.Parse() two things:
- No flag was parsed.
- No error occurred.
Back in the loop:
seen, err := f.parseOne()
if seen {
continue
}
if err == nil {
break
}
Because seen is false, the loop does not continue.
Because err is nil, the loop breaks normally.
Parsing is over.
The parser never calls parseOne() on:
-output
report.txt
That is the critical effect of --.
It does not transform -output into a positional argument through some special conversion. It prevents the flag parser from examining -output at all.
Once parsing stops, every string still present in f.args is treated as an unprocessed argument.
Retrieving the Remaining Arguments
FlagSet.Args() simply returns the parser's remaining slice:
func (f *FlagSet) Args() []string {
return f.args
}
The package-level flag.Args() delegates to the default CommandLine flag set:
func Args() []string {
return CommandLine.args
}
Therefore, after parsing:
$ ./example -verbose -- -output report.txt
the state is effectively:
verbose == true
output == ""
flag.Args() == []string{"-output", "report.txt"}
The final arguments are not “un-dashed.” They remain exactly as the program received them.
The only removed element is the terminator itself.
Following the Slice Step by Step
For this command:
$ ./example -verbose -- -output report.txt
flag.Parse() begins with:
f.args = []string{
"-verbose",
"--",
"-output",
"report.txt",
}
First call to parseOne()
The first argument is:
s == "-verbose"
It is recognized as a flag.
The parser consumes it:
f.args = []string{
"--",
"-output",
"report.txt",
}
It returns:
true, nil
The parse loop continues.
Second call to parseOne()
The next argument is:
s == "--"
The parser detects that:
s[0] == '-'
s[1] == '-'
len(s) == 2
It removes the terminator:
f.args = f.args[1:]
The remaining state becomes:
f.args = []string{
"-output",
"report.txt",
}
It then returns:
false, nil
The parse loop stops.
Final state
The parser never examines the next string as a possible flag.
The program receives:
flag.Args() == []string{
"-output",
"report.txt",
}
That is the entire implementation of the boundary.
--flag and -- Take Different Paths
The standalone terminator should not be confused with Go's double-dash spelling for ordinary flags.
The flag package permits both:
$ ./example -verbose
and:
$ ./example --verbose
In this package, one or two leading dashes are equivalent for named flags.
The parser starts with:
numMinuses := 1
If it finds a second dash, it increments the count:
if s[1] == '-' {
numMinuses++
}
For:
-verbose
the flag name starts after one dash:
name := s[1:]
For:
--verbose
the flag name starts after two:
name := s[2:]
Both produce:
verbose
The exact string -- is special because nothing follows the two dashes.
The distinction is therefore not:
one dash means flag
two dashes mean terminator
It is:
-verbose ordinary flag
--verbose ordinary flag
-- terminator
The parser determines the difference from the complete string, not merely from its prefix.
Go Also Stops at the First Non-Flag Argument
There is another important boundary in the same function:
if len(s) < 2 || s[0] != '-' {
return false, nil
}
Suppose the command is:
$ ./example alpha -verbose
The first remaining argument is:
alpha
It does not begin with -, so parseOne() returns:
false, nil
The parse loop stops immediately.
As a result:
verbose == false
flag.Args() == []string{"alpha", "-verbose"}
The later -verbose is never parsed as a flag.
This differs from parsers that allow flags and positional arguments to be interspersed:
$ some-command alpha -verbose beta
Go's standard flag package does not search the entire argument slice and rearrange it. It parses consecutive flags from the beginning and stops at either:
- the first non-flag argument, or
- the standalone
--.
The package documentation states this rule directly: flag parsing stops before the first non-flag argument or after the terminator --.
A Non-Boolean Flag Can Consume a Dash-Prefixed Value
The parser's behavior also depends on whether the current flag requires a value.
Consider:
$ ./example -output -- -verbose
Because output is a string flag, it requires a value. Once the parser recognizes -output, it takes the next argument as that value:
value, f.args = f.args[0], f.args[1:]
The next argument happens to be:
--
But at that moment, the parser is not looking for another flag. It is looking for the value belonging to -output.
The result is therefore:
output == "--"
flag.Args() == []string{"-verbose"}
Here, -- does not terminate flag parsing. It is consumed as the value of -output.
Compare these commands:
$ ./example -- -verbose
Result:
output == ""
flag.Args() == []string{"-verbose"}
But:
$ ./example -output -- -verbose
Result:
output == "--"
flag.Args() == []string{"-verbose"}
The string is identical. Its meaning depends on the parser's current state.
In the first command, the parser is looking for the next flag.
In the second, it is looking for the value of the already-recognized -output flag.
This is another reason -- should not be understood as a universal token with meaning independent of context. It is interpreted by a parser at a particular point in that parser's control flow.
A Minimal Reimplementation
The essential behavior can be demonstrated without the rest of the flag package:
package main
import "fmt"
func splitAtDoubleDash(args []string) (before, after []string) {
for i, arg := range args {
if arg == "--" {
return args[:i], args[i+1:]
}
}
return args, nil
}
func main() {
args := []string{
"-verbose",
"--",
"-output",
"report.txt",
}
before, after := splitAtDoubleDash(args)
fmt.Printf("before: %#v\n", before)
fmt.Printf("after: %#v\n", after)
}
Output:
before: []string{"-verbose"}
after: []string{"-output", "report.txt"}
Go's actual implementation is slightly different because it parses one flag at a time. It does not first search the entire slice for --.
But the central operation is the same:
args = args[1:]
Consume the terminator, then stop interpreting the remaining strings as flags.
The Shell Still Matters Before the Program Starts
Although the shell does not implement this particular -- boundary, it still constructs the argument vector.
For example:
$ ./example -- *.txt
The shell normally expands *.txt before launching the program.
If the directory contains:
a.txt
b.txt
the program may receive:
[]string{
"./example",
"--",
"a.txt",
"b.txt",
}
It does not receive the literal string:
*.txt
unless the wildcard is quoted or otherwise protected:
$ ./example -- '*.txt'
That produces an argument vector resembling:
[]string{
"./example",
"--",
"*.txt",
}
The order of events is:
- The shell parses its own command syntax.
- The shell performs expansions such as globbing.
- The shell starts the program with a list of strings.
- Go exposes those strings through
os.Args. - The
flagpackage interprets some of those strings. - The
flagpackage recognizes--and stops parsing flags.
-- controls step 5. It does not reverse or suppress work the shell already performed in steps 1 and 2.
Why the Implementation Is So Small
There is no complicated mechanism behind -- because none is necessary.
The parser already maintains a slice of unprocessed arguments:
f.args
It already has a return value that tells the outer loop whether parsing should continue:
seen bool
Supporting the terminator requires only two actions:
f.args = f.args[1:]
return false, nil
The first consumes the boundary marker.
The second exits the flag-parsing loop without reporting an error.
Everything after that follows from the parser's existing structure.
-- Is a Parser Convention Made Concrete
At the command line, -- can appear to be part of Unix itself.
In Go's implementation, its actual nature is visible.
It begins as an ordinary string:
"--"
It arrives inside an ordinary slice:
os.Args
It is passed to an ordinary function:
CommandLine.Parse(os.Args[1:])
That function eventually performs an ordinary comparison:
if len(s) == 2
The parser advances an ordinary slice:
f.args = f.args[1:]
Then it breaks an ordinary loop:
return false, nil
That is what it means for a program to “support --.”
A program can support it, ignore it, reject it, or assign it some other meaning. Go's standard flag package supports the conventional option-terminator behavior because its parser explicitly contains code implementing that convention.
The convention is widespread.
The implementation is local.
And the boundary exists only because the parser chooses to recognize it.
Source References
flag.Parsepassesos.Args[1:]to the defaultCommandLineflag set.FlagSet.Parserepeatedly callsparseOne.FlagSet.parseOnerecognizes the standalone--, removes it fromf.args, and ends flag parsing.- Package documentation for
flagdescribes the accepted flag syntax and parsing boundary.
Comments
No comments yet. Be the first!