Examples
The following examples illustrate both the usage and the utility of
staged-script. They build on each other in complexity, so it will
make the most sense to start here at the top and work your way down
through each in turn.
The Basics
We’ll start with what amounts to a “Hello World” example, with two simple stages to say “hello” and “goodbye”.
example/ex_0_the_basics.py 1
2import sys
3from typing import List
4
5from staged_script import StagedScript
6
7
8class MyScript(StagedScript):
9 @StagedScript.stage("hello", "Greeting the user")
10 def say_hello(self) -> None:
11 self.run("echo 'Hello World'", shell=True)
12
13 @StagedScript.stage("goodbye", "Bidding farewell")
14 def say_goodbye(self) -> None:
15 self.run("echo 'Goodbye World'", shell=True)
16
17 def main(self, argv: List[str]) -> None:
18 self.parse_args(argv)
19 try:
20 self.say_hello()
21 self.say_goodbye()
22 finally:
23 self.print_script_execution_summary()
24
25
26if __name__ == "__main__":
27 my_script = MyScript({"hello", "goodbye"})
28 my_script.main(sys.argv[1:])
The two methods say_hello() and say_goodbye() are stand-ins for
whatever you might want to do in the stages of your script. Note that
in this case they are simply running commands in the underlying
shell, but you could instead include
whatever Python code you want in here.
The main() method specifies the general form of the script, where
you first parse the command line arguments, then try to execute a series of
stages (lines 20–21), and finally print the script execution
summary, regardless of whether
anything went wrong in any of the stages. You’re welcome to construct
your scripts however you like—this general form is just a
recommendation.
Running the script, and passing the --help argument to it, yields
the following:
$ python3 ../../example/ex_0_the_basics.py --help
usage: ex_0_the_basics.py [-h] [--stage {goodbye,hello} [{goodbye,hello} ...]]
[--dry-run]
[--goodbye-retry-attempts GOODBYE_RETRY_ATTEMPTS]
[--goodbye-retry-delay GOODBYE_RETRY_DELAY]
[--goodbye-retry-timeout GOODBYE_RETRY_TIMEOUT]
[--hello-retry-attempts HELLO_RETRY_ATTEMPTS]
[--hello-retry-delay HELLO_RETRY_DELAY]
[--hello-retry-timeout HELLO_RETRY_TIMEOUT]
This is the description of the ArgumentParser in the StagedScript base class. This should be overridden in your subclass. See the docstring for details.
options:
-h, --help show this help message and exit
--stage {goodbye,hello} [{goodbye,hello} ...]
Which stages to run. (default: None)
--dry-run If specified, don't actually run the commands in the
shell; instead print the commands that would have been
executed. (default: False)
retry:
Additional options for retrying stages.
--goodbye-retry-attempts GOODBYE_RETRY_ATTEMPTS
How many times to retry the 'goodbye' stage. (default:
0)
--goodbye-retry-delay GOODBYE_RETRY_DELAY
How long to wait (in seconds) before retrying the
'goodbye' stage. (default: 0)
--goodbye-retry-timeout GOODBYE_RETRY_TIMEOUT
How long to wait (in seconds) before giving up on
retrying the 'goodbye' stage. (default: 60)
--hello-retry-attempts HELLO_RETRY_ATTEMPTS
How many times to retry the 'hello' stage. (default:
0)
--hello-retry-delay HELLO_RETRY_DELAY
How long to wait (in seconds) before retrying the
'hello' stage. (default: 0)
--hello-retry-timeout HELLO_RETRY_TIMEOUT
How long to wait (in seconds) before giving up on
retrying the 'hello' stage. (default: 60)
If you tell it you only want to run the hello stage, you’ll see
$ python3 ../../example/ex_0_the_basics.py --stage hello
[22:24:22] ╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
Executing: echo 'Hello World' staged_script.py:827
Hello World
`hello` stage duration: 0:00:00.009902 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
Skipping this stage. staged_script.py:440
`goodbye` stage duration: 0:00:00.003351 staged_script.py:475
───────────────── ex_0_the_basics.py Script Execution Summary ──────────────────
staged_script.py:920
➤ Ran the following:
ex_0_the_basics.py \
--stage hello \
--hello-retry-attempts 0 \
--hello-retry-delay 0 \
--hello-retry-timeout 60 \
--goodbye-retry-attempts 0 \
--goodbye-retry-delay 0 \
--goodbye-retry-timeout 60
➤ Commands executed:
echo 'Hello World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.009902 │
│ goodbye │ 0:00:00.003351 │
├─────────┼────────────────┤
│ Total │ 0:00:00.018706 │
└─────────┴────────────────┘
➤ Script result:
Success
─────────────── End ex_0_the_basics.py Script Execution Summary ────────────────
Note
It’s worth noting that StagedScript uses a
rich.console.Console for all output, so you may want to
familiarize yourself with the Rich documentation for the sake of
customizing the StagedScript.console attribute.
Customizing the Parser
Removing the Retry Arguments
For a script as simple as this one, all of the retry business in the
help text and script execution summary are rather distracting, because
our two stages don’t automatically retry themselves. We can
customize the argument parser
by adding the following to the MyScript class:
example/ex_1_removing_the_retry_arguments.py 1
2 @functools.cached_property
3 def parser(self) -> ArgumentParser:
4 my_parser = super().parser
5 my_parser.description = "Demonstrate removing the retry arguments."
6 self.retry_arg_group.title = argparse.SUPPRESS
7 self.retry_arg_group.description = argparse.SUPPRESS
8 self.hello_retry_attempts_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
9 self.hello_retry_delay_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
10 self.hello_retry_timeout_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
11 self.goodbye_retry_attempts_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
12 self.goodbye_retry_delay_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
13 self.goodbye_retry_timeout_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
Note
An upcoming release will refactor the retry argument attributes so
mypy will be happy with them. For now, just use the type:
ignore[attr-defined] comments.
Now when we look at the --help text, we see:
$ python3 ../../example/ex_1_removing_the_retry_arguments.py --help
usage: ex_1_removing_the_retry_arguments.py [-h]
[--stage {hello,goodbye} [{hello,goodbye} ...]]
[--dry-run]
Demonstrate removing the retry arguments.
options:
-h, --help show this help message and exit
--stage {hello,goodbye} [{hello,goodbye} ...]
Which stages to run. (default: None)
--dry-run If specified, don't actually run the commands in the
shell; instead print the commands that would have been
executed. (default: False)
This is much nicer from the user perspective. Now if we run the script with the same arguments as last time, we see:
$ python3 ../../example/ex_1_removing_the_retry_arguments.py --stage hello
[22:24:22] ╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
Executing: echo 'Hello World' staged_script.py:827
Hello World
`hello` stage duration: 0:00:00.009963 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
Skipping this stage. staged_script.py:440
`goodbye` stage duration: 0:00:00.003428 staged_script.py:475
──────── ex_1_removing_the_retry_arguments.py Script Execution Summary ─────────
staged_script.py:920
➤ Ran the following:
ex_1_removing_the_retry_arguments.py \
--stage hello
➤ Commands executed:
echo 'Hello World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.009963 │
│ goodbye │ 0:00:00.003428 │
├─────────┼────────────────┤
│ Total │ 0:00:00.018801 │
└─────────┴────────────────┘
➤ Script result:
Success
────── End ex_1_removing_the_retry_arguments.py Script Execution Summary ───────
Running Certain Stages by Default
By default, a staged script won’t run any stages unless you tell it to.
Keep in mind, this package was designed for improving replicability for
infrastructure automation, so we err on the side of explicit is better
than implicit. That said, you or your users may find this
inconvenient, and would prefer to make it such that certain stages run
by default when you don’t specify --stage on the command line. In
that case, you can add the highlighted line:
example/ex_2_running_certain_stages_by_default.py1
2 @functools.cached_property
3 def parser(self) -> ArgumentParser:
4 self.goodbye_retry_timeout_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
5 my_parser.set_defaults(stage=list(self.stages))
In this case I’m telling it to default the --stage argument to the
list of all the stages registered when instantiating a StagedScript
subclass. However, you could just as easily specify whatever subset of
stages you like here with, e.g., stage=["stage-1", "stage-2"]. Now,
if we run the script without any arguments, we see:
$ python3 ../../example/ex_2_running_certain_stages_by_default.py
[22:24:23] ╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
Executing: echo 'Hello World' staged_script.py:827
Hello World
`hello` stage duration: 0:00:00.010216 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
Executing: echo 'Goodbye World' staged_script.py:827
Goodbye World
`goodbye` stage duration: 0:00:00.004551 staged_script.py:475
────── ex_2_running_certain_stages_by_default.py Script Execution Summary ──────
staged_script.py:920
➤ Ran the following:
ex_2_running_certain_stages_by_default.py \
--stage hello goodbye
➤ Commands executed:
echo 'Hello World'
echo 'Goodbye World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.010216 │
│ goodbye │ 0:00:00.004551 │
├─────────┼────────────────┤
│ Total │ 0:00:00.020164 │
└─────────┴────────────────┘
➤ Script result:
Success
──── End ex_2_running_certain_stages_by_default.py Script Execution Summary ────
Note
It’s worth noting that the order of the stage names passed to the
--stages argument on the command line does not affect the order
in which the stages are run. That is governed by the contents of the
main() method, as shown in the basic example.
Adding Arguments
Now let’s see about adding some arguments to the parser beyond what
StagedScript provides. We can do this by adding arguments to the
argparse.ArgumentParser as you normally would.
example/ex_3_adding_arguments.py 1
2 @functools.cached_property
3 def parser(self) -> ArgumentParser:
4 my_parser.set_defaults(stage=list(self.stages))
5 my_parser.add_argument(
6 "--some-file",
7 required=True,
8 type=Path,
9 help="Some file your users need to point to for the script "
10 "to run.",
11 )
12 my_parser.add_argument(
13 "--some-flag",
14 action="store_true",
15 help="Some flag your users can toggle on if they like.",
16 )
Beyond that, though, you likely also want to augment the parsing of the arguments to handle these new options. You can do so by extending the parse_args() method provided by the base class.
example/ex_3_adding_arguments.py 1
2 def parse_args(self, argv: List[str]) -> None:
3 # The base class saves the parsed arguments as `self.args`.
4 super().parse_args(argv)
5
6 # If you like, you may wish to transfer some subset of the added
7 # arguments to instance attributes for convenience.
8 self.flag = self.args.some_flag
9
10 # You may also wish to do additional post-processing of certain
11 # arguments, whether you save them as instance attributes or
12 # not.
Note
Here’s we’re taking the file the user gives us on the command line and resolving it to an absolute path. This is again in service of improving replicability, as the script output will then point us directly to the exact file used at execution time, instead of just giving us a file name or relative path.
For the sake of using these new arguments in our script, let’s modify the two stages to take them into account.
example/ex_3_adding_arguments.py 1class MyScript(StagedScript):
2 @StagedScript.stage("hello", "Greeting the user")
3 def say_hello(self) -> None:
4 self.run("echo 'Hello World'", shell=True)
5 self.console.log(f"Processing file: {self.args.some_file}")
6
7 @StagedScript.stage("goodbye", "Bidding farewell")
8 def say_goodbye(self) -> None:
9 self.run("echo 'Goodbye World'", shell=True)
10 self.console.log(
11 "Some flag was " + ("not " if not self.flag else "") + "set!"
Now if we run the script, passing in a file on the command line, we see:
$ python3 ../../example/ex_3_adding_arguments.py --some-file foo.txt
[22:24:23] ╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
Executing: echo 'Hello World' staged_script.py:827
Hello World
Processing file: ex_3_adding_arguments.py:24
/home/docs/checkouts/readthedocs.org/user
_builds/staged-script/checkouts/v1.0.2/do
c/source/foo.txt
`hello` stage duration: 0:00:00.012157 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
Executing: echo 'Goodbye World' staged_script.py:827
Goodbye World
Some flag was not set! ex_3_adding_arguments.py:29
`goodbye` stage duration: 0:00:00.006115 staged_script.py:475
────────────── ex_3_adding_arguments.py Script Execution Summary ───────────────
staged_script.py:920
➤ Ran the following:
ex_3_adding_arguments.py \
--stage goodbye hello \
--some-file
/home/docs/checkouts/readthedocs.org/user_bu
ilds/staged-script/checkouts/v1.0.2/doc/sour
ce/foo.txt
➤ Commands executed:
echo 'Hello World'
echo 'Goodbye World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.012157 │
│ goodbye │ 0:00:00.006115 │
├─────────┼────────────────┤
│ Total │ 0:00:00.024123 │
└─────────┴────────────────┘
➤ Script result:
Success
──────────── End ex_3_adding_arguments.py Script Execution Summary ─────────────
Customizing Stage Behavior
For All Stages
We now move beyond customizing the parser to adjusting the behavior of the stages as they run. Recall from The Conceptual Stage that a stage is broken down into a number of phases, each of which has a corresponding method. These phase methods have reasonable default implementations in the base class, but may be extended or overridden in your subclasses.
example/ex_4_customizing_stage_behavior.py 1
2 def _run_pre_stage_actions(self) -> None:
3 # You can extend the default implementation by calling it via
4 # `super()` first.
5 super()._run_pre_stage_actions()
6 self.console.log("Checking to make sure it's safe to run a stage...")
7
8 def _skip_stage(self) -> None:
9 # You can override the default implementation if you don't like
10 # it by simply omitting the `super()` call.
11 self.console.log(
12 f"You didn't tell me to run the '{self.current_stage}' stage."
13 )
14
15 def _end_stage(self) -> None:
16 super()._end_stage()
17 self.print_heading(f"Finished stage '{self.current_stage}'.")
18
19 def _run_post_stage_actions(self) -> None:
20 super()._run_post_stage_actions()
21 self.console.log(
22 "Checking to make sure all is well after running the stage..."
Note
The _begin_stage() method is not included above, meaning it is
neither extended nor overridden, so the base class implementation
will be used.
Now when we run the script, we can see the changed behavior:
$ python3 ../../example/ex_4_customizing_stage_behavior.py --some-file foo.txt --some-flag --stage goodbye
[22:24:23] Checking to make sure it's safe ex_4_customizing_stage_behavior.py:77
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
You didn't tell me to run the ex_4_customizing_stage_behavior.py:82
'hello' stage.
`hello` stage duration: 0:00:00.004144 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'hello'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_4_customizing_stage_behavior.py:92
well after running the stage...
Checking to make sure it's safe ex_4_customizing_stage_behavior.py:77
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
Executing: echo 'Goodbye World' staged_script.py:827
Goodbye World
Some flag was set! ex_4_customizing_stage_behavior.py:29
`goodbye` stage duration: 0:00:00.006303 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'goodbye'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_4_customizing_stage_behavior.py:92
well after running the stage...
───────── ex_4_customizing_stage_behavior.py Script Execution Summary ──────────
staged_script.py:920
➤ Ran the following:
ex_4_customizing_stage_behavior.py \
--stage goodbye \
--some-file
/home/docs/checkouts/readthedocs.org/user_bu
ilds/staged-script/checkouts/v1.0.2/doc/sour
ce/foo.txt \
--some-flag
➤ Commands executed:
echo 'Goodbye World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.004144 │
│ goodbye │ 0:00:00.006303 │
├─────────┼────────────────┤
│ Total │ 0:00:00.032313 │
└─────────┴────────────────┘
➤ Script result:
Success
─────── End ex_4_customizing_stage_behavior.py Script Execution Summary ────────
You’re free to customize the phases of the stage however you like, but generally speaking the phases are useful for doing the following:
Phase |
Actions |
|---|---|
Pre-Stage Actions |
Checking pre-conditions and erroring appropriately if not met |
Begin Stage Actions |
Telling the user what’s about to happen and capturing state |
Skip Stage Actions |
Telling the user what’s happening and why |
End Stage Actions |
Capturing state; potentially displaying information |
Post-Stage Actions |
Checking post-conditions and erroring appropriately if not met |
For Individual Stages
In addition to tweaking the phase implementations for all stages, you
also have the flexibility to tailor things on a stage-by-stage
basis. This can be particularly helpful, e.g.,
in the Pre- and Post-Stage Actions for checking the pre- and
post-conditions specific to each stage. To customize a phase method for
a particular stage, you just need to define a phase method as before,
but then append _STAGE_NAME to the method name, where STAGE_NAME
is the name of the stage as provided to the StagedScript.stage() decorator.
example/ex_5_customizing_individual_stages.py 1
2 def _begin_stage_hello(self, heading: str) -> None:
3 # You can use whatever `_begin_stage()` method already exists,
4 # either whatever's been overridden or extended in the current
5 # class, or whatever's provided by the base class, and then
6 # extend it.
7 self._begin_stage(heading)
8 self.console.log("The first stage is underway...")
9
10 def _end_stage_goodbye(self) -> None:
11 # Or you can ignore whatever's been overridden/extended in the
12 # current class, and fall back to what's provided by the base
13 # class, and then extend it.
14 super()._end_stage()
15 self.print_heading(f"Finished the final stage: {self.current_stage}")
16
17 # You can also override things completely by omitting any `self`
18 # or `super()` calls to the default method for the corresponding
Now when we run both stages we see:
$ python3 ../../example/ex_5_customizing_individual_stages.py --some-file foo.txt
[22:24:24] Checking to make sure it's ex_5_customizing_individual_stages.py:77
safe to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
The first stage is ex_5_customizing_individual_stages.py:102
underway...
Executing: echo 'Hello World' staged_script.py:827
Hello World
Processing file: ex_5_customizing_individual_stages.py:24
/home/docs/checkouts/readthe
docs.org/user_builds/staged-
script/checkouts/v1.0.2/doc/
source/foo.txt
`hello` stage duration: 0:00:00.009217 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'hello'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_5_customizing_individual_stages.py:92
well after running the
stage...
Checking to make sure it's ex_5_customizing_individual_stages.py:77
safe to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
Executing: echo 'Goodbye World' staged_script.py:827
Goodbye World
Some flag was not set! ex_5_customizing_individual_stages.py:29
`goodbye` stage duration: 0:00:00.006187 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished the final stage: goodbye │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_5_customizing_individual_stages.py:92
well after running the
stage...
──────── ex_5_customizing_individual_stages.py Script Execution Summary ────────
staged_script.py:920
➤ Ran the following:
ex_5_customizing_individual_stages.py \
--stage goodbye hello \
--some-file
/home/docs/checkouts/readthedocs.org/user_bu
ilds/staged-script/checkouts/v1.0.2/doc/sour
ce/foo.txt
➤ Commands executed:
echo 'Hello World'
echo 'Goodbye World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.009217 │
│ goodbye │ 0:00:00.006187 │
├─────────┼────────────────┤
│ Total │ 0:00:00.037262 │
└─────────┴────────────────┘
➤ Script result:
Success
────── End ex_5_customizing_individual_stages.py Script Execution Summary ──────
For Retryable Stages
Now that we’re familiar with the process of customizing stage behavior, we can move on to introducing a retryable phase. For demonstration purposes, we’ll simply add a stage that fails twice before it passes, but in reality you would want to check on how things went when executing a particular stage, and if something happens where a human would look at it and say, “Yes, I understand what happened here; we just need to try again and it should pass,” you can program that logic into the stage itself.
Warning
You don’t want to use this feature to hide problems with your actual code base. If there’s flakiness in your application or infrastructure that’s your fault, you want those problems in your face to incentivize fixing them. However, if there’s flakiness due to various things outside your control (e.g., corporate or external infrastructure that may be unreachable from time to time), this can be a helpful way of hiding such problems from your team and avoiding the “everyone hates CI because it randomly fails” problem.
There’s a good deal to add to our script at this point, so let’s walk
through the pieces one at a time. First we’ll add an __init__()
method, but that’s just so we can keep track of the number of times our
flaky stage has been run.
example/ex_6_creating_retryable_stages.py 1class MyScript(StagedScript):
2 def __init__(
3 self,
4 stages: Set[str],
5 *,
6 console_force_terminal: Optional[bool] = None,
7 console_log_path: bool = True,
8 print_commands: bool = True,
9 ):
10 super().__init__(
11 stages,
12 console_force_terminal=console_force_terminal,
13 console_log_path=console_log_path,
14 print_commands=print_commands,
15 )
Next we’ll add the flaky stage itself. The keys here are raising the
RetryStage exception whenever you detect
something where a human would say, “Just try again,” and setting
self.script_success appropriately.
example/ex_6_creating_retryable_stages.py 1
2 @StagedScript.stage("flaky", "Trying an error-prone operation")
3 def try_error_prone_operation(self) -> None:
4 self.num_times_flaky_run += 1
5 num_times_to_fail = 2
6 if self.num_times_flaky_run <= num_times_to_fail:
7 self.console.log("[red]Oh no! Something went horribly wrong!")
8 self.script_success = False
9 raise RetryStage
10 self.console.log("[green]Thank goodness, everything worked this time.")
Next we need to adjust the parser to account for this new stage.
example/ex_6_creating_retryable_stages.py 1
2 @functools.cached_property
3 def parser(self) -> ArgumentParser:
4 my_parser = super().parser
5 my_parser.description = "Demonstrate adding arguments to the parser."
6 self.hello_retry_attempts_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
7 self.hello_retry_delay_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
8 self.hello_retry_timeout_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
9 self.goodbye_retry_attempts_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
10 self.goodbye_retry_delay_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
11 self.goodbye_retry_timeout_arg.help = argparse.SUPPRESS # type: ignore[attr-defined]
12 my_parser.set_defaults(
13 stage=list(self.stages),
14 flaky_retry_attempts=5,
15 flaky_retry_delay=1,
16 )
17 my_parser.add_argument(
18 "--some-file",
19 required=True,
20 type=Path,
21 help="Some file your users need to point to for the script "
22 "to run.",
23 )
24 my_parser.add_argument(
25 "--some-flag",
26 action="store_true",
27 help="Some flag your users can toggle on if they like.",
28 )
Note that we’ve removed the lines
self.retry_arg_group.title = argparse.SUPPRESS
self.retry_arg_group.description = argparse.SUPPRESS
so that when we look at the help text, we see:
$ python3 ../../example/ex_6_creating_retryable_stages.py --help
usage: ex_6_creating_retryable_stages.py [-h]
[--stage {flaky,goodbye,hello} [{flaky,goodbye,hello} ...]]
[--dry-run]
[--flaky-retry-attempts FLAKY_RETRY_ATTEMPTS]
[--flaky-retry-delay FLAKY_RETRY_DELAY]
[--flaky-retry-timeout FLAKY_RETRY_TIMEOUT]
--some-file SOME_FILE [--some-flag]
Demonstrate adding arguments to the parser.
options:
-h, --help show this help message and exit
--stage {flaky,goodbye,hello} [{flaky,goodbye,hello} ...]
Which stages to run. (default: ['flaky', 'goodbye',
'hello'])
--dry-run If specified, don't actually run the commands in the
shell; instead print the commands that would have been
executed. (default: False)
--some-file SOME_FILE
Some file your users need to point to for the script
to run. (default: None)
--some-flag Some flag your users can toggle on if they like.
(default: False)
retry:
Additional options for retrying stages.
--flaky-retry-attempts FLAKY_RETRY_ATTEMPTS
How many times to retry the 'flaky' stage. (default:
5)
--flaky-retry-delay FLAKY_RETRY_DELAY
How long to wait (in seconds) before retrying the
'flaky' stage. (default: 1)
--flaky-retry-timeout FLAKY_RETRY_TIMEOUT
How long to wait (in seconds) before giving up on
retrying the 'flaky' stage. (default: 60)
Now when we run all the stages, we see:
$ python3 ../../example/ex_6_creating_retryable_stages.py --some-file foo.txt
[22:24:24] Checking to make sure it's safe ex_6_creating_retryable_stages.py:106
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
The first stage is underway... ex_6_creating_retryable_stages.py:131
Executing: echo 'Hello World' staged_script.py:827
Hello World
Processing file: ex_6_creating_retryable_stages.py:40
/home/docs/checkouts/readthedocs
.org/user_builds/staged-script/c
heckouts/v1.0.2/doc/source/foo.t
xt
`hello` stage duration: 0:00:00.009153 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'hello'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_6_creating_retryable_stages.py:121
well after running the stage...
Checking to make sure it's safe ex_6_creating_retryable_stages.py:106
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Trying an error-prone operation │
╰──────────────────────────────────────────────╯
Oh no! Something went horribly ex_6_creating_retryable_stages.py:47
wrong!
`flaky` stage duration: 0:00:00.003776 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'flaky'. │
╰──────────────────────────────────────────────╯
Preparing to retry the 'flaky' stage... staged_script.py:546
<RetryCallState 139938439085280: attempt #1;
slept for 1.0; last result: failed (RetryStage
)>
[22:24:25] ╭──────────────────────────────────────────────╮ staged_script.py:937
│ Trying an error-prone operation │
╰──────────────────────────────────────────────╯
Oh no! Something went horribly ex_6_creating_retryable_stages.py:47
wrong!
`flaky` stage duration: 0:00:00.003990 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'flaky'. │
╰──────────────────────────────────────────────╯
Preparing to retry the 'flaky' stage... staged_script.py:546
<RetryCallState 139938439085280: attempt #2;
slept for 2.0; last result: failed (RetryStage
)>
[22:24:26] ╭──────────────────────────────────────────────╮ staged_script.py:937
│ Trying an error-prone operation │
╰──────────────────────────────────────────────╯
Thank goodness, everything ex_6_creating_retryable_stages.py:50
worked this time.
`flaky` stage duration: 0:00:00.004159 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'flaky'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_6_creating_retryable_stages.py:121
well after running the stage...
Checking to make sure it's safe ex_6_creating_retryable_stages.py:106
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
Executing: echo 'Goodbye World' staged_script.py:827
Goodbye World
Some flag was not set! ex_6_creating_retryable_stages.py:56
`goodbye` stage duration: 0:00:00.006242 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished the final stage: goodbye │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_6_creating_retryable_stages.py:121
well after running the stage...
────────── ex_6_creating_retryable_stages.py Script Execution Summary ──────────
staged_script.py:920
➤ Ran the following:
ex_6_creating_retryable_stages.py \
--stage hello goodbye flaky \
--flaky-retry-attempts 5 \
--flaky-retry-delay 1 \
--flaky-retry-timeout 60 \
--some-file
/home/docs/checkouts/readthedocs.org/user_bu
ilds/staged-script/checkouts/v1.0.2/doc/sour
ce/foo.txt
➤ Commands executed:
echo 'Hello World'
echo 'Goodbye World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.009153 │
│ flaky │ 0:00:00.003776 │
│ flaky │ 0:00:00.003990 │
│ flaky │ 0:00:00.004159 │
│ goodbye │ 0:00:00.006242 │
├─────────┼────────────────┤
│ Total │ 0:00:02.069642 │
└─────────┴────────────────┘
➤ Script result:
Success
──────── End ex_6_creating_retryable_stages.py Script Execution Summary ────────
Note that it took three tries before the flaky stage finally passed. If, however, the user is impatient and doesn’t want to retry the stage, we’d see:
$ python3 ../../example/ex_6_creating_retryable_stages.py --some-file foo.txt --flaky-retry-attempts 0
[22:24:27] Checking to make sure it's safe ex_6_creating_retryable_stages.py:106
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
The first stage is underway... ex_6_creating_retryable_stages.py:131
Executing: echo 'Hello World' staged_script.py:827
Hello World
Processing file: ex_6_creating_retryable_stages.py:40
/home/docs/checkouts/readthedocs
.org/user_builds/staged-script/c
heckouts/v1.0.2/doc/source/foo.t
xt
`hello` stage duration: 0:00:00.009169 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'hello'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_6_creating_retryable_stages.py:121
well after running the stage...
Checking to make sure it's safe ex_6_creating_retryable_stages.py:106
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Trying an error-prone operation │
╰──────────────────────────────────────────────╯
Oh no! Something went horribly ex_6_creating_retryable_stages.py:47
wrong!
`flaky` stage duration: 0:00:00.003815 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'flaky'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_6_creating_retryable_stages.py:121
well after running the stage...
Checking to make sure it's safe ex_6_creating_retryable_stages.py:106
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
Executing: echo 'Goodbye World' staged_script.py:827
Goodbye World
Some flag was not set! ex_6_creating_retryable_stages.py:56
`goodbye` stage duration: 0:00:00.006188 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished the final stage: goodbye │
╰──────────────────────────────────────────────╯
Checking to make sure all is ex_6_creating_retryable_stages.py:121
well after running the stage...
────────── ex_6_creating_retryable_stages.py Script Execution Summary ──────────
staged_script.py:920
➤ Ran the following:
ex_6_creating_retryable_stages.py \
--stage goodbye hello flaky \
--flaky-retry-attempts 0 \
--flaky-retry-delay 1 \
--flaky-retry-timeout 60 \
--some-file
/home/docs/checkouts/readthedocs.org/user_bu
ilds/staged-script/checkouts/v1.0.2/doc/sour
ce/foo.txt
➤ Commands executed:
echo 'Hello World'
echo 'Goodbye World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.009169 │
│ flaky │ 0:00:00.003815 │
│ goodbye │ 0:00:00.006188 │
├─────────┼────────────────┤
│ Total │ 0:00:00.047898 │
└─────────┴────────────────┘
➤ Script result:
Failure
──────── End ex_6_creating_retryable_stages.py Script Execution Summary ────────
Note that the flaky stage ran only once, and the script tells us of the overall failure at the end of the summary.
Note
The default retry behavior is governed by the retry phase
methods, which can be customized like
any other phase methods in the same way as demonstrated above. The
retry behavior is handled by tenacity.Retrying, so you might want
to familiarize yourself with the Tenacity documentation.
Customizing the Script Execution Summary
The purpose of the script execution summary is to give the user an
overview of what happened while the script was running, providing
sufficient details for them or their teammates to replicate what was run
and ease any debugging that may be necessary. By default,
StagedScript provides all the details you’ve seen in the examples
above, but you have the flexibility to extend the behavior for your subclasses.
example/ex_7_customizing_the_summary.py 1
2 def print_script_execution_summary(
3 self,
4 extra_sections: Optional[Dict[str, str]] = None,
5 ) -> None:
6 extras = {
7 "Machine details": (
8 f"hostname: {socket.gethostname()}\n"
9 f"platform: {platform.platform()}"
10 ),
11 }
12 if extra_sections is not None:
13 extras.update(extra_sections)
In this case we’re adding a new section to the summary with some details about the machine the script was run on. Now when we run just the first stage, we see:
$ python3 ../../example/ex_7_customizing_the_summary.py --some-file foo.txt --stage hello
[22:24:27] Checking to make sure it's safe ex_7_customizing_the_summary.py:108
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Greeting the user │
╰──────────────────────────────────────────────╯
The first stage is underway... ex_7_customizing_the_summary.py:133
Executing: echo 'Hello World' staged_script.py:827
Hello World
Processing file: ex_7_customizing_the_summary.py:42
/home/docs/checkouts/readthedocs.o
rg/user_builds/staged-script/check
outs/v1.0.2/doc/source/foo.txt
`hello` stage duration: 0:00:00.009193 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'hello'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is well ex_7_customizing_the_summary.py:123
after running the stage...
Checking to make sure it's safe ex_7_customizing_the_summary.py:108
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Trying an error-prone operation │
╰──────────────────────────────────────────────╯
You didn't tell me to run the ex_7_customizing_the_summary.py:113
'flaky' stage.
`flaky` stage duration: 0:00:00.003802 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished stage 'flaky'. │
╰──────────────────────────────────────────────╯
Checking to make sure all is well ex_7_customizing_the_summary.py:123
after running the stage...
Checking to make sure it's safe ex_7_customizing_the_summary.py:108
to run a stage...
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Bidding farewell │
╰──────────────────────────────────────────────╯
You didn't tell me to run the ex_7_customizing_the_summary.py:113
'goodbye' stage.
`goodbye` stage duration: 0:00:00.003583 staged_script.py:475
╭──────────────────────────────────────────────╮ staged_script.py:937
│ Finished the final stage: goodbye │
╰──────────────────────────────────────────────╯
Checking to make sure all is well ex_7_customizing_the_summary.py:123
after running the stage...
─────────── ex_7_customizing_the_summary.py Script Execution Summary ───────────
staged_script.py:920
➤ Ran the following:
ex_7_customizing_the_summary.py \
--stage hello \
--flaky-retry-attempts 5 \
--flaky-retry-delay 1 \
--flaky-retry-timeout 60 \
--some-file
/home/docs/checkouts/readthedocs.org/user_bu
ilds/staged-script/checkouts/v1.0.2/doc/sour
ce/foo.txt
➤ Commands executed:
echo 'Hello World'
➤ Timing results:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Stage ┃ Duration ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ hello │ 0:00:00.009193 │
│ flaky │ 0:00:00.003802 │
│ goodbye │ 0:00:00.003583 │
├─────────┼────────────────┤
│ Total │ 0:00:00.047636 │
└─────────┴────────────────┘
➤ Script result:
Success
➤ Machine details:
hostname:
build-24899160-project-1073165-staged-script
platform:
Linux-5.19.0-1028-aws-x86_64-with-glibc2.35
───────── End ex_7_customizing_the_summary.py Script Execution Summary ─────────