Skip to content

Commit 050acae

Browse files
zoobazooba
committed
Issue #6135: Adds encoding and errors parameters to subprocess
1 parent 173a1f3 commit 050acae

5 files changed

Lines changed: 154 additions & 100 deletions

File tree

‎Doc/library/subprocess.rst‎

Lines changed: 63 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,8 @@ compatibility with older versions, see the :ref:`call-function-trio` section.
3838

3939

4040
.. function:: run(args, *, stdin=None, input=None, stdout=None, stderr=None,\
41-
shell=False, timeout=None, check=False)
41+
shell=False, timeout=None, check=False, \
42+
encoding=None, errors=None)
4243

4344
Run the command described by *args*. Wait for command to complete, then
4445
return a :class:`CompletedProcess` instance.
@@ -60,15 +61,20 @@ compatibility with older versions, see the :ref:`call-function-trio` section.
6061

6162
The *input* argument is passed to :meth:`Popen.communicate` and thus to the
6263
subprocess's stdin. If used it must be a byte sequence, or a string if
63-
``universal_newlines=True``. When used, the internal :class:`Popen` object
64-
is automatically created with ``stdin=PIPE``, and the *stdin* argument may
65-
not be used as well.
64+
*encoding* or *errors* is specified or *universal_newlines* is True. When
65+
used, the internal :class:`Popen` object is automatically created with
66+
``stdin=PIPE``, and the *stdin* argument may not be used as well.
6667

6768
If *check* is True, and the process exits with a non-zero exit code, a
6869
:exc:`CalledProcessError` exception will be raised. Attributes of that
6970
exception hold the arguments, the exit code, and stdout and stderr if they
7071
were captured.
7172

73+
If *encoding* or *errors* are specified, or *universal_newlines* is True,
74+
file objects for stdin, stdout and stderr are opened in text mode using the
75+
specified *encoding* and *errors* or the :class:`io.TextIOWrapper` default.
76+
Otherwise, file objects are opened in binary mode.
77+
7278
Examples::
7379

7480
>>> subprocess.run(["ls", "-l"]) # doesn't capture output
@@ -85,6 +91,10 @@ compatibility with older versions, see the :ref:`call-function-trio` section.
8591

8692
.. versionadded:: 3.5
8793

94+
.. versionchanged:: 3.6
95+
96+
Added *encoding* and *errors* parameters
97+
8898
.. class:: CompletedProcess
8999

90100
The return value from :func:`run`, representing a process that has finished.
@@ -104,8 +114,8 @@ compatibility with older versions, see the :ref:`call-function-trio` section.
104114
.. attribute:: stdout
105115

106116
Captured stdout from the child process. A bytes sequence, or a string if
107-
:func:`run` was called with ``universal_newlines=True``. None if stdout
108-
was not captured.
117+
:func:`run` was called with an encoding or errors. None if stdout was not
118+
captured.
109119

110120
If you ran the process with ``stderr=subprocess.STDOUT``, stdout and
111121
stderr will be combined in this attribute, and :attr:`stderr` will be
@@ -114,8 +124,8 @@ compatibility with older versions, see the :ref:`call-function-trio` section.
114124
.. attribute:: stderr
115125

116126
Captured stderr from the child process. A bytes sequence, or a string if
117-
:func:`run` was called with ``universal_newlines=True``. None if stderr
118-
was not captured.
127+
:func:`run` was called with an encoding or errors. None if stderr was not
128+
captured.
119129

120130
.. method:: check_returncode()
121131

@@ -249,19 +259,22 @@ default values. The arguments that are most commonly needed are:
249259
.. index::
250260
single: universal newlines; subprocess module
251261

252-
If *universal_newlines* is ``False`` the file objects *stdin*, *stdout* and
253-
*stderr* will be opened as binary streams, and no line ending conversion is
254-
done.
262+
If *encoding* or *errors* are specified, or *universal_newlines* is True,
263+
the file objects *stdin*, *stdout* and *stderr* will be opened in text
264+
mode using the *encoding* and *errors* specified in the call or the
265+
defaults for :class:`io.TextIOWrapper`.
255266

256-
If *universal_newlines* is ``True``, these file objects
257-
will be opened as text streams in :term:`universal newlines` mode
258-
using the encoding returned by :func:`locale.getpreferredencoding(False)
259-
<locale.getpreferredencoding>`. For *stdin*, line ending characters
260-
``'\n'`` in the input will be converted to the default line separator
261-
:data:`os.linesep`. For *stdout* and *stderr*, all line endings in the
262-
output will be converted to ``'\n'``. For more information see the
263-
documentation of the :class:`io.TextIOWrapper` class when the *newline*
264-
argument to its constructor is ``None``.
267+
For *stdin*, line ending characters ``'\n'`` in the input will be converted
268+
to the default line separator :data:`os.linesep`. For *stdout* and *stderr*,
269+
all line endings in the output will be converted to ``'\n'``. For more
270+
information see the documentation of the :class:`io.TextIOWrapper` class
271+
when the *newline* argument to its constructor is ``None``.
272+
273+
If text mode is not used, *stdin*, *stdout* and *stderr* will be opened as
274+
binary streams. No encoding or line ending conversion is performed.
275+
276+
.. versionadded:: 3.6
277+
Added *encoding* and *errors* parameters.
265278

266279
.. note::
267280

@@ -306,7 +319,8 @@ functions.
306319
stderr=None, preexec_fn=None, close_fds=True, shell=False, \
307320
cwd=None, env=None, universal_newlines=False, \
308321
startupinfo=None, creationflags=0, restore_signals=True, \
309-
start_new_session=False, pass_fds=())
322+
start_new_session=False, pass_fds=(), *, \
323+
encoding=None, errors=None)
310324
311325
Execute a child program in a new process. On POSIX, the class uses
312326
:meth:`os.execvp`-like behavior to execute the child program. On Windows,
@@ -482,10 +496,14 @@ functions.
482496

483497
.. _side-by-side assembly: https://en.wikipedia.org/wiki/Side-by-Side_Assembly
484498

485-
If *universal_newlines* is ``True``, the file objects *stdin*, *stdout*
486-
and *stderr* are opened as text streams in universal newlines mode, as
487-
described above in :ref:`frequently-used-arguments`, otherwise they are
488-
opened as binary streams.
499+
If *encoding* or *errors* are specified, the file objects *stdin*, *stdout*
500+
and *stderr* are opened in text mode with the specified encoding and
501+
*errors*, as described above in :ref:`frequently-used-arguments`. If
502+
*universal_newlines* is ``True``, they are opened in text mode with default
503+
encoding. Otherwise, they are opened as binary streams.
504+
505+
.. versionadded:: 3.6
506+
*encoding* and *errors* were added.
489507

490508
If given, *startupinfo* will be a :class:`STARTUPINFO` object, which is
491509
passed to the underlying ``CreateProcess`` function.
@@ -601,11 +619,12 @@ Instances of the :class:`Popen` class have the following methods:
601619
Interact with process: Send data to stdin. Read data from stdout and stderr,
602620
until end-of-file is reached. Wait for process to terminate. The optional
603621
*input* argument should be data to be sent to the child process, or
604-
``None``, if no data should be sent to the child. The type of *input*
605-
must be bytes or, if *universal_newlines* was ``True``, a string.
622+
``None``, if no data should be sent to the child. If streams were opened in
623+
text mode, *input* must be a string. Otherwise, it must be bytes.
606624

607625
:meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``.
608-
The data will be bytes or, if *universal_newlines* was ``True``, strings.
626+
The data will be strings if streams were opened in text mode; otherwise,
627+
bytes.
609628

610629
Note that if you want to send data to the process's stdin, you need to create
611630
the Popen object with ``stdin=PIPE``. Similarly, to get anything other than
@@ -672,28 +691,30 @@ The following attributes are also available:
672691
.. attribute:: Popen.stdin
673692

674693
If the *stdin* argument was :data:`PIPE`, this attribute is a writeable
675-
stream object as returned by :func:`open`. If the *universal_newlines*
676-
argument was ``True``, the stream is a text stream, otherwise it is a byte
677-
stream. If the *stdin* argument was not :data:`PIPE`, this attribute is
678-
``None``.
694+
stream object as returned by :func:`open`. If the *encoding* or *errors*
695+
arguments were specified or the *universal_newlines* argument was ``True``,
696+
the stream is a text stream, otherwise it is a byte stream. If the *stdin*
697+
argument was not :data:`PIPE`, this attribute is ``None``.
679698

680699

681700
.. attribute:: Popen.stdout
682701

683702
If the *stdout* argument was :data:`PIPE`, this attribute is a readable
684703
stream object as returned by :func:`open`. Reading from the stream provides
685-
output from the child process. If the *universal_newlines* argument was
686-
``True``, the stream is a text stream, otherwise it is a byte stream. If the
687-
*stdout* argument was not :data:`PIPE`, this attribute is ``None``.
704+
output from the child process. If the *encoding* or *errors* arguments were
705+
specified or the *universal_newlines* argument was ``True``, the stream is a
706+
text stream, otherwise it is a byte stream. If the *stdout* argument was not
707+
:data:`PIPE`, this attribute is ``None``.
688708

689709

690710
.. attribute:: Popen.stderr
691711

692712
If the *stderr* argument was :data:`PIPE`, this attribute is a readable
693713
stream object as returned by :func:`open`. Reading from the stream provides
694-
error output from the child process. If the *universal_newlines* argument was
695-
``True``, the stream is a text stream, otherwise it is a byte stream. If the
696-
*stderr* argument was not :data:`PIPE`, this attribute is ``None``.
714+
error output from the child process. If the *encoding* or *errors* arguments
715+
were specified or the *universal_newlines* argument was ``True``, the stream
716+
is a text stream, otherwise it is a byte stream. If the *stderr* argument was
717+
not :data:`PIPE`, this attribute is ``None``.
697718

698719
.. warning::
699720

@@ -886,7 +907,9 @@ calls these functions.
886907
*timeout* was added.
887908

888909

889-
.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None)
910+
.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, \
911+
encoding=None, errors=None, \
912+
universal_newlines=False, timeout=None)
890913

891914
Run command with arguments and return its output.
892915

@@ -1142,7 +1165,7 @@ handling consistency are valid for these functions.
11421165
Return ``(status, output)`` of executing *cmd* in a shell.
11431166

11441167
Execute the string *cmd* in a shell with :meth:`Popen.check_output` and
1145-
return a 2-tuple ``(status, output)``. Universal newlines mode is used;
1168+
return a 2-tuple ``(status, output)``. The locale encoding is used;
11461169
see the notes on :ref:`frequently-used-arguments` for more details.
11471170

11481171
A trailing newline is stripped from the output.

‎Doc/whatsnew/3.6.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -589,6 +589,9 @@ proc: ...``) or call explicitly the :meth:`~subprocess.Popen.wait` method to
589589
read the exit status of the child process (Contributed by Victor Stinner in
590590
:issue:`26741`).
591591

592+
The :class:`subprocess.Popen` constructor and all functions that pass arguments
593+
through to it now accept *encoding* and *errors* arguments. Specifying either
594+
of these will enable text mode for the *stdin*, *stdout* and *stderr* streams.
592595

593596
telnetlib
594597
---------

‎Lib/subprocess.py‎

Lines changed: 45 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,8 @@ class Popen(args, bufsize=-1, executable=None,
3030
preexec_fn=None, close_fds=True, shell=False,
3131
cwd=None, env=None, universal_newlines=False,
3232
startupinfo=None, creationflags=0,
33-
restore_signals=True, start_new_session=False, pass_fds=()):
33+
restore_signals=True, start_new_session=False, pass_fds=(),
34+
*, encoding=None, errors=None):
3435
3536
3637
Arguments are:
@@ -104,20 +105,13 @@ class Popen(args, bufsize=-1, executable=None,
104105
If env is not None, it defines the environment variables for the new
105106
process.
106107
107-
If universal_newlines is False, the file objects stdin, stdout and stderr
108-
are opened as binary files, and no line ending conversion is done.
108+
If encoding or errors are specified or universal_newlines is True, the file
109+
objects stdout and stderr are opened in text mode. See io.TextIOWrapper for
110+
the interpretation of these parameters are used.
109111
110-
If universal_newlines is True, the file objects stdout and stderr are
111-
opened as a text file, but lines may be terminated by any of '\n',
112-
the Unix end-of-line convention, '\r', the old Macintosh convention or
113-
'\r\n', the Windows convention. All of these external representations
114-
are seen as '\n' by the Python program. Also, the newlines attribute
115-
of the file objects stdout, stdin and stderr are not updated by the
116-
communicate() method.
117-
118-
In either case, the process being communicated with should start up
119-
expecting to receive bytes on its standard input and decode them with
120-
the same encoding they are sent in.
112+
If no encoding is specified and universal_newlines is False, the file
113+
objects stdin, stdout and stderr are opened as binary files, and no
114+
line ending conversion is done.
121115
122116
The startupinfo and creationflags, if given, will be passed to the
123117
underlying CreateProcess() function. They can specify things such as
@@ -234,11 +228,8 @@ class Popen(args, bufsize=-1, executable=None,
234228
and stderr, until end-of-file is reached. Wait for process to
235229
terminate. The optional input argument should be data to be
236230
sent to the child process, or None, if no data should be sent to
237-
the child. If the Popen instance was constructed with universal_newlines
238-
set to True, the input argument should be a string and will be encoded
239-
using the preferred system encoding (see locale.getpreferredencoding);
240-
if universal_newlines is False, the input argument should be a
241-
byte string.
231+
the child. If the Popen instance was constructed in text mode, the
232+
input argument should be a string. Otherwise, it should be bytes.
242233
243234
communicate() returns a tuple (stdout, stderr).
244235
@@ -808,8 +799,8 @@ def getstatusoutput(cmd):
808799
""" Return (status, output) of executing cmd in a shell.
809800
810801
Execute the string 'cmd' in a shell with 'check_output' and
811-
return a 2-tuple (status, output). Universal newlines mode is used,
812-
meaning that the result with be decoded to a string.
802+
return a 2-tuple (status, output). The locale encoding is used
803+
to decode the output and process newlines.
813804
814805
A trailing newline is stripped from the output.
815806
The exit status for the command can be interpreted
@@ -859,7 +850,7 @@ def __init__(self, args, bufsize=-1, executable=None,
859850
shell=False, cwd=None, env=None, universal_newlines=False,
860851
startupinfo=None, creationflags=0,
861852
restore_signals=True, start_new_session=False,
862-
pass_fds=()):
853+
pass_fds=(), *, encoding=None, errors=None):
863854
"""Create new Popen instance."""
864855
_cleanup()
865856
# Held while anything is calling waitpid before returncode has been
@@ -912,6 +903,8 @@ def __init__(self, args, bufsize=-1, executable=None,
912903
self.pid = None
913904
self.returncode = None
914905
self.universal_newlines = universal_newlines
906+
self.encoding = encoding
907+
self.errors = errors
915908

916909
# Input and output objects. The general principle is like
917910
# this:
@@ -944,22 +937,28 @@ def __init__(self, args, bufsize=-1, executable=None,
944937
if errread != -1:
945938
errread = msvcrt.open_osfhandle(errread.Detach(), 0)
946939

947-
if p2cwrite != -1:
948-
self.stdin = io.open(p2cwrite, 'wb', bufsize)
949-
if universal_newlines:
950-
self.stdin = io.TextIOWrapper(self.stdin, write_through=True,
951-
line_buffering=(bufsize == 1))
952-
if c2pread != -1:
953-
self.stdout = io.open(c2pread, 'rb', bufsize)
954-
if universal_newlines:
955-
self.stdout = io.TextIOWrapper(self.stdout)
956-
if errread != -1:
957-
self.stderr = io.open(errread, 'rb', bufsize)
958-
if universal_newlines:
959-
self.stderr = io.TextIOWrapper(self.stderr)
940+
text_mode = encoding or errors or universal_newlines
960941

961942
self._closed_child_pipe_fds = False
943+
962944
try:
945+
if p2cwrite != -1:
946+
self.stdin = io.open(p2cwrite, 'wb', bufsize)
947+
if text_mode:
948+
self.stdin = io.TextIOWrapper(self.stdin, write_through=True,
949+
line_buffering=(bufsize == 1),
950+
encoding=encoding, errors=errors)
951+
if c2pread != -1:
952+
self.stdout = io.open(c2pread, 'rb', bufsize)
953+
if text_mode:
954+
self.stdout = io.TextIOWrapper(self.stdout,
955+
encoding=encoding, errors=errors)
956+
if errread != -1:
957+
self.stderr = io.open(errread, 'rb', bufsize)
958+
if text_mode:
959+
self.stderr = io.TextIOWrapper(self.stderr,
960+
encoding=encoding, errors=errors)
961+
963962
self._execute_child(args, executable, preexec_fn, close_fds,
964963
pass_fds, cwd, env,
965964
startupinfo, creationflags, shell,
@@ -993,8 +992,8 @@ def __init__(self, args, bufsize=-1, executable=None,
993992

994993
raise
995994

996-
def _translate_newlines(self, data, encoding):
997-
data = data.decode(encoding)
995+
def _translate_newlines(self, data, encoding, errors):
996+
data = data.decode(encoding, errors)
998997
return data.replace("\r\n", "\n").replace("\r", "\n")
999998

1000999
def __enter__(self):
@@ -1779,13 +1778,15 @@ def _communicate(self, input, endtime, orig_timeout):
17791778

17801779
# Translate newlines, if requested.
17811780
# This also turns bytes into strings.
1782-
if self.universal_newlines:
1781+
if self.encoding or self.errors or self.universal_newlines:
17831782
if stdout is not None:
17841783
stdout = self._translate_newlines(stdout,
1785-
self.stdout.encoding)
1784+
self.stdout.encoding,
1785+
self.stdout.errors)
17861786
if stderr is not None:
17871787
stderr = self._translate_newlines(stderr,
1788-
self.stderr.encoding)
1788+
self.stderr.encoding,
1789+
self.stderr.errors)
17891790

17901791
return (stdout, stderr)
17911792

@@ -1797,8 +1798,10 @@ def _save_input(self, input):
17971798
if self.stdin and self._input is None:
17981799
self._input_offset = 0
17991800
self._input = input
1800-
if self.universal_newlines and input is not None:
1801-
self._input = self._input.encode(self.stdin.encoding)
1801+
if input is not None and (
1802+
self.encoding or self.errors or self.universal_newlines):
1803+
self._input = self._input.encode(self.stdin.encoding,
1804+
self.stdin.errors)
18021805

18031806

18041807
def send_signal(self, sig):

0 commit comments

Comments
 (0)