@1st1, have you planned to fix this?

Yes. Will take a look in a week or two. If you have time to make a PR I'll review it.

Actually, uvloop can block only if you supply a preexec_fn argument to loop.subprocess_shell() or loop.subprocess_exec(). Could you please share your test?

@soonum could you please share your test as requested by @1st1 ?

Sorry I wasn't following up this old issue.
In fact my use case is that one, I provide a preexec_fn argument to these methods and as a result, the loop is blocking.

We have a component in Home Assistant; they call ffmpeg with a subprocess (async). They also end sometimes (less and more) in blocking and not working event loop :( I can't provide any test code because it's not all time simple to reproduce it in a laboratory situation.

The point is, we don't use preexec_fn and end in a blocking loop :(

The problem is forking. It looks like the current implementation doesn't work on a heavy load of the main process.

The point is, if we use subprocess inside executors, it work without issues. Also call the normal subprocess inside loop work with attach a streamreader to output. But the subprocess from uvloop hungs

any news to this? the HA community is waiting for a solution wistfully ;)

Hm very sad that its not going on here :( I would love using the HA stream component...
@1st1 any plan about this?

@1st1 any plan about this?

I still haven't received any clear answer to #118 (comment).

I know that uvloop is blocking when preexec_fn is used, but almost nobody uses it. Other than that, as @Martiusweb explained in his comment #118 (comment) it should not block.

@pvizeli

But the subprocess from uvloop hungs

Does uvloop behave differently from asyncio in your case?

@1st1 yes. The reason why we remove it from Home Assistant. However, #268 fix the problem and we can reuse it after that is merged.

Great, I'm working on a release right now, and will likely merge #268 or a variation of it.

• uvloop version: 0.14.0

Hi there,

I got some details after encountering another blocking case. I found that if you provide a value for the keyword argument cwd and experiencing a slow filesystem at runtime, the call to loop.subprocess_[exec|shell]() would block.

Here is a simple test to reproduce the behavior. It's relying on slowfs.

1. create a directory /home_slow
2. run the following command <path_to>/slowfs/env/bin/slowfs -c <path_to>/slowfs/slowfs/slowfs.cfg --allow-other /home /home_slow/
3. in another shell, run the python (tested with Python v3.8.0) example subprocess_test.py

You should see that Python is blocking during 5 seconds. This is related to the delay enforced by slowfs.

Here is the content of subprocess_test.py:

import asyncio
import datetime
import subprocess
import uvloop

async def subprocess_task():
    await asyncio.sleep(2)
    process = await asyncio.create_subprocess_shell("touch a_file.txt" , cwd="/home_slow/",) 
    await process.wait()
    print("Process terminated with code", process.returncode, flush=True)


async def print_task():
    """
    Print seconds elapsed since beginning of the program at 1Hz.
    """
    start_time = datetime.datetime.now()
    while True:
        now = datetime.datetime.now()
        print(round((now - start_time).total_seconds()), flush=True)
        await asyncio.sleep(1)


if __name__ == "__main__":
    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
    loop = asyncio.get_event_loop()
    task_1 = loop.create_task(print_task())
    task_2 = loop.create_task(subprocess_task())
    
    try:
        loop.run_forever()
    except KeyboardInterrupt:
        task_1.cancel()
        task_2.cancel()
        loop.close()

Here's the config used in slowfs.cfg:

access = 5
chmod = 0
chown = 0
getattr = 5
readdir = 5
readlink = 0
mknod = 0
rmdir = 0
mkdir = 0
statfs = 5
unlink = 0
symlink = 0
rename = 0
link = 0
utimens = 0
open = 0
create = 0
read = 0
write = 0
truncate = 0
flush = 0
release = 0
fsync = 0

By the way, #268 doesn't solve the issue regarding using a preexec function. The way I tested it is by defining a function that does a simple time.sleep(5) to see the loop blocking.

Hope that could help to slove this issue.

hi,

would be concurrent.futures import ProcessPoolExecutor a workaround?

Here is a simple test to reproduce the behavior. It's relying on slowfs.

Nice, that's great! Thanks for the slowfs tip, I didn't know about it before.

would be concurrent.futures import ProcessPoolExecutor a workaround?

Unfortunately no, they would still block.

@1st1 thanks for reply.
So the fix would be at the moment to make subprocess async -> run subprocess in thread (with some async flavour).

`would be concurrent.futures import ProcessPoolExecutor a workaround?

Unfortunately no, they would still block.`

@1st1 here is possible fix

def run_subprocess(cmd, loop=None):
    loop = loop or asyncio.get_event_loop()
    try:
        process = await asyncio.create_subprocess_shell(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    except NotImplementedError:
        with subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True) as process:
            try:
                result = await loop.run_in_executor(None, process.communicate)
            except Exception:  # muh pycodestyle
                def kill():
                    process.kill()
                    process.wait()

                await loop.run_in_executor(None, kill)
                raise

    else:
        result = await process.communicate()

    return [res.decode('utf-8') for res in result] ```