Proper way to use loguru.logger with multiprocessing( on Mac M1 / Windows)

[changchiyou]

My env

• Apple Mac M1 Air
• 8GB RAM
• Ventura 13.2.1（22D68）

Question

1. Is there more example of using loguru.logger with multiprocessing or other multi-process library and shared the configuration of loguru.logger? I have read Compatibility with multiprocessing using enqueue argument in Code snippets and recipes for loguru but many error occured on my project after I followed that.

2. Can I pass the loguru.logger instance through methods as argument and reload/reset/overwrite the loguru.logger in sub-process which is created by "spawning" instead of "forking"(I got many warning message when I tried to use "forking" on Mac M1)?

The error mentioned in 1., 2.

The setting of loguru.logger missed after my sub-process is created:

# settings including two `logger.add()` for logging on console and recording into logs
set_loguru(
    enqueue=True,
    console_level=logging.DEBUG
    if config_setting.args.DEBUG is True
    else logging.INFO,
)

# this line work perfectly with the setting above (from `set_loguru`)
logger.debug("start")

# but the logger inside missed the setting
workers = []
for section_name, mark in config_setting.get_matching_sections():
    worker = multiprocessing.Process(
        target=realtime,
        args=(mark, section_name, self.config_path, self.args_dict),
        daemon=True,
    )
    worker.start()
    workers.append(worker)

I have tried to set

multiprocessing.set_start_method("fork")

on the beginning of my project to avoid loguru.logger being recreated due to the spawning/forking differences mentioned in #908 (comment):

if the child process is created by "spawning" and not by "forking", then two different logger objects will be created, and they won't share the same handlers.

But I got the error message below due to the answer in Multiprocessing causes Python to crash and gives an error may have been in progress in another thread when fork() was called:

This error occurs because of added security to restrict multithreading in macOS High Sierra and later versions of macOS.

objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.

And I got more error message after I follow the tips and fixed the problem above although my project can barely run already :

2023-07-03 15:42:28.769 python[23120:3545276] +[NSXPCSharedListener endpointForReply:withListenerName:replyErrorCode:]: an error occurred while attempting to obtain endpoint for listener 'ClientCallsAuxiliary': Connection invalid
2023-07-03 15:42:33.765 python[23121:3545277] +[NSXPCSharedListener endpointForReply:withListenerName:replyErrorCode:]: an error occurred while attempting to obtain endpoint for listener 'ClientCallsAuxiliary': Connection invalid

2023-07-03 15:42:55.882 python[23121:3545277] Error received in message reply handler: Connection invalid
2023-07-03 15:42:55.882 python[23121:3545503] Connection Invalid error for service com.apple.hiservices-xpcservice.

2023-07-03 15:43:06.644 python[23121:3545277] Error received in message reply handler: Connection invalid
2023-07-03 15:43:06.644 python[23121:3545277] Error received in message reply handler: Connection invalid

I am not sure whether these warning messages were came from loguru since I have not used @logger.catch. Confusing now and have no idea about what happened now 😢

[changchiyou]

Small Update

I got the solution from #818 of my question 2.:

logger._core = logger_._core

I have the same reasons as @PakitoSec that this solution is not the best.

but I think this is not recommended since _core is not in the public api

[Delgan]

Hi.

There aren't that many more examples than those on the documentation page you've shared.

I'm not a Mac user and therefore I can't help you much with the issues you're facing. Using fork on Mac should work relatively well, though, as it's unit tested:

loguru/tests/test_multiprocessing.py

Lines 223 to 238 in 0b50708

	@pytest.mark.skipif(os.name == "nt", reason="Windows does not support forking")
	def test_process_fork(fork_context):
	writer = Writer()
	logger.add(writer, context=fork_context, format="{message}", enqueue=True, catch=False)
	process = fork_context.Process(target=subworker, args=(logger,))
	process.start()
	process.join()
	assert process.exitcode == 0
	logger.info("Main")
	logger.remove()
	assert writer.read() == "Child\nMain\n"

[changchiyou]

@Delgan Thanks for you replay 👍

What about the second question?

Can I pass the loguru.logger instance through methods as argument and reload/reset/overwrite the loguru.logger in sub-process which is created by "spawning" instead of "forking"(I got many warning message when I tried to use "forking" on Mac M1)?

Although I can’t find that issue yet, I’m pretty sure you mentioned in an issue before that developer should avoid creating logger objects repeatedly in different processes as much as possible. This move may lead to unexpected errors. Therefore I try to solved my needs without using logger.add() directly. Instead, I tried to copy entire logger instance from main process to sub-processes:

I got the solution from #818 of my question 2.:

logger._core = logger_._core

I have the same reasons as @PakitoSec that this solution is not the best.

but I think this is not recommended since _core is not in the public api

---

Or can I just re-implement it wtih loguru.logger.add() instead of trying to avoid it as much as possible? I'm afraid that this will lead to unexpected results in rotating or other places. 😢

[changchiyou]

I want to make my project runnable crossing different platforms, including Mac M1, Mac Intel and Windows also for sure. Therefore I can't use fork because it's unavailable on Windows.

Here is my current solution for my needs, which is not good enought for me:

_loguru.py:

class Logger:
    def __init__(self):
        self.logger: loguru.Logger

    def init(self, _logger: loguru.Logger):
        self.logger = _logger

logger = Logger()

main.py(in sub-process):

from _loguru import logger
def main(_logger: loguru.Logger):
    logger.init(_logger)
    logger.logger.debug("message")

---

Is there any way to make it more readable? I wish I can logging via executing from _loguru import logger and logger.info("msg") directly, instead logger.logger.info("msg") in my current solution above. 😢

[Delgan]

Can I pass the loguru.logger instance through methods as argument and reload/reset/overwrite the loguru.logger in sub-process which is created by "spawning" instead of "forking"(I got many warning message when I tried to use "forking" on Mac M1)?

Indeed, the best way to share the logger with a child process started by the "spawn" method is to pass it as a initializer argument.

However, there is no way currently to globally update the logger instance in the child process. I plan to add a logger.reinstall() method as explained in #818, but this is not yet implemented.

A possible workaround would be to use some kind of wrapper like that:

# _loguru.py
from loguru import logger as _logger

class LoggerDelegator:
    def __init__(self, logger):
        self._logger = logger

    def update_logger(self, logger):
        self._logger = logger

    def __getattr__(self, attr):
        return getattr(self._logger, attr)

logger = LoggerDelegator(_logger)

[changchiyou]

Update

The error below occured because I passed LoggerDelegator instance as the logger arg of LoggerDelegator.update_logger(). I am so dumb. 🤣

@Delgan Thanks for your reply, your solution helps me a lot.

---

@Delgan seems something went wrong with _getattr__()?

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/christopherchang/miniconda3/envs/facepcs/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/christopherchang/miniconda3/envs/facepcs/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/christopherchang/miniconda3/envs/facepcs/lib/python3.10/multiprocessing/spawn.py", line 126, in _main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/christopherchang/miniconda3/envs/facepcs/lib/python3.10/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/Users/christopherchang/Work/FacePCS/python-package/facepcs/utils/_loguru.py", line 16, in __getattr__
    self = reduction.pickle.load(from_parent)
  File "/Users/christopherchang/Work/FacePCS/python-package/facepcs/utils/_loguru.py", line 16, in __getattr__
    return getattr(self._logger, attr)
  File "/Users/christopherchang/Work/FacePCS/python-package/facepcs/utils/_loguru.py", line 16, in __getattr__
    return getattr(self._logger, attr)
  File "/Users/christopherchang/Work/FacePCS/python-package/facepcs/utils/_loguru.py", line 16, in __getattr__
    return getattr(self._logger, attr)
  File "/Users/christopherchang/Work/FacePCS/python-package/facepcs/utils/_loguru.py", line 16, in __getattr__
    return getattr(self._logger, attr)
  File "/Users/christopherchang/Work/FacePCS/python-package/facepcs/utils/_loguru.py", line 16, in __getattr__
    return getattr(self._logger, attr)
  [Previous line repeated 993 more times]
RecursionError: maximum recursion depth exceeded
    return getattr(self._logger, attr)
  [Previous line repeated 993 more times]
RecursionError: maximum recursion depth exceeded

I have copied your code and add type-hints into it like this:

# _loguru.py
from __future__ import annotations

import loguru
from loguru import logger as _logger


class LoggerDelegator:
    def __init__(self, logger: loguru.Logger):
        self._logger: loguru.Logger = logger

    def update_logger(self, logger: loguru.Logger):
        self._logger = logger

    def __getattr__(self, attr):
        return getattr(self._logger, attr)


logger = LoggerDelegator(_logger)