I have a server running on an old Android device. It recently started crashing on one of its web views and I wanted to take the opportunity to practice remote debugging. The server logs aren’t easily accessible to me because I don’t have an SSH server running on the phone, but I knew I should be able to see the crash logs through OTP’s built-in behaviors.
The caveat to what I’m about to share is that it’s dangerous and something you should only consider using in a small contained environment – not in production systems. Why? You could blow up your server.
Goal: redirect system logger output from running system to local development shell.
Like most systems, an Erlang system prints error logs to what is essentially stdout. That’s actually an oversimplification, but not wrong enough to derail this post. The problem is that when I connect my laptop to my old phone via starting an erl shell in distributed mode, I only see the error messages originating from my laptop. The web server crashes are still logging on my phone and unavailable to me.
This task turned out to be a little harder than I expected it to be because I didn’t know what I should be looking for and there are sparse docs for it.
In short, IO flows through a process’ group_leader, and that group leader is changeable, so if you swap group leaders for a remote process you can redirect its logs to anywhere you want.
Now given that I didn’t know what process was crashing on my phone I couldn’t efficiently do this by picking random posts off my system and inspecting their logs. I also wasn’t sure how to pick out my web server and watch its logs.
The solution is to connect a distributed Erlang node and then tell the remote node to swap the group leader for its logger_sup process.
-module(remote_logger).
-export([
activate/1,
deactivate/2,
remote_loader/1
]).
activate(Node) ->
{group_leader, LocalGL} = process_info(whereis(logger_sup), group_leader),
{ok, OriginalGL} = rpc:call(Node, erlang, apply, [fun remote_loader/1, [LocalGL]]),
{ok, OriginalGL}.
deactivate(Node, OriginalGL) ->
{ok, PrevGL} = rpc:call(Node, erlang, apply, [fun remote_loader/1, [OriginalGL]]),
{ok, PrevGL}.
remote_loader(RemoteGL) ->
Sup = whereis(logger_sup),
{group_leader, OriginalGL} = process_info(Sup, group_leader),
group_leader(RemoteGL, Sup),
logger:reconfigure(),
{ok, OriginalGL}.
#erlang #remote-debugging
Also on Tumblr