<- Example 1: Modifying a config file | Example 2: Unraveling the X-Y problem | Example 3: Grouping integers into ranges ->

Unraveling the X-Y problem

This example has a difficulty rating of easy, but it is presented with a twist. This twist makes it appear to be a problem of medium difficulty. The primary goal of this page is to show how the presentation of a problem can mislead you into coming up with suboptimal solutions.

The question, as it was presented to us originally, was an X-Y problem.

Rather than give the correct problem specification up front, let's show the way things played out in real life. This will help you see how the presentation influenced the thought processes of the people involved, as well as the evolution of the understanding of the question.

About 90% of the work in solving a #bash problem is understanding the question. The other 10% is actually writing the code.

Initial presentation

<asker> hello, i have a script with 2 functions, one has sleep 180, and the other sleep 3.
        I call them later by doing function1 & and function2 &, but function2 outputs at
        the rate of function1
<asker> what am i doing wrong?
<helper> what do you mean "at the rate of"?
<asker> it's outputting every 180 seconds
<helper> neither one should output *anything* if they only contain sleep commands
<asker> they contain stuff i want to output

At this point, the question appears to involve some sort of interprocess communications (IPC), and an issue with synchronization of the two processes. 

<helper> are you TRYING to say  f1() { while sleep 3; do echo "f1"; done; } ;
         f2() { while sleep 180; do echo "f2"; done; }  ?
<helper> if that is the case, then how are you ACTUALLY calling the functions?
<asker> https://gist.github.com/fbb84024d020f9fb7a272953a94909ec

At this point, I made a fundamental error. I opened the link and read the code, rather than pushing harder for an explanation of the goal. All this did was further entrench the notion that this was an IPC problem.

Long story short, the submitted code used some command called twitchnotifier, in a loop, writing to a file. A second command was supposed to read from this file and do... something with it.

To me, this all looked like "I am creating a log file, but I also want to see the logfile on my terminal in real time". 

<helper> I have no idea how "twitchnotifier" works.  It could be a buffering issue.
<helper> oh jesus, is one of them writing to the same file that the other one is
         *reading*?  And you didn't even use tail -f?
<helper> How are you surprised that they both "run at the same rate" when one of them
         is using the other one's output?
<asker> because i barely know bash
<asker> i assumed that since the output is just a file sitting there, the second
        function would be able to read at any rate
<helper> If there's nothing new in the file, what did you expect it to do?
<asker> well if the file has 4 lines, i expected it to go over those 4 lines every
        3 seconds. Instead, it goes over each line every 180 seconds
<helper> Is your goal this:  while sleep 180; do twitchnotifier -c "$user" -n;
         done > "$file" & tail -f "$file"

My proposal here would solve what I believed to be the goal at this point: generate the log file in the background, and display it on the terminal in the foreground. However, the asker was still stuck in "my two processes are not communicating properly", or some kind of synchronization or desynchronization issue. Very unclear, and very confusing. 

<asker> ah i see, so will this go over to the file over and over starting from the
        top when it reaches the end?
<helper> "man tail", read what -f does

<asker> your line keeps appending the output of twitchnotifier, correct? instead 
        of rewriting the file altogether?
<helper> Yes, because it's a cumulative log, and tail -f just keeps showing you
         the new pieces.
<asker> but the thing about my usecase is that if one of them goes offline, i don't
        want them displayed anymore. Rewriting the whole file solves this issue
<helper> any... one... of ... what?!

Here, we start to get a glimpse of the actual goal. At this point, I still assumed that each invocation of twitchnotifier (whatever that is) was writing a single event (line) to the log file. 

<asker> twitchnotifier outputs online twitch channels
<helper> ...
<helper> is a single invocation of twitchnotifier -c "$user" following MORE THAN ONE
         THING?
<asker> hm?
<helper> You ran one instance of twitchnotifier in a loop, writing to one file.
<helper> Now somehow there's a PLURAL in the question.
<asker> it writes one line to the file per each online channel, every time I run
        twitchnotifier
<helper> and?
<asker> so the file might contain 1, 2 or 10 lines each time it gets rewritten
<helper> and?
<asker> if i instead append the output, it's going to keep adding channels that go 
        online, and retain the ones that went offline

At this point, it appears twitchnotifier actually gives a multi-line result set, and that he wants to see only the most current result set at any given time. So: 

<helper> So, is THIS your new goal:   while :; do tput clear;
         twitchnotifier -c "$user" -n; sleep 180; done
<asker> but now there's no file?
<helper> Why do you need a file?
<asker> maybe i'm not explaining right. I need two things: 1: every 3 minutes, check 
        for the output of twitchnotifier. 2: every 30 seconds, cycle through each line
        of the output of twitchnotifier
<helper> What does "cycle through" mean?
<asker> i thought a file would provide the simplest solution
<helper> I thought you just wanted to SEE the most recent results.
<asker> print one line every 30s, until it gets to the end, then start again
<helper> Why can't you just show them ALL AT ONCE?

(Did you notice that the "sleep 3" from the start of the session has magically become "30 seconds"?)

But, but, here it comes! 

<asker> the purpose is to use this with polybar https://u.teknik.io/pimF6.png so space
        is the limiting factor
<helper> One hour and one minute.
<helper> hat's how long it took us to get "Oh by the way I have an output formatting
         restriction" from the start of the question.
<helper> So, STARTING FROM THE BEGINNING.   You have something called "polybar".  It is 
         like, I'm guessing, a 40x1 terminal?  Something along those lines?  You can only
         write one small line of text at a time in it?
<helper> And you have some program called "twitchnotifier" which you can use to poll a 
         web service and it returns MANY lines, not always the same number of lines, and
         you don't know how many there will be on any given poll.
<asker> it's a status bar. I could write them all at once but it would fill up the whole 
        bar, so i want it to sit there and cycle through one at a time
<helper> So you want to run your polling-thing every 180 seconds, and this populates a 
         POOL OF LINES that your polybar-updating-thing can use to cycle the display of 
         lines in the polybar.
<helper> Am I on the right track yet?
<asker> yes

At this point, if you want to try to solve the problem yourself, stop reading here. I have intentionally made the problem harder for you than it has to be, and it's only going to get worse before it gets better.

First proposed solution

Now, remember the context here. The problem was initially presented as some kind of synchronization or buffering issue between multiple processes involved in an interprocess communications setup. A plain file was being used as the medium for information transfer between the processes. 

<helper> So the fundamental issue is that your polybar-updating-thing will retain the pool
         of lines in memory, but you need to be able to *communicate* to it when it should
         flush its pool and grab the new set of lines.  Use a signal handler for that.
<helper> So, poller will sleep, run twitchnotifier, output to a file, send a signal to updater,
         and repeat.
<asker> do i need to communicate when to flush if i just set it to flush every 3 minutes tho?
<helper> Updater will read from the pool, store lines in an array, loop through the array,
         but always be ready to receive a signal indicating it should flush the array and start
         over.
<helper> Those are the pieces.  Have fun.

And a slight addendum: 

<helper> Actually, you don't *need* to process the signal immediately.  You can take the easy
         road and just let the trap fire when your sleep finishes.
<helper> That makes it much simpler.
<asker> isn't that what i just asked?
<helper> As opposed to the immediate-signal-processing version, which would typically require 
         that you run the sleep in the background and "wait" for it, so that the wait can be 
         interrupted by the signal.
<helper> A foreground sleep will not be interrupted.

This was, I believe, the end of the IRC session with this person. I gave a proposal for a multi-process algorithm that could be used to solve it, but didn't write any code. The asker probably went away completely bewildered.

Second proposed solution

What happens if we're not working at the frantic pace of IRC, with three different problems going on at the same time?

What if we're given the specification up front, and we start from there? Maybe the problem looks much different.

What do we have, and what do we actually want?

We don't actually need multiple processes for this. Maybe the polling takes a few seconds -- so what? The original code/question had a 30 second (or was it 3 second?) delay between outputs. If one of the displays takes 32 seconds instead of 30, who cares?

What do we need, then?

This can be done in a single script, with a single "thread" (not literal) of execution. We don't need any background processes, or any files.

Let's start with the polling function: 

user="whatever"

poll() {
  mapfile -t result < <(twitchnotifier -c "$user" -n)
  curr=0
  n=${#result[@]}
  SECONDS=0
}

Everything should be obvious here except the SECONDS variable. That's an internal bash variable that we can use to measure how long the script has been running. Setting it to 0 resets the timer. When we check the variable later, we'll know how many seconds have elapsed since that reset.

Then, the infinite loop is relatively simple too. I'm going to assume the output device is a vanilla terminal with one line, so we use a carriage return character to move to the start of the line, and space padding at the end to make sure the old result isn't partially left behind. If writing to the "polybar" thing works differently, this can be adjusted. 

expire=180
delay=30
poll

while true; do
  if ((SECONDS > expire)); then poll; fi
  printf '\r%-40.40s' "${result[curr]}"
  curr=$(( (curr + 1) % n ))
  sleep "$delay"
done

The convenience variable n is simply the size of the result set (number of lines). By doing a "modulus n" after incrementing the curr variable, we "wrap around" to the first result after the last result.

Lessons

If you're asking for help with a problem, start by stating the problem. The actual, original, problem. Not the current state of your mind after you've been banging your head against the problem for an hour. Not "here's what I have so far". Not the solutions you've tried. Start at the beginning.

Start with what you have. This does not mean the code that you've written. Any code you've written is going to be thrown away. Sorry, that's just how it is.

Identify the inputs you're working with, and the restrictions that you're working around. For example, "I can only display one line of output at a time" is fundamental to the problem! If it takes you literally 61 minutes to say that, that's a whole hour wasted.

If it's a homework question with completely insane restrictions, post a link to the actual question. Don't paraphrase it. Don't write what you think the problem says. Bash homework questions are mind-bogglingly stupid (always!), and the obvious, correct answer will be disallowed. Remember, the teacher is trying to get you to come up with the teacher's answer, not the correct answer. The only way we can guess what the teacher's answer is, is to see the original question.

The people who are trying to help you can't read your mind. Nor can we read your boss's mind, or your teacher's mind, or your customer's mind.

If someone is asking you for help, make sure you get the problem up front. Be strong. Don't open the unsolicited URL. Seeing the user's code will only waste your time and make the problem even less clear. Insist on an explanation of the problem. If that means you can't help this user, so be it. The user is the one being obnoxious and stubborn. By refusing to state the problem, they are demonstrating that they are not helpable. You have no moral obligation to help the unhelpable, especially when they are rude, selfish, lazy, incompetent....

<- Example 1: Modifying a config file | Example 2: Unraveling the X-Y problem | Example 3: Grouping integers into ranges ->

BashProgramming/Ex02 (last edited 2019-04-15 18:57:08 by GreyCat)