<- [[../Ex01|Example 1: Modifying a config file]] | '''Example 2: Unraveling the X-Y problem''' | [[../Ex03|Example 3: Grouping integers into ranges]] -> = Unraveling the X-Y problem = This example has a difficulty rating of '''easy''', but it is presented with a twist. This twist makes it appear to be a problem of '''medium''' difficulty. The primary goal of this page is to show how the ''presentation'' of a problem can mislead you into coming up with suboptimal solutions. The question, as it was presented to us originally, was an [[XyProblem|X-Y problem]]. Rather than give the correct problem specification up front, let's show the way things played out in real life. This will help you see how the presentation influenced the thought processes of the people involved, as well as the evolution of the ''understanding'' of the question. About 90% of the work in solving a #bash problem is ''understanding the question''. The other 10% is actually writing the code. == Initial presentation == {{{ hello, i have a script with 2 functions, one has sleep 180, and the other sleep 3. I call them later by doing function1 & and function2 &, but function2 outputs at the rate of function1 what am i doing wrong? what do you mean "at the rate of"? it's outputting every 180 seconds neither one should output *anything* if they only contain sleep commands they contain stuff i want to output }}} At this point, the question appears to involve some sort of interprocess communications (IPC), and an issue with synchronization of the two processes. {{{ are you TRYING to say f1() { while sleep 3; do echo "f1"; done; } ; f2() { while sleep 180; do echo "f2"; done; } ? if that is the case, then how are you ACTUALLY calling the functions? https://gist.github.com/fbb84024d020f9fb7a272953a94909ec }}} At this point, I made a fundamental error. I ''opened the link and read the code'', rather than pushing harder for an explanation of the goal. All this did was further entrench the notion that this was an IPC problem. Long story short, the submitted code used some command called `twitchnotifier`, in a loop, writing to a file. A second command was supposed to read from this file and do... something with it. To me, this all looked like "I am creating a log file, but I also want to see the logfile on my terminal in real time". {{{ I have no idea how "twitchnotifier" works. It could be a buffering issue. oh jesus, is one of them writing to the same file that the other one is *reading*? And you didn't even use tail -f? How are you surprised that they both "run at the same rate" when one of them is using the other one's output? because i barely know bash i assumed that since the output is just a file sitting there, the second function would be able to read at any rate If there's nothing new in the file, what did you expect it to do? well if the file has 4 lines, i expected it to go over those 4 lines every 3 seconds. Instead, it goes over each line every 180 seconds Is your goal this: while sleep 180; do twitchnotifier -c "$user" -n; done > "$file" & tail -f "$file" }}} My proposal here would solve what I ''believed'' to be the goal at this point: generate the log file in the background, and display it on the terminal in the foreground. However, the asker was still stuck in "my two processes are not communicating properly", or some kind of synchronization or desynchronization issue. Very unclear, and very confusing. {{{ ah i see, so will this go over to the file over and over starting from the top when it reaches the end? "man tail", read what -f does }}} {{{ your line keeps appending the output of twitchnotifier, correct? instead of rewriting the file altogether? Yes, because it's a cumulative log, and tail -f just keeps showing you the new pieces. but the thing about my usecase is that if one of them goes offline, i don't want them displayed anymore. Rewriting the whole file solves this issue any... one... of ... what?! }}} Here, we start to get a ''glimpse'' of the actual goal. At this point, I still assumed that each invocation of `twitchnotifier` (whatever ''that'' is) was writing a single event (line) to the log file. {{{ twitchnotifier outputs online twitch channels ... is a single invocation of twitchnotifier -c "$user" following MORE THAN ONE THING? hm? You ran one instance of twitchnotifier in a loop, writing to one file. Now somehow there's a PLURAL in the question. it writes one line to the file per each online channel, every time I run twitchnotifier and? so the file might contain 1, 2 or 10 lines each time it gets rewritten and? if i instead append the output, it's going to keep adding channels that go online, and retain the ones that went offline }}} At ''this'' point, it appears `twitchnotifier` actually gives a multi-line result set, and that he wants to see only the most current result set at any given time. So: {{{ So, is THIS your new goal: while :; do tput clear; twitchnotifier -c "$user" -n; sleep 180; done but now there's no file? Why do you need a file? maybe i'm not explaining right. I need two things: 1: every 3 minutes, check for the output of twitchnotifier. 2: every 30 seconds, cycle through each line of the output of twitchnotifier What does "cycle through" mean? i thought a file would provide the simplest solution I thought you just wanted to SEE the most recent results. print one line every 30s, until it gets to the end, then start again Why can't you just show them ALL AT ONCE? }}} (Did you notice that the "sleep 3" from the start of the session has magically become "30 seconds"?) But, but, here it comes! {{{ the purpose is to use this with polybar https://u.teknik.io/pimF6.png so space is the limiting factor One hour and one minute. hat's how long it took us to get "Oh by the way I have an output formatting restriction" from the start of the question. So, STARTING FROM THE BEGINNING. You have something called "polybar". It is like, I'm guessing, a 40x1 terminal? Something along those lines? You can only write one small line of text at a time in it? And you have some program called "twitchnotifier" which you can use to poll a web service and it returns MANY lines, not always the same number of lines, and you don't know how many there will be on any given poll. it's a status bar. I could write them all at once but it would fill up the whole bar, so i want it to sit there and cycle through one at a time So you want to run your polling-thing every 180 seconds, and this populates a POOL OF LINES that your polybar-updating-thing can use to cycle the display of lines in the polybar. Am I on the right track yet? yes }}} At this point, if you want to try to solve the problem yourself, stop reading here. I have intentionally made the problem harder for you than it has to be, and it's only going to get ''worse'' before it gets better. == First proposed solution == Now, remember the context here. The problem was initially presented as some kind of synchronization or buffering issue between multiple processes involved in an interprocess communications setup. A plain file was being used as the medium for information transfer between the processes. {{{ So the fundamental issue is that your polybar-updating-thing will retain the pool of lines in memory, but you need to be able to *communicate* to it when it should flush its pool and grab the new set of lines. Use a signal handler for that. So, poller will sleep, run twitchnotifier, output to a file, send a signal to updater, and repeat. do i need to communicate when to flush if i just set it to flush every 3 minutes tho? Updater will read from the pool, store lines in an array, loop through the array, but always be ready to receive a signal indicating it should flush the array and start over. Those are the pieces. Have fun. }}} And a slight addendum: {{{ Actually, you don't *need* to process the signal immediately. You can take the easy road and just let the trap fire when your sleep finishes. That makes it much simpler. isn't that what i just asked? As opposed to the immediate-signal-processing version, which would typically require that you run the sleep in the background and "wait" for it, so that the wait can be interrupted by the signal. A foreground sleep will not be interrupted. }}} This was, I believe, the end of the IRC session with this person. I gave a proposal for a multi-process algorithm that could be used to solve it, but didn't write any code. The asker probably went away completely bewildered. == Second proposed solution == What happens if we're ''not'' working at the frantic pace of IRC, with three different problems going on at the same time? What if we're given the specification ''up front'', and we start from there? Maybe the problem looks much different. What do we ''have'', and what do we actually ''want''? * We have a data source that we can poll for a result set every few minutes. The result set has an indeterminate number of "lines" (results). Every time we poll, we want to discard the previous result set and only use the new result set. * We have an output device that can only display one "line" (result) at a time. * We want to rotate through the current result set, displaying one result every few seconds, until it's time to refresh the result set by polling again. We don't actually ''need'' multiple processes for this. Maybe the polling takes a few seconds -- so what? The original code/question had a 30 second (or was it 3 second?) delay between outputs. If one of the displays takes 32 seconds instead of 30, who cares? What do we ''need'', then? * A function that will poll the web service and store the results in an indexed array. * A variable to keep track of which result we're currently showing. * A loop that will run forever, showing the current result and then changing the variable to the next result. * ... until the result set is over 180 seconds old, at which point we call the poll function again. This can be done in a single script, with a single "thread" (not literal) of execution. We don't need any background processes, or any files. Let's start with the polling function: {{{ user="whatever" poll() { mapfile -t result < <(twitchnotifier -c "$user" -n) curr=0 n=${#result[@]} SECONDS=0 } }}} Everything should be obvious here except the `SECONDS` variable. That's an internal bash variable that we can use to measure how long the script has been running. Setting it to 0 resets the timer. When we check the variable later, we'll know how many seconds have elapsed since that reset. Then, the infinite loop is relatively simple too. I'm going to assume the output device is a vanilla terminal with one line, so we use a carriage return character to move to the start of the line, and space padding at the end to make sure the old result isn't partially left behind. If writing to the "polybar" thing works differently, this can be adjusted. {{{ expire=180 delay=30 poll while true; do if ((SECONDS > expire)); then poll; fi printf '\r%-40.40s' "${result[curr]}" curr=$(( (curr + 1) % n )) sleep "$delay" done }}} The convenience variable `n` is simply the size of the result set (number of lines). By doing a "modulus n" after incrementing the `curr` variable, we "wrap around" to the first result after the last result. == Lessons == If you're asking for help with a problem, start by stating the problem. The actual, ''original'', problem. Not the current state of your mind after you've been banging your head against the problem for an hour. Not "here's what I have so far". Not the solutions you've tried. Start at the ''beginning''. Start with what you ''have''. This ''does not mean'' the code that you've written. Any code you've written is going to be thrown away. Sorry, that's just how it is. Identify the ''inputs'' you're working with, and the ''restrictions'' that you're working around. For example, "I can only display one line of output at a time" is ''fundamental'' to the problem! If it takes you literally 61 minutes to say that, that's a whole hour wasted. If it's a homework question with ''completely insane'' restrictions, post a link to the ''actual question''. Don't paraphrase it. Don't write what you ''think'' the problem says. Bash homework questions are [[BashFAQ/064|mind-bogglingly stupid]] ('''always'''!), and the obvious, correct answer will be disallowed. Remember, the teacher is trying to get you to come up with ''the teacher's answer'', not the ''correct'' answer. The only way we can guess what ''the teacher's answer'' is, is to see the original question. The people who are trying to help you can't read your mind. Nor can we read your boss's mind, or your teacher's mind, or your customer's mind. If someone is asking ''you'' for help, make sure you get the problem up front. Be strong. '''Don't''' open the unsolicited URL. '''Seeing the user's code will only waste your time and make the problem even less clear.''' Insist on an explanation of the problem. If that means you can't help this user, so be it. The user is the one being obnoxious and stubborn. By refusing to state the problem, they are demonstrating that they are not helpable. You have '''no''' moral obligation to help the unhelpable, especially when they are rude, selfish, lazy, incompetent.... ---- <- [[../Ex01|Example 1: Modifying a config file]] | '''Example 2: Unraveling the X-Y problem''' | [[../Ex03|Example 3: Grouping integers into ranges]] ->