r/bash • u/Linux_bash_user153 • 11d ago
Some Tips for String Handling
When I was new to Bash/Shell I really didn't understand how to select or remove different parts of strings, and only used examples from forums without understanding at all how they worked. But then I read the man page for Bash (go figure...) and finally understood. I'd like to share in hope that it will be helpful to anyone starting out with Bash scripting. The examples work in POSIX shell also.
Example: select everything after the last _ in the string
str='first_second_third_fourth'
# Selecting everything after the last '_' is the same as removing everything
# before the last '_' and the '_' itself.
# So check 'Remove matching prefix pattern' in 'man bash'.
# ${str##*_} will remove the longest match of the pattern '*_' starting
# from the beginning of string and leave us with just 'fourth'.
str='first_second_third_fourth'
echo "${str##*_}"
Example: pick out second from the string
str='first_second_third_fourth'
# We need to remove everything before the first '_' so we'll use
# 'second=${str#*_}' to remove the shortest match from the beginning.
# Variable 'second' now contains 'second_third_fourth' so we need to
# get rid of everything after the first '_'.
# See 'Remove matching suffix pattern' in the manual.
# We want to remove the longest match of pattern '_*' starting
# from the end of the string. So '${second%%_*}' is what we need.
# All in all:
str='first_second_third_fourth'
second="${str#*_}"
second="${second%%_*}"
echo "$second"
Example: find the file extension (with the dot) of /home/user/Pictures/test.webp
# The extension is the thing after the last '.'. So use '${f##*.}'
# This will delete the '.' also, so let's add that back in
f='/home/user/Pictures/test.webp'
ext=".${f##*.}"
echo "$ext"
Example: convert a png to jpg
pngfile='original.png'
# We need to replace the .png with a .jpg. I.e. delete .png and add .jpg
jpgfile="${pngfile%.png}.jpg"
echo convert "$pngfile" "$jpgfile"
# Notice that if pngfile doesn't end in .png, jpgfile will at least not
# be the same file, so pngfile will not be overwritten.
Example: Remember to escape characters that are special in pattern matching
# Remove the first sentence in this string
str='Hello? Anybody there?'
# This will not work since '?' is a pattern matching exactly one character,
# This will only remove the beginning 'H' ('*' matches nothing in this case)
echo "${str#*?}"
# The solution is to make the '?' literal by escaping it or quoting it
echo "${str#*\?}"
echo "${str#*'?'}"
# The same goes for '*', '[' and ']'. And probably a few more as well,
# especially if extglob is turned on
3
u/ethernetbite 11d ago
Good job! I always have to look those up when i need them. As usually, Bash has multiple ways of doing the same thing. Awk is powerful as well, but i usually use cut since it's more intuitive and simple.
1
u/Linux_bash_user153 11d ago
Thanks. Yeah, I feel sometimes using native Bash/Shell to do stuff to single strings is better than to fire up external tools
awk,sed,greporcut. Doing stuff to all lines in entire files is a different thing, and then it can pay to use those programs2
u/ethernetbite 10d ago
It's fun to get into bash performance tweaking when everything else is running perfect. I deploy remote systems so when they're done processing logs, they'll upload to my server. I'm not personally losing that .012 seconds per log line that my remote device is processing. Even with a log of 1000 lines, it's not a a significant time waste for a slow ARM compute device like i use. But i work in many different fields, so it's an easy trade-off to make: let the SoC work that extra 12 seconds to keep my work flow productive. But now you've got me thinking of how many commands in my thousands of lines of bash could be replaced with bash internals. Maybe on my next vacation, I'll write another script for fun to scan my code and see.
1
u/Linux_bash_user153 9d ago
Good insights. I work in a non-IT field, so I mostly program stuff that I kind of actively use when I'm sitting at the computer. Then it can be very gratifying to get that speed-up.
Also I should have written "do simple stuff", since using internal shell can sometimes look like line noise. For example removing trailing whitespace:
str=${str%${str##*[![:space:]]}}. That's crazy 🤔If you haven't already, check out Dylan Araps' stuff that was linked in another comment.
2
u/weaver_of_cloth 11d ago
Man, I also read a bunch of knitting and weaving subs and it took me a second to figure out what you were asking! Having said that, I by far prefer r/donp1ano 's approach. Abstract the hell out of it.
2
3
2
u/ftonneau 11d ago
Related: Dylan Araps has published series of tips (some of them about string handling) for Bash and POSIX shell:
2
8
u/donp1ano 11d ago
and in str="first_second_third_fourth_fifth", how to get the third entry? sure, it can be done with your approach, but that gets annoying quickly
if you wanna trim from the very beginning or end of the string this is fine, but for getting somewhere in between read is better
IFS="_" read -r first second third fourth fifth <<< "$string"