Shell Script Utility To Read a File Line By Line

#!/bin/bash
# Shell script utility to read a file line line.
# Once line is read it can be process in processLine() function
# You can call script as follows, to read myfile.txt:
# ./readline myfile.txt
# Following example will read line from standard input device aka keyboard:
# ./readline
# -----------------------------------------------
# Copyright (c) 2005 nixCraft <http://cyberciti.biz/fb/>
# This script is licensed under GNU GPL version 2.0 or above
# -------------------------------------------------------------------------
# This script is part of nixCraft shell script collection (NSSC)
# Visit http://bash.cyberciti.biz/ for more information.
# -------------------------------------------------------------------------
 
# User define Function (UDF)
processLine(){
  line="$@" # get all args
  #  just echo them, but you may need to customize it according to your need
  # for example, F1 will store first field of $line, see readline2 script
  # for more examples
  # F1=$(echo $line | awk '{ print $1 }')
  echo $line
}
 
### Main script stars here ###
# Store file name
FILE=""
 
# Make sure we get file name as command line argument 
# Else read it from standard input device
if [ "$1" == "" ]; then
   FILE="/dev/stdin"
else
   FILE="$1"	
   # make sure file exist and readable
   if [ ! -f $FILE ]; then 
  	echo "$FILE : does not exists"
  	exit 1
   elif [ ! -r $FILE ]; then
  	echo "$FILE: can not read"
  	exit 2
   fi
fi
# read $FILE using the file descriptors
 
# Set loop separator to end of line
BAKIFS=$IFS
IFS=$(echo -en "\n\b")
exec 3<&0
exec 0<"$FILE"
while read -r line
do
	# use $line variable to process line in processLine() function
	processLine $line
done
exec 0<&3
 
# restore $IFS which was used to determine what the field separators are
IFS=$BAKIFS
exit 0
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source, and DevOps topics via:
Category List of Unix and Linux commands
File Management cat
Firewall Alpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilities dig host ip nmap
OpenVPN CentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Manager apk apt
Processes Management bg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searching grep whereis which
User Information groups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPN Alpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
39 comments… add one
  • Azaz Feb 8, 2012 @ 5:43

    Cool, I found this script very useful. I modified this and used to make our syncronisation across servers fully automated. Great Job

  • cavin Nov 10, 2011 @ 7:16

    THANK YOU for sharing the script..it’s very generous/kind sharing…THUMBS UP

  • anon Aug 23, 2011 @ 17:00

    Don’t listen that fool Dedeepthi ↑
    Here is the real code:
    —–cut here—————
    #!/bin/bash
    for i in ‘cat file.txt’
    do
    echo “$i”
    done
    —–cut here—————

  • Dedeepthi Mar 31, 2011 @ 9:10

    Hi,
    You can try this simple code to read the file “file.txt” line-by-ine….

    while read line
    do
    echo $line;
    sleep 1;
    done < file.txt

  • DIPANKAR GHOSH Jan 31, 2011 @ 12:53

    hi i want to update a port number in a file such like……
    file content…..abc.cfg

    # Temporary use by Boyajian. Thank you.
    # Plz feel free to change as per ur need. Thnx – Ayan
    ###—SBJ_VIEW_NAME=devB_4140sdd
    #SBJ_VIEW_NAME=bagay_dev_mts_1440sdd
    SBJ_VIEW_NAME=devD_4018
    #SBJ_VIEW_NAME=devD_4017
    ACE_MATCH_SERVER_HOST=localhost

    ACE_MATCH_SERVER_PORT=16711

    ACE_JASI_SERVER_PORT=75675
    ENT_CLIENT_HOSTS=localhost,imts1,charon
    ENT_SERVER_PORT=45785
    ENT_SERVER_HOST=imts1
    ENT_VIRTUAL_MEMORY=1000000000
    ENT_NUM_ACCUMULATORS=8
    SBJ_NO_SIG_CHECK=1
    AREA_PROD_BRANCH=s2010

    now if i want to update the value of ACE_MATCH_SERVER_PORT from 16711 to 78777
    what would be the code to do this stub? please give a solution at the earliest

  • CrackTheInterview Jan 21, 2011 @ 11:29

    for line in `cat abc.txt`;do
    touch path/$line
    done

    this 2 line is code enuff

  • SK Oct 19, 2010 @ 8:07

    Hi Very informative post. Thanks for this.
    But I have one confussion that
    what following code is doing. Please healp.
    BAKIFS=$IFS
    IFS=$(echo -en “\n\b”)
    exec 3<&0
    exec 0<"$FILE"

  • Vimarsh Aug 23, 2010 @ 10:16

    Guys,

    I need to read a file and see for the occurence /local/svncheckout/* at the begining of each line. If this instance is found I need to delete that particular line.

    I need to write this program using shell script (bash shell)

    Thanks a lot

  • sj Jul 9, 2010 @ 4:47

    I edited the code to input the contents of $line when ever the loop runs. it inputs line by line in a file that contains a list of id’s I pulled out of etc/passwd. for some reason it’s stops at the 25th line. it starts combining the variables to I dont’ know what’s going on. how it could work for the first 25 entries then break.

  • Samuel May 14, 2010 @ 20:31

    Thanks for posting it!!!

  • Sanjay Mar 8, 2010 @ 14:50

    Thank you for the post.
    It’s work perfectly but i need some changes
    can u help me?
    i want a search a specific pattern in a file and copy a a search result to xml file
    Thanks in advance.

  • vinu Feb 16, 2010 @ 18:55

    This is not working for me.

    sh read.sh v1.txt
    : not found:
    v1.txt
    : not found: }
    : not found:
    : not found:
    read.sh: 40: Syntax error: “elif” unexpected (expecting “then”)

  • Thushara Dec 29, 2009 @ 5:34

    Thank you for the post!!

  • ijo Dec 13, 2009 @ 17:47

    THANK YOU VERY MUCH!!!

  • Hari Oct 28, 2009 @ 21:15

    I have used your example countless time when i wanted to write a quick hack. Thank You!

  • lost Oct 18, 2009 @ 17:16

    Thanks, I used this code and edited it. It’s the closest I’ve been to the output I need but for some reason using this to export each line as an individual variable isn’t working.

    Here is the script, my guess is that it’s a syntax issue of some sort. Thanks in advance for anyone who offers help! http://pastebin.com/m3472cd27

    • 🐧 Vivek Gite Oct 20, 2009 @ 17:13

      Can you tell me what are you trying to create here? If possible please use our forum for further discussion.

  • ravi Aug 26, 2009 @ 11:39

    for i in ‘input filename’
    do
    echo $i
    done

    This code will print file line by line.

  • Maverick Crank GRey Jul 29, 2009 @ 15:16

    [code]
    gawk ‘{ system(“echo [” $1 “]”) }’ your.file
    [/code]

  • Nikhil Jul 27, 2009 @ 8:45

    Thanks A Ton!
    It Helped!! :)

  • Troy Frericks Jun 10, 2009 @ 17:02

    Thanks… used an an example.
    Troy.
    #

  • Shashidhara HN Apr 28, 2009 @ 13:16

    Good one…

  • arvind Feb 17, 2009 @ 6:52

    Hi,
    I have 2 CSV files. let say emp_data_Jan.csv (will contain the data of employees till Jan) and emp_data_feb.csv (will contain the data of employees till Feb). I need a Shell script which can read both these files and compare the data between emp_data_Jan.csv and emp_data_feb.csv. After comparision i need the output data in new CSV file. Output data will be records which are present in emp_data_Feb.csv and not in emp_data_Jan.csv. Can any one help me out in this.

    Thanks in Advance

    Regards
    Arvind

  • jinson Feb 5, 2009 @ 11:39

    Want to know how to read word by word for each line , from the file . what could be the seperator value for IFS

  • Arindam Jan 27, 2009 @ 7:20

    Hi All,
    Here is another way-

    #!/bin/sh
    ## This will read a file line by line
    
    x=`wc -l $1 |awk '{print $1}'`
    echo "No of lines are: $x \n"
    
    i=1
    while [ $i -le $x ]
    do
            echo "Line Number: $i -->\n"
            echo  "`head -$i  $1 | tail -1` \n"
            i=`expr $i + 1`
    done
  • Arty Jan 23, 2009 @ 10:58

    I was having a real mental block on the best way to read files in shell scripts and I found this site and example script. It was a real help. I did have the following thoughts though:

    I don’t think that it is really necessary to mess with IFS nor with saving and restoring FD 0. I have found that the following adaptation seems to do the job. (With one alteration: not defaulting the input file to stdin)

    # Make sure we get file name as command line argument
    FILE=${1?"No file name specified"}
    # Check that file exists and is readable
    [ ! -f $FILE ] && { echo "$FILE: does not exist"; exit 1; }
    [ ! -r $FILE ] && { echo "$FILE: cannot be read"; exit 2; }
    
    #Open file for reading
    exec 3
    

    YMMV. Hope this helps someone else.
    Toodles,
    Arty

  • Jinson Jan 6, 2009 @ 5:54

    Thanks a lot. It works fine. when i read a text from a file and create a new file using that text two small letters disappearing from the text and that leads to an invalid file path . two letters are “e”and “n”. while reading from the file it fails to read these two letters . could you please tell me possible reason for this . is it taking those letters as some delimiter or something ?

    Pre requesties

    1)template.txt in the same path contains a text “ordprg”and a blank line

    2)filenames.sh in the same path contains two lines
    “ABCDEFJKLMNOPQRSTUVWXYZ”
    “abcdefghijklmnopqrtuvwxyz”
    Following program will eliminate “e” and “n”

    1377621854603e6b2d1e23a_000001

    # Please Respond to this Thanks in Advance

  • Ravi Jan 5, 2009 @ 10:42

    Thanks it worked

  • Jinson Jan 5, 2009 @ 7:08

    Great Thanks. This is really useful .

  • David Nov 18, 2008 @ 15:34

    Thank you so much, this is just what I was looking for :)

  • vivek Oct 31, 2008 @ 14:16

    @michelek,

    This can be easily fixed by adding following code before and after while loop:

    ORIGIFS=$IFS
    IFS=$(echo -en "\n\b")
    # add your while loop here
    IFS=$ORIGIFS
    

    while / for / untile loop is using $IFS to determine what the field separators are. I’ve modified this to use end of line. Here is final modified code:

    ORIGIFS=$IFS
    IFS=$(echo -en "\n\b")
    
    exec 3
    

    The script has been updated to fix this issue. Just download updated version.

    HTH

  • michelek Oct 30, 2008 @ 12:10

    bad news for everybody who has lines with more than one space between words.

  • John Sanabria Aug 7, 2008 @ 20:50

    Excellent! very illustrative,

    thanks for your contribution! ;-)

  • Brett Jul 24, 2008 @ 19:29

    Hey this worked perfectly. I used it to automatically patch my sun system. I used smpatch analyze > patch.txt then used your script to read in the outputed patch numbers.

  • aye Jul 19, 2008 @ 16:14

    thanks!

  • vikas kale Jun 14, 2008 @ 9:09

    good…….

Leave a Reply

Your email address will not be published.

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.