Dec 192009
How to remove BOM from UTF-8?
Answer:
# awk '{if(NR==1)sub(/^\xef\xbb\xbf/,"");print}' text.txt
Source: http://stackoverflow.com/questions/1068650/using-awk-to-remove-the-byte-order-mark
Updated: (Suggested by Van Overveldt Peter)
# tail --bytes=+4 text.txt
My preferred command to get rid of the BOM, is:
tail --bytes=+4 UTF8WithBom.txt > UTF8WithoutBom.txt
Thanks for your suggestion, I will include your comments in the post soon.
Thanks again.
This little line of code saved my tail. I had a latex source file that had picked this up and it was refusing to compile it to a PDF because of this nasty little character.
Thanks a bundle guys
Same problem here, I searched around iconv:
iconv -c -f utf8 -t iso88591 document.txt | iconv -f iso88591 -t utf8 -o document-without-bom.txt
While searching I read uconv has a --remove-signature option, http://linux.die.net/man/1/uconv (well uconv is a Ruby application).
And finally I found the tail command which works fine, nice to have found that out.
Cheers
Another one, in Ruby, modifying the file in place:
ruby -e 'data = File.read(ARGV.first).sub(/\A\xef\xbb\xbf/,""); File.open(ARGV.first, "w") { |f| f.write data }' /path/to/file
Be aware that the "tail" version will work ONLY if the file actually contains the BOM.
In other words, use the "awk" version if you're not sure whether the input file contains a BOM or not.
cat text.txt | sed 's/\xef\xbb\xbf//g'