UNIX - Add column at the end of CSV with double quotes using SED

I have data in following format :

"1";"abc" "2";"dfg" "3";"hij" 

I used the following command to add a column:

sed "s/$/;"newc"/" file.csv 

but i get the last column without quotes:

"1";"abc";newc "2";"dfg";newc "3";"hij";newc 

can not figure out how to update it to add double quotes and get:

"1";"abc";"newc" "2";"dfg";"newc" "3";"hij";"newc" 

1 Answer

The issue was solved in comments without proper explanation:

Surrounding sed script with single quotes doesn't work? 's/$/;"newc"/'

yes, it worked!

This answer is going to shed some light on what happened and why the solution works.

In your original command quoting goes like this:

sed "s/$/;"newc"/" file.csv # ^ ^ a matching pair of quotes # ^ ^ another pair of quotes # s/$/; / these fragments are quoted # newc this fragment is not quoted at all 

The quotes you used are consumed by the shell. Their presence tells the shell to treat quoted strings somewhat differently than non-quoted ones, e.g. the quoted semicolon (;) is not a command separator; then they disappear, i.e. the shell doesn't pass them to sed.

Note newc contains no characters special to the shell, it behaves in the same way whether it's quoted or not. This means newc might as well be quoted, like this:

sed "s/$/;""newc""/" file.csv # ^ ^ added pair of quotes 

But this is equivalent to

sed "s/$/;newc/" file.csv 

and after the shell consumes the quotes sed gets these arguments: s/$/;newc/, file.csv. As you can see the tool gets no quotes at all.

To pass quotes to sed you need to make them "survive" parsing done by the shell. There are few ways to do this. Two common approaches:

1. Escaping with \. Inside double quotes you can escape a double quote character, so it is treated as a part of the quoted string, not as a closing quote. In your case:

   sed "s/$/;\"newc\"/" file.csv 

2. Mixing quotes. A double quote inside single quotes remains. The mentioned solution uses this fact:

   sed 's/$/;"newc"/' file.csv 

   A single quote inside double quotes also remains. E.g. if you need to pass a literal '" argument to echo, this will work:

   echo "'"'"' # ^ ^ # a pair of double quotes that make the single quote survive # ^ ^ # a pair of single quotes that make the double quote survive 

Sometimes it's good to invoke set -x before a "misbehaving" command to learn what is left after the shell parses it. Your original command and two fixed ones generate this (output from sed omitted for clarity):

$ sed "s/$/;"newc"/" file.csv # original command + sed s/$/;newc/ file.csv $ # the above line contains what sed really got $ sed "s/$/;\"newc\"/" file.csv # fixed + sed s/$/;"newc"/ file.csv $ # this time sed got the right string $ sed 's/$/;"newc"/' file.csv # also fixed + sed s/$/;"newc"/ file.csv $ # again the right string $ 

Note: at the end invoke set +x to revert what set -x did.